akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Issue with Running PyTorch Model on MPS of Apple M1

username: BoofKoor
cooked: <p>I tried to run my model on the MPS of my MacBook Air, but this warning and error raised:</p>
<p>UserWarning: The operator ‘aten::sgn.out’ is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at /Users/runner/work/_temp/anaconda/conda-bld/pytorch_1670525498485/work/aten/src/ATen/mps/MPSFallback.mm:11.)<br>
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass</p>
<p>UserWarning: Error detected in ConvolutionBackward0. Traceback of forward call that caused the error</p>
<p>(Triggered internally at /Users/runner/work/_temp/anaconda/conda-bld/pytorch_1670525498485/work/torch/csrc/autograd/python_anomaly_mode.cpp:119.)<br>
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass</p>
<p>File “/Users/user/miniconda3/envs/torch/lib/python3.10/site-packages/torch/autograd/<strong>init</strong>.py”, line 197, in backward<br>
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
RuntimeError: Function ‘ConvolutionBackward0’ returned nan values in its 0th output.<br>
My system specifications are:<br>
MacOS:Ventura 13.2<br>
Chip Apple M1<br>
Python  3.10.9<br>
PyTorch 1.13.1</p>

username: rogerstaxxxx
cooked: <p>Same here! Please help us!</p>

username: standev
cooked: <p>Having the same issue here</p>

username: marksaroufim
cooked: <p>If you need this please follow this issue <a href="https://github.com/pytorch/pytorch/issues/86805" class="inline-onebox" rel="noopener nofollow ugc">[MPS] Add support for aten::sgn.out for MPS backend · Issue #86805 · pytorch/pytorch · GitHub</a></p>

username: standev
cooked: <p>Thank you! I will watch it</p>
RuntimeError: gather(): Expected dtype int64 for index

username: Aniruddha-JU
cooked: <p>def select_beam_items(x, ids):<br>
id_shape = list(ids.size())<br>
id_rank = len(id_shape)<br>
assert len(id_shape) == 2<br>
x_shape = list(x.size())<br>
x = torch.reshape(x, [batch_size, K] + x_shape[1:])<br>
x_rank = len(x_shape) + 1<br>
assert x_rank &gt;= 2<br>
if id_rank &lt; x_rank:<br>
ids = torch.reshape(<br>
ids, id_shape + [1] * (x_rank - id_rank))<br>
ids = ids.expand(id_shape + x_shape[1:])<br>
x=x.type(torch.int64)<br>
y = torch.gather(x, 1, ids)<br>
y = torch.reshape(y, x_shape)<br>
return y</p>
<p>My code was running earlier, but now I am getting RuntimeError: gather(): Expected dtype int64 for index</p>

username: shivammehta007
cooked: <p>What I suspect is that ids are a float tensor and gather requires them to be integer could you try doing <code>y = torch.gather(x, 1, ids.int())</code> to see if that is the case?</p>
<p>Also, to have better code formatting put the code into three <code>```</code>   like <code>``` if x == 0: print 0 ```</code> when done on a code block it will retrain the indentations and help with the formatting.</p>

username: Aniruddha-JU
cooked: <p>def select_beam_items(x, ids):<br>
id_shape = list(ids.size())<br>
id_rank = len(id_shape)<br>
assert len(id_shape) == 2<br>
x_shape = list(x.size())<br>
x = torch.reshape(x, [batch_size, K] + x_shape[1:])<br>
x_rank = len(x_shape) + 1<br>
assert x_rank &gt;= 2<br>
if id_rank &lt; x_rank:<br>
ids = torch.reshape(<br>
ids, id_shape + [1] * (x_rank - id_rank))<br>
ids = ids.expand(id_shape + x_shape[1:])<br>
#ids=ids.type(torch.int64)<br>
y = torch.gather(x, 1, ids.int64())<br>
y = torch.reshape(y, x_shape)<br>
return y</p>

username: Aniruddha-JU
cooked: <p>RuntimeError: gather(): Expected dtype int64 for index  same error</p>

username: levitation
cooked: <p>Use <code>torch.gather(x, 1, ids.long())</code> instead. Note <code>.long()</code> here.</p>
Float32 to Bfloat16 conversion

username: Arul
cooked: <p>When pytorch converts fp32 to bfloat16, does it do truncation or rounding by default?</p>
<p>x = torch.randn([3,4]) # fp32<br>
x.to(torch.bfloat16) <span class="hashtag">#bfloat16</span></p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/main/c10/util/BFloat16.h#L45">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/main/c10/util/BFloat16.h#L45" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/main/c10/util/BFloat16.h#L45" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/main/c10/util/BFloat16.h#L45</a></h4>



    <pre class="onebox"><code class="lang-h">
      <ol class="start lines" start="35" style="counter-reset: li-counter 34 ;">
          <li>  // but it fails in the HIP environment.</li>
          <li>  tempRes = reinterpret_cast&lt;float*&gt;(&amp;tmp);</li>
          <li>  res = *tempRes;</li>
          <li>#else</li>
          <li>  std::memcpy(&amp;res, &amp;tmp, sizeof(tmp));</li>
          <li>#endif</li>
          <li>
          </li>
<li>  return res;</li>
          <li>}</li>
          <li>
          </li>
<li class="selected">inline C10_HOST_DEVICE uint16_t bits_from_f32(float src) {</li>
          <li>  uint32_t res = 0;</li>
          <li>
          </li>
<li>#if defined(USE_ROCM)</li>
          <li>  // We should be using memcpy in order to respect the strict aliasing rule</li>
          <li>  // but it fails in the HIP environment.</li>
          <li>  uint32_t* tempRes = reinterpret_cast&lt;uint32_t*&gt;(&amp;src);</li>
          <li>  res = *tempRes;</li>
          <li>#else</li>
          <li>  std::memcpy(&amp;res, &amp;src, sizeof(res));</li>
          <li>#endif</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
I see that it has utility functions to do both but how can I find which gets triggered by default?</p>

username: eqy
cooked: <p>I think you can write a simple script to test this:</p>
<pre><code class="lang-auto">import torch

elemns = 10
a = torch.rand(elemns, dtype=torch.float32)
b = a.bfloat16()
c = b.float()
print(f"got {(c&gt;a).sum()} elements rounded up out of {elemns}")
print(c, a)
</code></pre>
<pre><code class="lang-auto">got 6 elements rounded up out of 10
tensor([0.0253, 0.2080, 0.1826, 0.1118, 0.3809, 0.4434, 0.5742, 0.9453, 0.8789,
        0.4004]) tensor([0.0253, 0.2080, 0.1821, 0.1119, 0.3806, 0.4438, 0.5732, 0.9467, 0.8782,
        0.3997])
</code></pre>
OSError: [Errno Incorrect file size] ffhq-dataset-v2.json

username: shrbrh
cooked: <p>Hello, I am trying to download the FFHQ dataset from the repository in this <a href="https://github.com/NVlabs/ffhq-dataset" rel="noopener nofollow ugc">link</a>. But when I run the command <code>python3 download_ffhq.py --json --images </code> I get the following error:</p>
<pre data-code-wrap="File"><code class="lang-plaintext">    raise IOError('Incorrect file size', file_path)
OSError: [Errno Incorrect file size] ffhq-dataset-v2.json
</code></pre>
<p>I am following the instructions provided in the repository but it won’t work. Can anyone please tell me what does this error mean and how do I solve this?</p>

username: ptrblck
cooked: <p>The question doesn’t seem to be related to PyTorch so I would recommend asking the code owners in their repository.</p>
ValueError: Target size (torch.Size([2, 200])) must be the same as input size (torch.Size([2, 5]))

username: korawit_longjun
cooked: <p>import sklearn<br>
from sklearn.datasets import make_circles<br>
import torch<br>
from torch import nn</p>
<p>n_sample = 1000</p>
<p>device = “cpu”</p>
<p>x, y = make_circles(n_sample,<br>
noise=0.03,<br>
random_state=42</p>
<pre><code>                )
</code></pre>
<p>x,y</p>
<p>import pandas as pd</p>
<p>circles = pd.DataFrame({“X1”: x[:, 0], “X2”: x[:,1],“label”:y})<br>
circles.head(10)<br>
import matplotlib.pyplot as plt</p>
<p>plt.scatter(x=x[:,0],y=x[:,1],c=y,cmap=plt.cm.RdYlBu)<br>
X_sample = x[0]</p>
<p>y_sample = y[0]</p>
<p>X_sample<br>
X = torch.from_numpy(x).type(torch.float)</p>
<p>y = torch.from_numpy(y).type(torch.float)<br>
from sklearn.model_selection import train_test_split</p>
<p>X_train, y_train, X_test,y_test = train_test_split(X,<br>
y,<br>
test_size=0.2,<br>
random_state=42)</p>
<p>y_train.squeeze(dim=1)<br>
class CircleModelV0(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.layer_1 = nn.Linear(in_features=2, out_features=5)<br>
self.layer_2 = nn.Linear(in_features=5, out_features=2)<br>
def forward(self, x):<br>
return self.layer_2(self.layer_1(X))</p>
<p>model_0 = CircleModelV0()<br>
model_0<br>
with torch.inference_mode():<br>
untrained_pred = model_0(X_train)<br>
untrained_pred<br>
loss_fn = nn.BCEWithLogitsLoss()</p>
<p>optimizer = torch.optim.SGD(params=model_0.parameters(),<br>
lr=0.01<br>
)</p>
<p>loss_fn,optimizer<br>
def accuracy_fn(y_true, y_pred):<br>
correct = torch.eq(y_true, y_pred).sum().item()<br>
acc = (correct/len(y_pred)) * 100<br>
return acc<br>
model_0.eval()</p>
<p>with torch.inference_mode():</p>
<p>y_logits = model_0(X_test.to(device))[:5]</p>
<p>y_logits.shape<br>
y_preds_probs = torch.sigmoid(y_logits)</p>
<p>y_preds_probs</p>
<p>torch.round(y_preds_probs)<br>
y_pred = torch.round(y_preds_probs)</p>
<p>y_pred_label = torch.round(torch.sigmoid(model_0(X_test)))</p>
<p>epochs = 100</p>
<p>for epochs in range(epochs):<br>
model_0.train()<br>
y_logtis = torch.round(X_train).squeeze()<br>
y_pred = torch.round(torch.sigmoid(y_logits))<br>
loss = loss_fn(y_logits,y_train)<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()</p>

username: ptrblck
cooked: <p>The error message points to a shape mismatch between the model output and the target, which are expected to have the same shape for <code>nn.BCEWithLogitsLoss</code>.<br>
Check both tensors and try to isolate why the shape is different.</p>
<p>Your code is unfortunately neither executable nor properly formatted, you also didn’t explain your use case, so would need to try to narrow down the shape mismatch a bit more.</p>
Masking gradients intra-minibatch per example in a Linear layer

username: Matthew_Kennel
cooked: <p>I’d like some advice from this forum.</p>
<p>Here’s my problem: I want to selectively mask (set to zero) certain elements of the gradient of a dense linear matmul+bias forward operator, selected by a low-cardinality categorical per train example, a vector of which is passed simultaneously through the inputs.</p>
<p>This means somewhere I need access to per-example gradients, mask them accordingly and then sum over the minibatch in the backward pass.   A hook to post multiply gradients is not enough, because they need to be selected per example.</p>
<p>I think I may need to write a custom backward() operator, as the backward is not exactly the gradient of the forward operator.</p>
<p>A an alternate source code implementation of nn.Linear, including explicit backward pass where the sum over the minibatch is explicitly implemented (so I can selectively mask right before that part) would be helpful.  But I have not been able to find this—it’s a difficult thing to search for as there’s much too much posted with the keywords but without these particular features.   I’ve seen libraries which post-hoc extract the per-example gradients but that isn’t enough, they need to be modified during backprop.</p>
<p>The overall goal is to guide the training process with knowledge of the categories, but at inference/model deployment time this categorical variable is not known and the forward inference is a standard matrix multiplication.  This is for a real commercial product btw.</p>
FSDP- Fully sharded dataparallel on model that has requires_grad=False

username: Andras_Iani
cooked: <p>Is it possible to wrap a teacher model in FSDP that is in eval mode and has requires_grad=False?<br>
My setup is knowledge distillation between a huge teacher and a small student. I`d like to only partition weights of the teacher across 6 gpus, or both teacher and student.</p>

username: agu
cooked: <p>FSDP should be able to work on the teacher only, though if see something like the gradients of your student model are not being computed correctly, feel free to file an issue on Github!</p>

username: Andras_Iani
cooked: <p>Thank you! Indeed I managed to wrap transformer blocks of my teacher only.<br>
However, I see there are multiple ways of defining the wrapping policy.<br>
I have 2 questions here:</p>
<ol>
<li>
<p>How is it possible to have more FSDP modules than nr of ranks (gpus) ? Is there a documentation explaining this, I understood so far that each ran will store only one shard of parameters/optimizers/gradients, but can there be more shards/FSDP modules than nr of gpus? If so, why?</p>
</li>
<li>
<p>What policy would you recommend. Are there any guidelines? What happens if I set too many FSDP modules?</p>
<p>auto_wrap_policy = functools.partial(<br>
transformer_auto_wrap_policy, transformer_layer_cls={GPT2Block}</p>
<p>auto_wrap_policy = functools.partial(<br>
size_based_auto_wrap_policy, min_num_params=2000000<br>
)</p>
</li>
</ol>
<p>Thanks for your feedback, it is very very useful in my work!</p>
How can we skip a step with NaN loss in the training_step when using Distributed Data Parallel (DDP)?

username: wangyanhui666
cooked: <p>How can we skip a step with NaN loss in the training_step when using Distributed Data Parallel (DDP) across multiple machines and multiple GPUs?</p>

username: Wei-Hsin
cooked: <p>i guess you could use function torch.nan_to_num(<em>input</em> , <em>nan=0.0</em> , <em>posinf=None</em> , <em>neginf=None</em> , *** , <em>out=None</em> ) after each layer of models so that value wont be nan, but I suggest to trace back models and loss functions where cause nan because nan would be happened when there have some bug</p>

username: wangyanhui666
cooked: <p>Thank you for your reply. Can I set the gradient to 0 for variables with NaN before the optimizer? NaN issues rarely occur, but once they do, they can lead to errors in subsequent training. Therefore, I want to skip the step that causes NaN.</p>

username: Wei-Hsin
cooked: <p>yes,but i suggest set none zero parameter would better</p>
Isn't nn.Transformer confusing with num_decoder_layers=0?

username: ragnaarok
cooked: <p>Hi,<br>
I was trying to use nn.Transformer without decoding layers and I found the behavior of the model with num_decoder_layers=0 to be a bit confusing.<br>
Perhaps it’s just my point of view.<br>
When I set num_decoder_layers to 0, I would like the model to do:</p>
<ul>
<li>out = encoders(src)</li>
</ul>
<p>but what the model seems to be doing is:</p>
<ul>
<li>mem = encoders(src)</li>
<li>out = norm(tgt)</li>
</ul>
<p><a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/transformer.html#Transformer" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.modules.transformer — PyTorch 2.0 documentation</a> (decoder(tgt, mem) without decoding layers seems to be returning norm(tgt))</p>
<p>It seems that the “issue” has already been notified: <a href="https://github.com/pytorch/pytorch/issues/68053" class="inline-onebox" rel="noopener nofollow ugc">[docs] `nn.Transformer` is not possible to be used to implement BERT · Issue #68053 · pytorch/pytorch · GitHub</a> / <a href="https://discuss.pytorch.org/t/how-to-implement-bert-using-torch-nn-transformer/90981" class="inline-onebox">How to implement BERT using torch.nn.Transformer?</a><br>
I guess the official way to do this is by using nn.TransformerEncoder and nn.TransformerEncoderLayer.</p>
<p>It would perhaps be less confusing if this line of code was added to the init of nn.Transformer?</p>
<ul>
<li>self.decoder = self.decoder if num_decoder_layers else lambda dec_args: dec_args[1]<br>
(or a cleaner version of this)</li>
</ul>
<p>or, if the setting “num_decoder_layers=0” isn’t expected, an assert could be used to avoid this confusing behavior.</p>
<p>Thanks</p>
Reduce dimensionality 5D -> 3D using conv3d

username: r00bi
cooked: <p>I want to reduce a tensor with shape of (32,32,26, 40,10) to (32,32,512). Basically, I want to keep the batch size and sequence length, and embed the last three dimension to 512.<br>
How can I do this reduction using conv3d?<br>
Thanks!</p>

username: ptrblck
cooked: <p>I don’t know how you are interpreting the input data, but I guess you are treating the channel dimension in <code>dim1</code> as the “sequence length”?<br>
3D convolutions expect an input in the shape <code>[batch_size, channels, depth, height, width]</code>.<br>
I don’t think using a plain <code>nn.Conv3d</code> would work directly as you are increasing the <code>depth</code> size from <code>26</code> to <code>512</code>.<br>
Maybe you want to flatten the <code>depth, height, width</code> into a single feature dimension and apply an <code>nn.Linear</code> layer instead?</p>
<pre><code class="lang-python">x = torch.randn(32, 32, 26, 40, 10)
x = x.view(x.size(0), x.size(1), -1)
linear = nn.Linear(26*40*10, 512)
out = linear(x)
print(out.shape)
# torch.Size([32, 32, 512])
</code></pre>

username: r00bi
cooked: <p>Hi,<br>
Yes, the sequence length is interpreted as channel. using a linear on flatten d, h, w is a solution.<br>
Thanks!</p>
Meaning of positive and negative values of saliency map

username: davids
cooked: <p>To help interpret the neural network, calculation of saliency map is often used. A saliency map is calculated by using backpropagation to compute the gradients of logits (of the correct label) with respect to the input of the network (image, time-series…).</p>
<p>The calculated saliency map is a tensor of same shape as input and includes both positive and negative values. In related work, people always use the absolute value of saliency map to determine the important regions of the input.</p>
<p>Why is that so? To my understanding, the positive values in the saliency map are the regions of the input that push the prediction towards the correct class, while the negative values push the prediction to the other-class. Is my understanding correct? Therefore, to extract the regions that include information that is typical for the correct class, we should only take the regions of the positive saliency map. Right?</p>
Tensor[1] get the original tensor and tensor[[1]] gets a copy, is this expected?

username: JuanFMontesinos
cooked: <p>Hi,<br>
quick surprising question, I always learn something new <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<pre><code class="lang-auto">w=torch.rand(10)
w[[4,5]].zero_()
print(w)
w[4:5].zero_()
print(w)
</code></pre>
<pre><code class="lang-auto">w:
tensor([0.8315, 0.6297, 0.2248, 0.7077, 0.9125, 0.7869, 0.5703, 0.2823, 0.0974, 0.5943])
w[[4,5]].zero_()
tensor([0.8315, 0.6297, 0.2248, 0.7077, 0.9125, 0.7869, 0.5703, 0.2823, 0.0974, 0.5943])
w[4:5].zero_()
tensor([0.8315, 0.6297, 0.2248, 0.7077, 0.0000, 0.7869, 0.5703, 0.2823, 0.0974, 0.5943])
</code></pre>
<p>Seems when masking it gets a copy and when slicing it takes the subtensor. Is this expected?</p>
Warning messages Visual Studio 2022 using TorchProject

username: Jerome_Cohen
cooked: <p>hi - I just downloaded the pytorch 2022 plugin and i can build my project. Works fine with cuda etc., but i get a TON of messages when compiling</p>
<p>how can i make them disappear? here are just a few, but in total i have it seems like 100 messages</p>
<p>C:\dev\libtorch-win-shared-with-deps-2.0.0+cu117\libtorch\include\c10/util/Optional.h(212,1): warning C4624: ‘c10::constexpr_storage_t’: destructor was implicitly defined as deleted<br>
1&gt;        with<br>
1&gt;        [<br>
1&gt;            T=c10::SymInt<br>
1&gt;        ]<br>
1&gt;C:\dev\libtorch-win-shared-with-deps-2.0.0+cu117\libtorch\include\c10/util/Optional.h(411): message : see reference to class template instantiation ‘c10::constexpr_storage_t’ being compiled<br>
1&gt;        with<br>
1&gt;        [<br>
1&gt;            T=c10::SymInt<br>
1&gt;        ]<br>
1&gt;C:\dev\libtorch-win-shared-with-deps-2.0.0+cu117\libtorch\include\c10/util/Optional.h(549): message : see reference to class template instantiation ‘c10::trivially_copyable_optimization_optional_base’ being compiled<br>
1&gt;        with<br>
1&gt;        [<br>
1&gt;            T=c10::SymInt<br>
1&gt;        ]<br>
1&gt;C:\dev\libtorch-win-shared-with-deps-2.0.0+cu117\libtorch\include\c10/util/Optional.h(549): message : see reference to alias template instantiation ‘c10::OptionalBase’ being compiled<br>
1&gt;        with<br>
1&gt;        [<br>
1&gt;            T=c10::SymInt<br>
1&gt;        ]<br>
1&gt;C:\dev\libtorch-win-shared-with-deps-2.0.0+cu117\libtorch\include\c10/core/TensorImpl.h(1605): message : see reference to class template instantiation ‘c10::optional<a>c10::SymInt</a>’ being compiled<br>
1&gt;C:\dev\libtorch-win-shared-with-deps-2.0.0+cu117\libtorch\include\c10/util/Optional.h(446,1): warning C4624: ‘c10::trivially_copyable_optimization_optional_base’: destructor was implicitly defined as deleted<br>
1&gt;        with<br>
1&gt;        [<br>
1&gt;            T=c10::SymInt<br>
1&gt;        ]</p>
Training from saved chekpoints failing on multiple gpus

username: HRU
cooked: <p>I am using pytorch-lightining for training . i trained a model using 4 gpus and now i want to resume the trainig from save checkpoints but i am getting following traceback error when i set number of devices &gt;2 gpus.</p>
<p>Traceback (most recent call last):<br>
File “train.py”, line 70, in <br>
trainer.fit(model, dm)#,ckpt_path=“./580sdysk/checkpoints/Content_Emb-epoch=09-val_loss=0.01.ckpt”)<br>
File “/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py”, line 608, in fit<br>
call._call_and_handle_interrupt(<br>
File “/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py”, line 36, in _call_and_handle_interrupt<br>
return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)<br>
File “/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/multiprocessing.py”, line 113, in launch<br>
mp.start_processes(<br>
File “/lib/python3.8/site-packages/torch/multiprocessing/spawn.py”, line 197, in start_processes<br>
while not context.join():<br>
File “/lib/python3.8/site-packages/torch/multiprocessing/spawn.py”, line 140, in join<br>
raise ProcessExitedException(<br>
torch.multiprocessing.spawn.ProcessExitedException: process 1 terminated with signal SIGKILL</p>
AP and AR equals to 0 or -1

username: meyssank
cooked: <p>I am following this tutorial <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">TorchVision Object Detection Finetuning Tutorial — PyTorch Tutorials 2.2.0+cu121 documentation</a><br>
While training, the cocoevaluator returns AP and AR equals to 0 or -1 on the test set.<br>
Does someone have an understanding of what is going on ?<br>
Did someone stepped upon this problem ?</p>
<p>My LR is at 0.0000000 even when i set it up to 1e-5.</p>
<p>here is an image :<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/1/918eda6d85f6b3b019011c7e50f7dad5af8bd64b.png" alt="image" data-base62-sha1="kLFihqCxbwzA3AqHXczoh9x1lfJ" width="574" height="205"></p>
<p>EDIT : Ok so i switched to another .ipynb notebook and the problem disappeared</p>

username: meyssank
cooked: <p>It happened to me again, solution : increase learning rate</p>
Best way to run different models on the same data in parallel

username: pepper8362
cooked: <p>Assume I have a list of models <code>models = [model1, model2, model3, ....]</code>, I want to get the prediction of these models on the same <code>data</code> tensor in parallel. What would be the best way to do it? I tried to use multiprocessing.Pool, but it seems the process starts very slow.</p>
<p>Thanks for help!</p>

username: eqy
cooked: <p>(I’m assuming that you either have just a single hardware device)</p>
<p>Simply leveraging CUDA Graphs and running the models in sequence might be tough to beat if your workload has static shapes:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/297;"><img src="https://pytorch.org/assets/images/cudagraphs-pytorch.png" class="thumbnail" width="690" height="297"></div>

<h3><a href="https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/" target="_blank" rel="noopener nofollow ugc">Accelerating PyTorch with CUDA Graphs</a></h3>

  <p>Today, we are pleased to announce a new advanced CUDA feature, CUDA Graphs, has been brought to PyTorch. Modern DL frameworks have complicated software stacks that incur significant overheads associated with the submission of each operation to the...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>It would be unlikely, but if you could express your model(s) as subnetworks in a single large model (e.g., by combining that could also be a way to accelerate your workload. A trivial example of this would be if your models only had linear layers (where each corresponding layer had the same shape), you could combine the smaller models into a single model with batch matrix multiplication: <a href="https://pytorch.org/docs/stable/generated/torch.bmm.html" class="inline-onebox" rel="noopener nofollow ugc">torch.bmm — PyTorch 2.0 documentation</a> (b would be the <em>model</em> dimension in this case).</p>
Understanding gradients of gradients

username: chenqb1989
cooked: <p>My question is how autograd behave when you take the grad of a grad. Suppose <em>y=wx, out=relu(y), dout/dw = dout/dy * dy/dw = dout/dy * x</em>; In my opinion, If I take the derivative of <em>dout/dw</em>, <em>d(dout/dw)/dw</em> should all be 0, for there is no <em>w</em> in <em>dout/dw</em><br>
However, in the following example, if I take <strong>conv+relu</strong>, I get all zeros; If I take <strong>conv+batchnorm+relu</strong>, the derivative is not all zero; If I just use <strong>conv</strong>, I get an error “One of the differentiated Tensors appears to not have been used in the graph.”<br>
Anyone can help explain this? Thanks!</p>
<pre><code class="lang-auto">def main():
    resnet = ptcv_get_model("resnet18", pretrained=True)
    resnet_modules = list(resnet.modules())

    model = nn.Sequential(
        resnet_modules[0].features.init_block.conv.conv,
        # resnet_modules[0].features.init_block.conv.bn,
        resnet_modules[0].features.init_block.conv.activ,
    )

    inputs = torch.rand(size = (1, 3, 224, 224), dtype=torch.float32, requires_grad=True)
    outputs = model(inputs)
    outputs.backward(torch.ones_like(outputs, dtype=torch.float), create_graph=True)

    params, grads = get_params_grad(model)
    model.zero_grad()
    v = [
        torch.randint_like(p, high=2)
        for p in params
    ]

    # generate Rademacher random variables
    for v_i in v:
        v_i[v_i == 0] = -1
    
    Hv = torch.autograd.grad(grads,
                             params,
                             grad_outputs=v,
                             only_inputs=True,
                             retain_graph=True)
    print(Hv)

def get_params_grad(model):
    params = []
    grads = []
    for param in model.parameters():
        if not param.requires_grad:
            continue
        params.append(param)
        grads.append(0. if param.grad is None else param.grad + 0.)
    return params, grads
</code></pre>
<p>In fact, I want to use grad of grad to compute hessian trace, but I cannot know its meaning.</p>
OutOfMemoryError Hlep please

username: Peter_Stanton
cooked: <p>Hi guys,<br>
I am getting this error code when running Stable Diffusion, Any fixes would be great. I am not very tech savvy so please be gentle <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<p>OutOfMemoryError: CUDA out of memory. Tried to allocate 480.00 MiB (GPU 0; 6.00 GiB total capacity; 4.54 GiB already allocated; 0 bytes free; 4.60 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</p>
<p>Running GeForce GTX 1660 SUPER on Windows 11</p>

username: ptrblck
cooked: <p>Try to reduce the batch size which would also reduce the required GPU memory for the execution.</p>

username: Peter_Stanton
cooked: <p>how would I go about that mate?</p>

username: Peter_Stanton
cooked: <p>Batch size is only one.</p>

username: ptrblck
cooked: <p>If the batch size is already 1 you could check if <a href="https://pytorch.org/docs/stable/checkpoint.html#torch-utils-checkpoint"><code>torch.utils.checkpoint</code></a> can be applied to the model to trade compute for memory (assuming you want to train the model).<br>
If you are not interested in training you should make sure the model runs in a <code>with torch.inference_mode()</code> context.<br>
Also, you might want to check if running the model in a <code>with torch.cuda.amp.autocast()</code> context would help.<br>
However, I’m not familiar with your use case as you haven’t shared much information so I don’t know if any of these utils. are already applied.</p>
TypeError: __init__() got an unexpected keyword argument 'gpus'

username: show_Wang
cooked: <pre><code class="lang-auto">early_stop_callback = EarlyStopping(
    monitor="val_loss", 
    min_delta=0.00, 
    patience=10, 
    verbose=False, 
    mode="min",
)
checkpoint_callback = ModelCheckpoint(save_top_k=1, monitor="val_loss")
trainer = pl.Trainer(
    gpus=1, 
    callbacks=[early_stop_callback, checkpoint_callback],
    max_epochs=500,
    val_check_interval=len(train_dataloader),
)
trainer.fit(segformer_finetuner)
</code></pre>
<p>after running…</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-25-07f7faa3f83d&gt; in &lt;cell line: 11&gt;()
      9 checkpoint_callback = ModelCheckpoint(save_top_k=1, monitor="val_loss")
     10 
---&gt; 11 trainer = pl.Trainer(
     12     gpus=1,
     13     callbacks=[early_stop_callback, checkpoint_callback],

/usr/local/lib/python3.9/dist-packages/pytorch_lightning/utilities/argparse.py in insert_env_defaults(self, *args, **kwargs)
     67 
     68         # all args were already moved to kwargs
---&gt; 69         return fn(self, **kwargs)
     70 
     71     return cast(_T, insert_env_defaults)

TypeError: __init__() got an unexpected keyword argument 'gpus'
</code></pre>
<p>This link is the original file I referred to, and there were no significant changes made to it. <a href="https://colab.research.google.com/drive/1bwxxJX4rYmF4k4923PgJeXEWjxxb67jN?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a> The program’s environment is on Colab</p>

username: ptrblck
cooked: <p>The Lightning <code>Trainer</code> class doesn’t seem to support the <code>gpus</code> argument as seen in their <a href="https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.trainer.trainer.Trainer.html#lightning.pytorch.trainer.trainer.Trainer">docs</a> so you might need to update your code and maybe use <code>devices</code> instead.</p>

username: show_Wang
cooked: <p>What suggestions do you have for improvement? Thank you.</p>
Is there an uncollate function?

username: Alon_Hazan
cooked: <p>The <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc">Dataloader</a> collates (with <a href="https://pytorch.org/docs/stable/data.html?highlight=collate#torch.utils.data._utils.collate.collate" rel="noopener nofollow ugc">this</a>?) the samples such that the labels are also collated within each key of the sample dictionary. I mistakenly thought I could extract a specific sample meta by simply using:</p>
<pre><code class="lang-python">ind = 12
meta_of_one_sample = {key : labels[key][ind] for key in labels}
</code></pre>
<p>it gets more complicated when the sample dictionary has a value that is a nested list, then the nested lists collates too and the code above returns nonsense.<br>
Is there an uncollate function?</p>
Training Faster R-CNN model with COCO dataset has been consistently unsuccessful

username: Zantou
cooked: <p>My computer has downloaded the COCO dataset and now I want to use PyTorch to load the dataset and train a Faster R-CNN object detection model. However, there seems to be a problem with loading the data. Can you help me solve this issue?The following is my code.</p>
<pre><code class="lang-python">import torchvision
import torch
import torchvision.transforms as transforms
from torch.utils.data import DataLoader
from torchvision.datasets import CocoDetection
from torchvision.models.detection import FasterRCNN
from torchvision.models.detection.rpn import AnchorGenerator
import torch.optim as optim
import torch.nn as nn
from torch.utils.tensorboard import SummaryWriter

## dataloader
transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])

train_data_dir = '../data/coco/train2017'
train_ann_file = '../data/coco/annotations/instances_train2017.json'
val_data_dir = '../data/coco/val2017'
val_ann_file = '../data/coco/annotations/instances_val2017.json'

train_dataset = CocoDetection(root=train_data_dir, annFile=train_ann_file, transform=transform)
test_dataset = CocoDetection(root=val_data_dir, annFile=val_ann_file, transform=transform)
train_loader = DataLoader(train_dataset, batch_size=4, shuffle=True)
val_loader = DataLoader(test_dataset, batch_size=4, shuffle=False)

## model
backbone = torchvision.models.mobilenet_v2(pretrained=True).features
backbone.out_channels = 1280
anchor_generator = AnchorGenerator(sizes=((32, 64, 128, 256, 512),), aspect_ratios=((0.5, 1.0, 2.0),))
roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0], output_size=7, sampling_ratio=2)
model = FasterRCNN(backbone, num_classes=80, rpn_anchor_generator=anchor_generator, box_roi_pool=roi_pooler)

## optimizeer and others
params = [p for p in model.parameters() if p.requires_grad]
optimizer = optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005)
lr_scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.1)
criterion = nn.SmoothL1Loss()

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')


## train model
model.train()
num_epochs = 10
for epoch in range(num_epochs):
    for i, (images, targets) in enumerate(train_loader):
        images = list(image.to(device) for image in images)
        targets = [{k: v.to(device) for k, v in t.items()} for t in targets]

        loss_dict = model(images, targets)
        losses = sum(loss for loss in loss_dict.values())

        optimizer.zero_grad()
        losses.backward()
        optimizer.step()

        if i % 100 == 0:
            print(f"Epoch [{epoch+1}/{num_epochs}], Step [{i+1}/{len(train_loader)}], Total Loss: {losses.item()}")
</code></pre>
Tensor.__getitem__ docs

username: HjalmarLucius
cooked: <p>I’ve been looking everywhere for docs on the various ways to index using <code>Tensor.__getitem__</code> but cannot find it - does it exist? Details on what to expect when indexing with <code>int</code> / <code>slice</code> / <code>BoolTensor</code> / <code>LongTensor</code> / <code>tuple[LongTensor, ...]</code> / <code>ellipsis</code> etc.</p>
How is torch resent trained?

username: wasabi
cooked: <p>Are the resent in torch model zoo trained as in <a href="https://arxiv.org/pdf/1512.03385.pdf" rel="noopener nofollow ugc">1512.03385.pdf (arxiv.org)</a>? Namely, SGD with 256 batch size etc? Thanks!</p>

username: ptrblck
cooked: <p>The recipes are linked <a href="https://pytorch.org/vision/stable/models.html#table-of-all-available-classification-weights">here</a> and you could check them depending on the model you are interested in.</p>
Jacfwd become slower after update pytorch ("We’ve integrated functorch into PyTorch---Documentation")

username: miraboreasu
cooked: <p>Hello, I used to use</p>
<pre><code class="lang-auto">from functorch import jacfwd
</code></pre>
<p>and Jacfwd takes 3.5s, after the update, I am using</p>
<pre><code class="lang-auto">from torch.func import jacfwd
</code></pre>
<p>Jacfwd is taking almost 6s, is anyone noticed this?</p>
Learning weights of ensembles

username: Subhankar_Ghosh
cooked: <p>We have a time series dataset (spatiotemporal, but not an image/video). The dataset is in 3D, where each (x,y,t) coordinate has a numeric value (such as the temperature of the sea at that location and at that specific point in time). So we can think of it as a matrix with a temporal component. The dataset is similar to this but with just one channel:<br>
<a href="https://i.stack.imgur.com/tP1Lz.png" rel="noopener nofollow ugc">Dataset</a></p>
<p>We have the ground truth and two or more deepnets trained on data that approximate the ground truth. Can we learn the weights of an ensemble of the two/more models that would output a good approximation to the ground truth? Thanks!</p>

username: arunppsg
cooked: <p>Assuming that the weights in the ensemble should sum to 1, we can initialize a parameter <code>a</code> via <code>nn.Parameter</code> which represented the weight assigned to a model 1 (<code>a</code>) and the other model will have weight (<code>1-a</code>). We can then train the parameter with inputs as output of two models and true value as the ground truth, to find an optimal parameter,</p>

username: Subhankar_Ghosh
cooked: <p>Hi <a class="mention" href="/u/arunppsg">@arunppsg</a> , thanks a lot for the response.</p>
<p>Is it possible to generalize it to more than two models? Let’s say if we had 5?</p>
<p>Thanks!</p>

username: arunppsg
cooked: <p>Maybe we can use <code>nn.Parameter(5)</code> for 5 weights from 5 models and then tune the parameters with additional constraints on the parameters such that sum of parameters is less than 1.</p>
PyTorch + Golang?

username: aplusm
cooked: <p>I’m aware that PyTorch 1.0 now supports deployment to C++ using tracing, etc. and there’s also ONNX for cross-framework deployments. I’m wondering if there’s any recommendation about running models in a production Golang API?</p>
<p>The only option I could think of would be PyTorch-&gt;ONNX-&gt;TensorFlow to use TF’s Go bindings, which seems like a very indirect path.</p>
<p>Any additional tips would be helpful!</p>

username: smth
cooked: <p>at the moment that’s your best bet. Alternatively writing Go -&gt; PyTorch C++ bindings for the functions you want, but that seems like it’ll be maintenance overhead for you.</p>

username: aplusm
cooked: <p>Thanks Soumith! I hope that Go bindings (and generally more language bindings) might be on the roadmap for future PyTorch! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: rchavezj
cooked: <aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/orktes/go-torch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/606347?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/orktes/go-torch" target="_blank" rel="nofollow noopener">orktes/go-torch</a></h3>

<p>LibTorch (PyTorch) bindings for Golang. Contribute to orktes/go-torch development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: mazzma12
cooked: <p>Hey, do you have any update on it ?<br>
What would be the best way to load a torch model in golang app ?<br>
Thank you</p>
Attention block without copies for contiguity

username: vadimkantorov
cooked: <p>In dinoV2: <a href="https://github.com/facebookresearch/dinov2/blob/main/dinov2/layers/attention.py#L72" class="inline-onebox" rel="noopener nofollow ugc">dinov2/attention.py at main · facebookresearch/dinov2 · GitHub</a></p>
<p>the basic attention goes like this:</p>
<pre><code class="lang-python">B, N, C = x.shape
qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4)
q, k, v = qkv[0] * self.scale, qkv[1], qkv[2]

attn = q @ k.transpose(-2, -1)
</code></pre>
<p>the FlashAttention path goes like this:</p>
<pre><code class="lang-python">B, N, C = x.shape
qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, C // self.num_heads)
q, k, v =  xformers.ops.unbind(qkv, 2)

x = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=attn_bias, op=xformers.ops.fmha.MemoryEfficientAttentionFlashAttentionOp if attn_bias is not None else None)
</code></pre>
<p>Because of permutes / non-contig (notably, the number of heads must move to the batch dimensions), does it force any copies in matmul and in FlashAttention? (especially, the FlashAttention path)</p>
<p>If so, any hints on alleviating this copy? E.g. is it worth replacing the <code>qkv</code> transform by bmm, so that the unbind can go via the 0th dimension?</p>
Padding size and input dimension

username: ado_sar
cooked: <p>I want to perform some “reflection” padding on an input but I am getting the following error:</p>
<p><code>RuntimeError: Argument #4: Padding size should be less than the corresponding input dimension, but got: padding (2, 2) at dimension 2 of input 3</code></p>
<p>This is the code example that produces the above error:</p>
<pre><code class="lang-auto">inp = torch.tensor([[1.0, 2.0], [10.0, 20.0]]).reshape(-1, 2, 2)
m = torch.nn.ReflectionPad2d(2)
m(inp)
</code></pre>

username: sinatv52
cooked: <p>Hi<br>
The error is correct.</p>
<p><code>torch.nn.ReflectionPad2d()</code> reflects the values based on the boundary. You need at least 3*3 to operate reflection padding on with 2 in the argument.<br>
the example in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.ReflectionPad2d.html" rel="noopener nofollow ugc">documents</a> highlights what I say :</p>
<pre><code class="lang-auto">&gt;&gt;&gt; m = nn.ReflectionPad2d(2)
&gt;&gt;&gt; input = torch.arange(9, dtype=torch.float).reshape(1, 1, 3, 3)
&gt;&gt;&gt; input
tensor([[[[0., 1., 2.],
          [3., 4., 5.],
          [6., 7., 8.]]]])
&gt;&gt;&gt; m(input)
tensor([[[[8., 7., 6., 7., 8., 7., 6.],
          [5., 4., 3., 4., 5., 4., 3.],
          [2., 1., 0., 1., 2., 1., 0.],
          [5., 4., 3., 4., 5., 4., 3.],
          [8., 7., 6., 7., 8., 7., 6.],
          [5., 4., 3., 4., 5., 4., 3.],
          [2., 1., 0., 1., 2., 1., 0.]]]])
&gt;&gt;&gt; # using different paddings for different sides
&gt;&gt;&gt; m = nn.ReflectionPad2d((1, 1, 2, 0))
&gt;&gt;&gt; m(input)
tensor([[[[7., 6., 7., 8., 7.],
          [4., 3., 4., 5., 4.],
          [1., 0., 1., 2., 1.],
          [4., 3., 4., 5., 4.],
          [7., 6., 7., 8., 7.]]]])
</code></pre>

username: ado_sar
cooked: <p>I missed the first line on the documentation…Apologize. I should read more carefully.</p>
Extract graph for editing

username: Chame_call
cooked: <p>What I need is:</p>
<ol>
<li>to edit graph for original nn.Module model;</li>
<li>train it</li>
<li>to edit graph again</li>
</ol>
<p>The only way I’ve found to achieve graph editing using pytorch2.0 tools is using custom_backend but the point that it triggers only one time so that my third case in the list above is not achievable.<br>
What are the existing options?</p>

username: marksaroufim
cooked: <p>This tutorial might help <a href="https://colab.research.google.com/drive/1Zh-Uo3TcTH8yYJF-LLo5rjlHVMtqvMdf#scrollTo=MWUvovoo6KYK" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a>, you can move from an aten graph back to an nn module easily with some of this infra</p>

username: Chame_call
cooked: <ol>
<li>is this applicable for aten graph only (not for torch graph)?</li>
<li>if yes for first question than considering some example from the notebook</li>
</ol>
<pre><code class="lang-auto">import torch._dynamo
from torch._functorch.aot_autograd import aot_module_simplified

def toy_backend(gm, sample_inputs): 
    def my_compiler(gm, sample_inputs):
        # &lt;implement your compiler here&gt;
        #place1
        gm.print_readable()
        return gm.forward

    # Invoke AOTAutograd
    return aot_module_simplified(
        gm,
        sample_inputs,
        fw_compiler=my_compiler
    )

torch._dynamo.reset()
fn = torch.compile(backend=toy_backend, dynamic=True)(model)

# triggers compilation of forward graph on the first run
out = fn(input)

# triggers compilation of backward graph on the first run
out.sum().backward()
</code></pre>
<p>we can change graph in the first time in place1 but what about second time?</p>