akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
In-place operations in forward method

username: silvester
cooked: <p>I have the following in the forward method:</p>
<pre><code class="lang-auto">
def forward(self, x, member_id=None):
     model_output = F.softmax(self.models(x), dim=1)
     model_output[:, 2: 4] = (torch.min(torch.cat((model_output[:, :2], model_output[:, 5:]), dim=1), dim=1)[0].repeat(2,1).permute(1,0)

     return model_output
</code></pre>
<p>When I apply <code>loss.backward()</code>, the following error pops up:</p>
<p><code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 10]], which is output 0 of SoftmaxBackward, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</code></p>
<p><em><strong>There is no error without <code>F.softmax</code>.</strong></em></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="silvester" data-post="1" data-topic="91441">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/6bbea6/48.png" class="avatar"> silvester:</div>
<blockquote>
<p>There is no error without <code>F.softmax</code> .</p>
</blockquote>
</aside>
<p>This would mean that <code>F.softmax</code> needs its output to calculate the gradients (as also given in the definition in <a href="https://github.com/pytorch/pytorch/blob/bfa94487b968ccb570ef8cd9547029b967e76ed0/tools/autograd/derivatives.yaml#L1258">derivatives.yaml</a>). This would disallow all inplace manipulations in the model output.</p>
Transfer learning on 3d data

username: lycaenidae
cooked: <p>Hi,</p>
<p>I am wondering can I use pre-trained models (2d) like that order?</p>
<p>Pretrainedmodel- 3dconv-3dconv etc</p>

username: ptrblck
cooked: <p><code>nn.Conv3d</code> expects an input in the shape <code>[batch_size, channels, depth, height, width]</code>, so you would need to reshape the output of the pretrained model to this shape.<br>
Depending which model you are using, it might already provide the desired shape or you would need to figure out how this should be done.</p>
<p>E.g. if you are using a model for a multi-class classification, such as Resnet50, your output would have the shape <code>[batch_size, nb_classes]</code> and would contain the logits for each class.<br>
While you might be able to reshape this output to a 5-dim tensor, it might not yield a good performance, as the <code>nb_classes</code> dimension containing the logits wouldn’t have any volumetric information.<br>
In that case you could remove the last classification layer and use the output features, which would have the shape <code>[batch_size, channels, height, width]</code>. While the <code>depth</code> dimension would still be missing, I assume you have an idea how to create it.</p>
Slow training with nn.DataParallel

username: samra-irshad
cooked: <p>I am training an image segmentation model and the time taken per each step in epoch seems to be rising throughput whole epoch. I am not sure if it has something to do with the way I am feeding my model with images. I am using DataLoader utility. I have pasted time for one epoch. Is there anyway to improve the training speed?</p>
<pre><code class="lang-auto">Epoch: 0 [&gt;              ] 1.9% Mean Dice: 0.0227, time: 10.9979s
[Training] Epoch: 0 [&gt;              ] 3.8%  time: 12.0363s
[Training] Epoch: 0 [&gt;              ] 5.7%  time: 13.0289s
[Training] Epoch: 0 [=&gt;             ] 7.5%  time: 14.0112s
[Training] Epoch: 0 [=&gt;             ] 9.4%  time: 15.8508s
[Training] Epoch: 0 [=&gt;             ] 11.3% time: 16.8213s
[Training] Epoch: 0 [=&gt;             ] 13.2%  time: 17.7810s
[Training] Epoch: 0 [==&gt;            ] 15.1%  time: 18.7687s
[Training] Epoch: 0 [==&gt;            ] 17.0%  time: 19.7607s
.
.
.

[Training] Epoch: 0 [==============&gt;] 96.2% time: 60.4288s
[Training] Epoch: 0 [==============&gt;] 98.1% time: 61.3712s
</code></pre>

username: ptrblck
cooked: <p>Are you only seeing this slowdown using <code>nn.DataParallel</code> or also a single GPU?<br>
In the latter case, are you also seeing an increased memory usage?<br>
If that’s the case, you might accidentally store some tensors which are still attached to the computation graph, or somehow extend the computation graph in each iteration.</p>

username: samra-irshad
cooked: <p>Thanks I was saving some tensors in loop.</p>
Not being able to use PyTorch properly on Jetson Nano

username: Nitish_Agrawal
cooked: <p>I am trying to install PyTorch for using YOLOv5 on my Jetson Nano but i am unable to do so. Can anyone please help me with the same? Basically i want to run YOLOv5 on my Jetson Nano</p>

username: ptrblck
cooked: <p>You can find the Jetson binaries <a href="https://github.com/pytorch/pytorch#nvidia-jetson-platforms">here</a>. I’m not familiar with the YOLOv5 implementation, but if you managed to run it on in another setup, I assume it should also work on the Nano assuming you are not running out of memory.</p>

username: Nitish_Agrawal
cooked: <p>Hello sir,</p>
<p>Thank you for your reply.</p>
<p>I am using a 128 GB SD card for the Jetson Nano so I believe it is not the issue with memory. It was working for me on my Windows 10 PC without a GPU and even on my Linux PC with a GPU. I do not know why is it not working on Jetson Nano. It gives a lot of errors, the last error being an AttributeError. I have formatted the SD card and I am going to do the installations again so I would really appreciate your help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Thanking you in advance for your help.</p>
<p>Best Regards,<br>
Nitish.</p>
RuntimeError: The size of tensor a (40) must match the size of tensor b (64) at non-singleton dimension 1

username: brenth
cooked: <p>Hi all,</p>
<p>Am attempting to implement MMD from <a href="https://github.com/ZhaoZhibin/UDTL/blob/45c256f69c7f0bf06d8b9505834edf991d4484d4/loss/DAN.py" rel="noopener nofollow ugc">https://github.com/ZhaoZhibin/UDTL/blob/45c256f69c7f0bf06d8b9505834edf991d4484d4/loss/DAN.py</a><br>
but have encountered the error of adding/subtracting tensors of different sizes.</p>
<p>This is the calling function :</p>
<blockquote>
<p>distance_loss=DAN(features.narrow(0, 0, labels.size(0)),features.narrow(0, labels.size(0), inputs.size(0)-labels.size(0)))</p>
</blockquote>
<p>The first argument is meant to extract features from source while the second argument refers to the features from the target source,</p>
<p>And this is the function (from the link) for the operations</p>
<blockquote>
<p>def DAN(source, target, kernel_mul=2.0, kernel_num=5, fix_sigma=None):<br>
batch_size = int(source.size()[0])<br>
kernels = gaussian_kernel(source, target,<br>
kernel_mul=kernel_mul, kernel_num=kernel_num, fix_sigma=fix_sigma)<br>
XX = kernels[:batch_size, :batch_size]<br>
YY = kernels[batch_size:, batch_size:]<br>
XY = kernels[:batch_size, batch_size:]<br>
YX = kernels[batch_size:, :batch_size]<br>
loss = torch.mean(XX + YY - XY - YX)<br>
return loss</p>
</blockquote>
<p>Sizes:<br>
‘features’  -  [104,128,125] per batch<br>
‘labels’  - [64] per batch</p>
<p>I believe the problem to the error code is in the loss statement where loss= XX+YY-XY-YX.</p>
<p>Appreciate any advice on this. Thanks</p>

username: ptrblck
cooked: <p>Could you check the shape of <code>kernels</code> and the value of <code>batch_size</code>?<br>
If I’m not mistaken this code would only work, if the shapes are <code>batch_size*2</code> as seen here:</p>
<pre><code class="lang-python"># works
kernels = torch.randn(128, 128)
batch_size = 64

XX = kernels[:batch_size, :batch_size]
YY = kernels[batch_size:, batch_size:]
XY = kernels[:batch_size, batch_size:]
YX = kernels[batch_size:, :batch_size]
loss = torch.mean(XX + YY - XY - YX)

# fails
kernels = torch.randn(128, 127)
batch_size = 64

XX = kernels[:batch_size, :batch_size]
YY = kernels[batch_size:, batch_size:]
XY = kernels[:batch_size, batch_size:]
YX = kernels[batch_size:, :batch_size]
loss = torch.mean(XX + YY - XY - YX)
</code></pre>
Some confusions about torch.optim.Adam().step()'s principle

username: credoxfromcqut
cooked: <p>when i implemented a neural network to fitting a function of some datas with numpy and i noticed that its performance is so bad and even with no any use .i found its gradient is so big that the parameters W go to minus-large number immediatly. obviously, it didnt work for the test data.in that program,i use params-=grad(compute by  Chain derivative)*learningrate.when i implement the same network with pytorch,i found it works well than numpy-version. i start to find the diffrence between the two version.i find the rule of  torch.optim.Adam().step() dont work as my thoughts,it update the weights(params) by params-=(the sign of grad(+or-)<em>lr,not the grad</em>learningrate,why is this.is that pytorch will do some Optimization of processing to help training my model?and i didnt see some Similar operation in  torch.optim.Adam().step()'s sorce code from pytorch’s Document.hope someone can help me expain this.thanks in advance.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/2/92c934aae7748f2a4a71e03ae229de1e2a6f2446.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/92c934aae7748f2a4a71e03ae229de1e2a6f2446" title="QQ截图20200802002238"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92c934aae7748f2a4a71e03ae229de1e2a6f2446_2_236x500.jpeg" alt="QQ截图20200802002238" data-base62-sha1="kWwN6mZG250DQqP4h0Fdx11dijs" width="236" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92c934aae7748f2a4a71e03ae229de1e2a6f2446_2_236x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92c934aae7748f2a4a71e03ae229de1e2a6f2446_2_354x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92c934aae7748f2a4a71e03ae229de1e2a6f2446_2_472x1000.jpeg 2x" data-dominant-color="F1F2F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">QQ截图20200802002238</span><span class="informations">620×1312 118 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: credoxfromcqut
cooked: <p>sry   i forgot the optimizer i choose is Adam…not SGD</p>