akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Huge difference in gradient performing the back propagation sample by sample or with the whole batch

Torcione: <p>Hi,<br>
I noticed that working with a “big” network- as VGG16 there is a big difference in the gradient’s magnitude that I get performing the back-propagation sample by sample and the one that I get calculating it on the whole batch in a single time.</p>
<p>I don’t expect this difference since I’m using the cross entropy loss function with the default <code>reduction=‘mean’</code> option. Also if the option would be set on sum, since I used a batch size of 30 I would have expected a difference of magnitude of this order, while the observed factor gap in the 2 cases is ~100000.</p>
<p>Also taking a batch of 2 element shows a huge difference with respect to a batch of a single element.</p>
<p>Why do I observe such a gap in the 2 cases? Could it be related to the presence of <code>nn.BatchNorm2d</code>?</p>

tom: <p>Personally, I would expect that with <code>reduction='mean'</code> you would have to take the mean over the gradients you get from individual backpropagation.<br>
If you use VGG, I’m assuming this is VGG without batch norm?</p>
<p>Best regards</p>
<p>Thomas</p>

Torcione: <p>Hi Tom, thank you for your comment! I also was expecting this.<br>
My VGG has actually the batch norm in his architecture, I think that this would be the issue.<br>
One question: Does BatchNorm gives good result with very small batch size (different from 1), like <code>batch_size=2</code>?</p>

tom: <blockquote>
<p>Does BatchNorm gives good result with very small batch size (different from 1), like <code>batch_size=2</code></p>
</blockquote>
<p>No. BatchNorm needs rather large batches because it essentially assumes that the current batch statistics are a reasonable estimate for the overall dataset statistics.<br>
People have tried to mitigate this by modifying BN training mode into “update the statistics, then use the statistics as in eval mode” but this is quite finicky, in particular at the beginning.<br>
This is why everyone is glad that transformers use layernorm instead. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Best regards</p>
<p>Thomas</p>
Pytorch model inference is slower when running inside Django server

vivekstorm: <p>I have a custom Pytorch vision model which takes &lt;4 seconds for performing inference on 5 images on CPU when calling as a standalone python program, but when the same inference method is called from Django server on CPU, the time for getting output is ~200 seconds. How can I solve this issue? I tried increasing number of threads from 4 to 16, still it doesn’t work.</p>
Torch.cuda.device_count() always return 0

hgy1025: <p>Hello, There are 0 to 4 gpu in the server, and one of them or multi-use is intended.<br>
so, I tried to use gpu using os.environ[“CUDA_VISIBLE_DEVICES”]=‘3’ or ‘0, 1, 2, 3’<br>
}. However, torch.cuda.current_device() continues to return 0. Why is it like this?</p>
<p>How do I solve this problem?<br>
thank you</p>
<pre><code class="lang-auto">import os
import torch

os.environ["CUDdfA_DEVICE_ORDER"]="PCI_BUS_ID" 
os.environ["CUDA_VISIBLE_DEVICES"]= "3" 
print('Current cuda device:', torch.cuda.current_device())
print('Count of using GPUs:', torch.cuda.device_count())


################print######################
Current cuda device: 0
Count of using GPUs: 1
</code></pre>

albanD: <p>Hi,</p>
<p>This is expected. The cuda devices are always numbered from 0 and up.<br>
So if you have 1 GPU available, it will be <code>0</code>. If you have 2, they will be 0 and 1. etc</p>