akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why does the inference time suddenly increase in a certain batch size?

jhp: <p>When I ran a model with different batch sizes and I found that the inference time increases linearly with above certain batch size.<br>
Can any bottleneck occur when increasing batch size too much? Don’t think CPU utilization matters since I measure the time only for the inference line. (I might be wrong)<br>
Below is the inference time I measure.</p>
<pre><code class="lang-auto">batchsize:5
0.015910625457763672
batchsize:30
0.015015363693237305
batchsize:50
0.017632007598876953
batchsize:100
0.033460140228271484
batchsize:200
0.06149935722351074
batchsize:400
0.11984658241271973
</code></pre>

ptrblck: <p>I’m not sure what exactly you are measuring and how you are doing it.<br>
If you are trying to profile a GPU operation, you would have to synchronize the code before starting and stopping the timers. If that’s already done, you might see e.g. <a href="https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#dim-quantization">quantization effects</a> but this depends on your actual use case, the used kernels etc.</p>
A neural network with single layer of ConvTranspose2d output different result in 2 evaluations on GPU

y_x: <p>Hi experts,</p>
<p>Here I have a neural network with 1 single layer of nn.ConvTranspose2d. It outputs different results in 2 repeated runs on GPU, but outputs the same results in 2 repeated runs on CPU. I want to understand what is the reason behind this? Do we usually ignore the difference? Thanks.</p>
<pre><code class="lang-auto">ngf=1
nz=1
device=torch.device('cuda')
class Generator_test_3(nn.Module):
    def __init__(self):
        super(Generator_test_3, self).__init__()

        self.main3=nn.Sequential(
            nn.ConvTranspose2d( ngf , ngf, 4, 2, 1, bias=False),
            )
        
    def forward(self, input_noise):


        input=self.main3(input_noise)


        return input
net_test_3 = Generator_test_3().to(device)

torch.manual_seed(0)
noise=torch.randn(1,nz,8,8).to(device=device)
torch.sum((net_test_3(noise)-net_test_3(noise))!=0)

</code></pre>

ptrblck: <p>I get exactly the same results in my setup, but note that you might be running into the expected small errors caused by the limited floating point precision.<br>
Take a look at the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a> to see how deterministic behavior can be achieved.</p>
An error occured when backwards

Loyd: <p>I have move the model and the data to the same device, but I still get  the following errors.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 418, in &lt;module&gt;
    main(args)
  File "train.py", line 298, in main
    train_loss, train_acc1, train_acc5 = train_one_epoch(model, criterion, optimizer, data_loader, device, epoch,
  File "train.py", line 69, in train_one_epoch
    scaler.scale(loss).backward()
  File "/home/lyd/.conda/envs/lyd/lib/python3.8/site-packages/torch/_tensor.py", line 363, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/lyd/.conda/envs/lyd/lib/python3.8/site-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:1! (when checking argument for argument weight in method wrapper__native_batch_norm_backward)
</code></pre>

ptrblck: <p>Could you post a minimal, executable code snippet showing this error, please?</p>