Device-side assert triggered error on AWS EC2 instance

FlorianR: <p>Hi,</p>
<p>I try to train my model using PyTorch and a AWS EC2 Instance (g3s.xlarge), but there is a common error, the famous <strong>RuntimeError: CUDA error: device-side assert triggered</strong></p>
<p>So have a more clear stacktrace, I run my sscript using CUDA_LAUNCH_BLOCKING=1, and there is all the stacktrace I have:</p>
<pre><code class="lang-auto">  /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda [](int)-&gt;auto::operator()(int)-&gt;auto: block: [0,0,0], thread: [44,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda [](int)-&gt;auto::operator()(int)-&gt;auto: block: [0,0,0], thread: [26,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCCachingHostAllocator.cpp line=265 error=59 : device-side assert triggered
Traceback (most recent call last):
  File "train.py", line 367, in &lt;module&gt;
    backup_every=args.backup_every)
  File "train.py", line 267, in train
    if torch.isnan(loss):
RuntimeError: CUDA error: device-side assert triggered
</code></pre>
<p>This error came during the first epoch, or during the test of the first epoch (it depends)</p>
<p>I check my dataset and:</p>
<ul>
<li>
<ul>
<li>There is no negative number</li>
</ul>
</li>
<li>
<ul>
<li>All the output files are in Yolo format</li>
</ul>
</li>
<li>
<ul>
<li>There is exactly the same number of files and labels</li>
</ul>
</li>
</ul>
<p>The number of classes I have is 17 (so I use 17 to generate my cfg, and I enter 17 in my .data file)<br>
I can’t identify where is the mistake, do you have any idea ?</p>
<p>Thanks in advance and have a good day !</p>
<p>Kind regards,<br>
Florian</p>

ptrblck: <p>Is your code working fine on the CPU?<br>
If the error is raised in the first epoch, if might not take too long to run it on the CPU.<br>
Based on the stack trace I’m not sure, where the failing indexing operation is used.</p>

FlorianR: <p>This code already works in the past with another dataset on GPU (I already have this error message but was my fault with error in number of classes)</p>
<p>This time I can’t identify the problem.</p>
<p>When i run the same command on CPU (just change the batch_size, and number of epochs), I have the following error:</p>
<pre><code class="lang-auto">Using CPU

Traceback (most recent call last):
  File "train.py", line 367, in &lt;module&gt;
    backup_every=args.backup_every)
  File "train.py", line 198, in train
    load_darknet_weights(model, f'weights/{cutoff_name}')
  File "/home/florian/dev/utils/models.py", line 278, in load_darknet_weights
    conv_w = torch.from_numpy(weights[ptr:ptr + num_w]).view_as(conv_layer.weight)
RuntimeError: shape '[512, 256, 3, 3]' is invalid for input of size 776331
</code></pre>

ptrblck: <p>If changing the batch size yields this error, most likely you are using a wrong <code>view</code> or <code>reshape</code> operation.<br>
Could you post the model definition and also try to make sure to keep the batch dimension constant e.g. via <code>x.view(x.size(0), -1)</code> instead of <code>x.view(-1, some_features)</code>.</p>

FlorianR: <p>My model is Darknet / Yolo Tiny model, I didn’t change the model definition, but you can find the model definition here: <a href="https://paste.ofcode.org/YPZftRpwZFsNydkuCfNZvc" rel="nofollow noopener">https://paste.ofcode.org/YPZftRpwZFsNydkuCfNZvc</a></p>
<p>I check the code, and I think i keep the batch dimension constant all the time</p>

FlorianR: <p>To update this post:</p>
<ul>
<li>
<p>The same code works with epochs = 9, batch_size = 16 and not using the Yolo weights (from scratch) and with less data (around 800 images instead of 10k)</p>
</li>
<li>
<p>But this code doesn’t work with epoch = 9, batch_size = 16 and not using the Yolo weight, if I keep all the dataset (it return the same CUDA error)</p>
</li>
</ul>
<p>It’s very weird …</p>