RuntimeError: cuda runtime error (59) : device-side assert triggered at /media/nvidia/WD_BLUE_2.5_1TB/pytorch-v1.1.0/aten/src/THC/generic/THCTensorMath.cu:26

hmk: <p>Hi, im building a jetbot with the sparkfun jetson nano 2GB kit. The V01-00 image worked though the training model isn’t working properly. It seems to have issues with the cuda system. These our the errors I have been getting:<br>
1.RuntimeError: CUDA error: device-side assert triggered - at one of the times I tried to run the program.<br>
2. RuntimeError: cuda runtime error (59) : device-side assert triggered at /media/nvidia/WD_BLUE_2.5_1TB/pytorch-v1.1.0/aten/src/THC/generic/THCTensorMath.cu:16.<br>
3.RuntimeError: cuda runtime error (59) : device-side assert triggered at /media/nvidia/WD_BLUE_2.5_1TB/pytorch-v1.1.0/aten/src/THC/generic/THCTensorMath.cu:26</p>
<p><strong>This is my code:</strong><br>
NUM_EPOCHS = 30<br>
BEST_MODEL_PATH = ‘best_model.pth’<br>
best_accuracy = 0.0</p>
<p>optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)<br>
for i, data in enumerate(all_dataloader):</p>
<pre><code>for epoch in range(NUM_EPOCHS):

    for images, labels in iter(train_loader):
        images = images.to(device)
        labels = labels.to(device)
        optimizer.zero_grad()
        outputs = model(images)
        loss = F.cross_entropy(outputs, labels)
        loss.backward()
        optimizer.step()

    test_error_count = 0.0
    for images, labels in iter(test_loader):
        images = images.to(device)
        labels = labels.to(device)
        outputs = model(images)
        test_error_count += float(torch.sum(torch.abs(labels - outputs.argmax(1))))

    test_accuracy = 1.0 - float(test_error_count) / float(len(test_dataset))
    print('%d: %f' % (epoch, test_accuracy))
    if test_accuracy &gt; best_accuracy:
        torch.save(model.state_dict(), BEST_MODEL_PATH)
        best_accuracy = test_accuracy
</code></pre>
<h2><a name="this-is-the-error-1" class="anchor" href="#this-is-the-error-1"></a><strong>This is the error:</strong></h2>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in <br>
13         outputs = model(images)<br>
14         loss = F.cross_entropy(outputs, labels)<br>
—&gt; 15         loss.backward()<br>
16         optimizer.step()<br>
17</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)<br>
105                 products. Defaults to <code>False</code>.<br>
106         “”"<br>
 → 107         torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
108<br>
109     def register_hook(self, hook):</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/autograd/<strong>init</strong>.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)<br>
91     Variable._execution_engine.run_backward(<br>
92         tensors, grad_tensors, retain_graph, create_graph,<br>
—&gt; 93         allow_unreachable=True)  # allow_unreachable flag</p>
<p>RuntimeError: cuda runtime error (59) : device-side assert triggered at /media/nvidia/WD_BLUE_2.5_1TB/pytorch-v1.1.0/aten/src/THC/generic/THCTensorMath.cu:26</p>
<p><strong>picture is attached.</strong></p>
<p>I would appreciate your assistance to proceed in my project. <a class="mention" href="/u/alband">@albanD</a> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed206d9439081e26ab3ddbadde016b832b49409e.png" data-download-href="https://discuss.pytorch.org/uploads/default/ed206d9439081e26ab3ddbadde016b832b49409e" title="pic_!"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed206d9439081e26ab3ddbadde016b832b49409e_2_690x405.png" alt="pic_!" data-base62-sha1="xPIDvCqF9jXN6rxhBrzjuf6YgQC" width="690" height="405" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed206d9439081e26ab3ddbadde016b832b49409e_2_690x405.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed206d9439081e26ab3ddbadde016b832b49409e_2_1035x607.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed206d9439081e26ab3ddbadde016b832b49409e.png 2x" data-dominant-color="F3ECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pic_!</span><span class="informations">1267×744 43.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

albanD: <p>Hi,</p>
<p>You can try to run with CUDA_LAUNCH_BLOCKING=1 to get a more accurate error.<br>
Also can you try running the code on the CPU? It might be a problem with the embedded GPU?</p>