How to debug this backward error?

lxtGH: <p>I meet this error as following, is there some actions is not differentiable or in-place operation cause this error. I want to know how to debug this error.<br>
Traceback (most recent call last):<br>
File “training/train_super_pixel_net.py”, line 370, in <br>
main(args)<br>
File “training/train_super_pixel_net.py”, line 103, in main<br>
avg_loss = train(args, model, optimizer, epoch, trainloader, scheduler, best_iou, loss_fn)<br>
File “training/train_super_pixel_net.py”, line 151, in train<br>
loss.backward()<br>
File “/home/lxt/anaconda3/lib/python3.6/site-packages/torch/tensor.py”, line 93, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/home/lxt/anaconda3/lib/python3.6/site-packages/torch/autograd/<strong>init</strong>.py”, line 90, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
File “/home/lxt/anaconda3/lib/python3.6/site-packages/torch/autograd/function.py”, line 76, in apply<br>
return self._forward_cls.backward(self, *args)<br>
File “/home/lxt/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/_functions.py”, line 30, in backward<br>
return (None,) + ReduceAddCoalesced.apply(ctx.input_device, ctx.num_inputs, *grad_outputs)<br>
File “/home/lxt/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/_functions.py”, line 41, in forward<br>
return comm.reduce_add_coalesced(grads, destination)<br>
File “/home/lxt/anaconda3/lib/python3.6/site-packages/torch/cuda/comm.py”, line 119, in reduce_add_coalesced<br>
flat_tensors = [_flatten_dense_tensors(chunk) for chunk in chunks]<br>
File “/home/lxt/anaconda3/lib/python3.6/site-packages/torch/cuda/comm.py”, line 119, in <br>
flat_tensors = [_flatten_dense_tensors(chunk) for chunk in chunks]<br>
File “/home/lxt/anaconda3/lib/python3.6/site-packages/torch/_utils.py”, line 144, in _flatten_dense_tensors<br>
flat = torch.cat([t.contiguous().view(-1) for t in tensors], dim=0)<br>
RuntimeError: cuda runtime error (77) : an illegal memory access was encountered at /pytorch/aten/src/THC/THCCachingHostAllocator.cpp:271</p>

kuba456: <p>This looks like a bug in cuda code: <a href="https://github.com/NVIDIA/vid2vid/issues/19" rel="nofollow noopener">https://github.com/NVIDIA/vid2vid/issues/19</a><br>
I bet if you run in cpu mode this will work fine.</p>

tom: <p>I would probably first try to establish where the exact location is. Due to the asynchronous nature of GPU computing, the error is usually reported later. <a href="https://lernapparat.de/debug-device-assert/" rel="nofollow noopener">A while ago, I wrote a quick howto for this.</a> There some operations that may cause errors like this when used with invalid data, e.g. indexing functions when you demand an index that is outside the array.</p>
<p>Best regards</p>
<p>Thomas</p>

lxtGH: <p>Hi ! Thanks for you all. It is cuda code problem, it takes me a long time to find the location of the code</p>

NightRainXiaoxiang: <p>I met a similar issue.<br>
Could tell me the reason you found about this issue?<br>
Thanks a lot.</p>

NightRainXiaoxiang: <p>Hi! I have add "os.environ[‘CUDA_LAUNCH_BLOCKING’] = “1"” in my code, but it seems that the  traceback is still random and useless. Do you have any advice to debug such kind of error?<br>
Thanks for any suggestions！</p>

tom: <p>Did you add it at the very top, though?<br>
It needs to be there when PyTorch is first imported.</p>
<p>Best regards</p>
<p>Thomas</p>

NightRainXiaoxiang: <p>Yes, I add it at the very top.</p>

thegopieffect: <p>So this worked for me, basically I downgraded from python 3.7 to 3.6.9 and installed <a href="https://download.pytorch.org/whl/cu100/torch-1.0.0-cp36-cp36m-linux_x86_64.whl" rel="nofollow noopener">pytorch 1.0.0</a> using pip.</p>
<p>Here’s my config,</p>
<ol>
<li>Ubuntu 18.04 LTS</li>
<li>CUDA 10.2</li>
<li>CuDNN 7.6.1</li>
<li>Nvidia TITAN RTX x4</li>
</ol>