Autograd vague error "returned NULL without setting an error"

username: Tomas_Lysek
cooked: <p>Hi,<br>
I am running into strange error when using autograd:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/user/anaconda3/envs/glowtts/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "/home/user/tts/glow-tts/train.py", line 92, in train_and_eval
    train(rank, epoch, hps, generator, optimizer_g, train_loader, None, None)
  File "/home/user/tts/glow-tts/train.py", line 119, in train
    loss_g.backward()
  File "/home/user/anaconda3/envs/glowtts/lib/python3.8/site-packages/torch/tensor.py", line 245, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/user/anaconda3/envs/glowtts/lib/python3.8/site-packages/torch/autograd/__init__.py", line 145, in backward
    Variable._execution_engine.run_backward(
SystemError: &lt;built-in method run_backward of torch._C._EngineBase object at 0x7f09d7de3840&gt; returned NULL without setting an error
</code></pre>
<p>It is from <a href="https://github.com/jaywalnut310/glow-tts" rel="noopener nofollow ugc">this</a> repository.</p>
<p>I am using CUDA 11.2 with builded pytorch (I have tried to use latest ngc pytorch image - pytorch:20.12-py3 - and it has same error and it has same strange behavior).</p>
<p>After few hours of investigation, i located module which is failing - it is weights in <a href="https://github.com/jaywalnut310/glow-tts/blob/master/modules.py" rel="noopener nofollow ugc">InvConvNear</a></p>
<p>If I give <code>weight = torch.ones_like(weight).cuda()</code> on line 224 it is not throwing exception (but obviously it’s not learning anything).</p>
<p>Do you have any idea how to debug this error or solve this problem?</p>
<p>Thanks!</p>

username: Tomas_Lysek
cooked: <p>After I set requires requires_grad to False on weight, It is working (but obviously it is learning wrong). Is there any way to get normal error log from pytorch? I realy don’t know how to interpret “returned NULL without seting an error”</p>

username: albanD
cooked: <p>Hi,</p>
<p>This is the first time I see this error message :o This hasn’t happened in a while! haha</p>
<p>Can you re-run the program  after setting <code>TORCH_SHOW_CPP_STACKTRACES=1</code> please? And share the c++ stacktraces that it will print when it errors out.</p>
<p>Also do you have an easy way for us to reproduce this (like a simple colab notebook for example)?</p>

username: ptrblck
cooked: <p>Also, feel free to ping me here in case you think the issue might be caused by CUDA11.2 (or any other GPU lib).</p>

username: HeyangQin
cooked: <p>I got this error when trying to run <a href="https://github.com/petuum/adaptdl/tree/master/examples/pytorch-cifar" rel="noopener nofollow ugc">https://github.com/petuum/adaptdl/tree/master/examples/pytorch-cifar</a>. This seems to be a randomly happening error. I tried different versions of Pytorch (1.6, 1.7, 1.7.1) but the error is still here. In the end I got around it by setting <code>CUDA_VISIBLE_DEVICES="0"</code> to disable DDP.</p>

username: NoahDrisort
cooked: <p>I have the same error, in my case:</p>
<ul>
<li>CUDA 11.0 on A100</li>
<li>Conda environment with</li>
</ul>
<pre><code class="lang-auto">python -m pip install torch==1.7.0+cu110 torchvision==0.8.1+cu110 torchaudio==0.7.0
</code></pre>
<ul>
<li>Apex using the lastest from repo</li>
<li>I have tried all python version 3.6,7,8<br>
=&gt; Are there anyone have solved this problem <img src="https://discuss.pytorch.org/images/emoji/apple/hot_face.png?v=9" title=":hot_face:" class="emoji" alt=":hot_face:"> , I test with many pytorch from source but nothing change, I think this is from Cuda11.0 because I have tried my source code successfully on V100 Cuda10.2</li>
</ul>

username: NoahDrisort
cooked: <p>I have fixed this by install pytorch from source and using suitable python version (my case is python = 3.6.10)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/b/db9ca30b1c0c754799edc6a25263209840ec34e0.png" alt="image" data-base62-sha1="vkM9Rfq9hzF7jMdpzveNeeUNYnC" width="625" height="58"></p>

username: ptrblck
cooked: <p>That’s good to hear, as more recent releases ship with the latest bug fixes.<br>
Note that <code>apex.amp</code> is deprecated (in case you are using this utility from <code>apex</code>) and you should use the native implementation via <code>torch.cuda.amp</code>.</p>

username: amsword
cooked: <p>A general question: recently I’m using apex.amp O2 to save memory with large batch size for speed-up without accuracy drop. However, the native torch.cuda.amp does not save memory. Is there any alternative of apex.amp O2 if apex.amp is deprecated?</p>

username: ptrblck
cooked: <p>No, not yet as we are investigating how the legacy “O2-style” amp could work in the native implementation.</p>

username: amsword
cooked: <aside class="quote no-group" data-username="Tomas_Lysek" data-post="1" data-topic="112781">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tomas_lysek/48/30315_2.png" class="avatar"> Tomas_Lysek:</div>
<blockquote>
<p><code>method run_backward of torch._C._EngineBase</code></p>
</blockquote>
</aside>
<p>Recently, I also encountered such issues, but it seems like eventually, it is OOM issue. By reducing the batch size, it starts working as normal.</p>