GeForce 3090 Hanging

username: 3DTOPO
cooked: <p>I’m experiencing an intermittent issue with my GeForce 3090. The issue was occurring with nightly before 1.7 was released, and with the 1.7 version. It was also occurring with the nvidia driver 455.23 and after updating to 455.32.</p>
<p>For the release version, after installing 455.32, I installed PyTorch with:</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch
</code></pre>
<p>Sometimes the python process hangs after say the ten thousandth iteration or the 5 millionth. The issue occurs regardless the network that I am training (GAN, CNN, etc).</p>
<p>I don’t know that this is a PyTorch issue, but I am hopeful for suggestions on how to investigate what might be causing the hang so that I may report the issue to the appropriately. Any suggestions would be appreciated.</p>

username: ptrblck
cooked: <p>Are you using a single GPU or multiple ones?<br>
Could you kill the process when it hangs and post the stack trace here, please?</p>

username: 3DTOPO
cooked: <p>Hi ptrblck!</p>
<p>Just a single GPU.</p>
<p>I would be happy to post a stack trace, but when I kill the process (control-c) it just quits. Once control-c didn’t quit the process, so I did a control-z to send it to the background and was able to kill it with kill -9. But in either case, no stack trace was shown.</p>
<p>Any suggestions on how to see the stack trace?</p>

username: ptrblck
cooked: <p><code>kill -9</code> wouldn’t show the stack trace.<br>
Could you check for any <code>XIDs</code> in <code>dmesg</code> after the training is killed?</p>

username: 3DTOPO
cooked: <p>So like this?</p>
<pre><code class="lang-auto">dmesg | grep -i xid
</code></pre>
<p>I just ran dmesg and see a number of these type messages:</p>
<pre><code class="lang-auto">[269351.243437] pcieport 0000:00:03.0: AER: Multiple Corrected error received: 0000:00:03.0
[269351.243442] pcieport 0000:00:03.0: AER: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[269351.244286] pcieport 0000:00:03.0: AER:   device [8086:6f08] error status/mask=00000001/00002000
[269351.244710] pcieport 0000:00:03.0: AER:    [ 0] RxErr  
</code></pre>
<p>By the way, thanks for all your contributions to this forum! I must have read 30+ threads with your help when searching for various help.</p>

username: 3DTOPO
cooked: <p>Is there another way besides using kill or control-c so that it shows the stack trace?</p>

username: ptrblck
cooked: <p>You can send the process to the background via <code>CTRL-Z</code> and send a <code>SIGHUP</code> to the main process (if you are using DDP).<br>
Do you see any <code>XID</code> fields with an identifier directly?<br>
The posted output doesn’t show any of these ids.</p>

username: 3DTOPO
cooked: <p>OK thanks! I’m not sure what DDP is, I’ll look into it.</p>
<p>I don’t see any matches for XID. The entire output of dmesg are similar messages that I posted.</p>

username: ptrblck
cooked: <p>DDP is <code>DistributedDataParallel</code>, so a multi-GPU training, but as you’ve already mentioned you are using a single device, so skip this part please.</p>
<p>If the code hangs, are you seeing any GPU utilization in <code>nvidia-smi</code>?</p>

username: 3DTOPO
cooked: <p>Thanks!</p>
<p>I see the process is still allocated, but the utilization is 0%, and I don’t see any CPU activity going on either. Next time it happens I’ll take a screen shot of <code>nvidia-smi</code> and run <code>dmesg</code> looking for xid entries.</p>
<p>Would anything else I can do be insightful?</p>

username: 3DTOPO
cooked: <p>I’ll try a to use the signal method in this SO <a href="https://stackoverflow.com/questions/132058/showing-the-stack-trace-from-a-running-python-application" rel="noopener nofollow ugc">answer</a>.</p>
<p>Might take a while, but when it happens again I’ll post the backtrace here.</p>

username: 5had3z
cooked: <p>I’m currently using nightly 1.8.0 dev20201022+cu110 with my 3090 on 455.23. On the rare occasion I will get some kind of dataloader related crash issue. However I don’t have any halting issues.</p>

username: J3ssi
cooked: <p>I am having sort of the same question.<br>
My Geforce 1080 is hanging between those two print statements:</p>
<pre><code class="lang-auto"> print(f'test 3')
    for batch_idx, (inputs, targets) in enumerate(trainloader):
        print(f'test 2')
</code></pre>
<p>Test 3 is printed. Test 2 not.<br>
After Ctrl+C test 2 is printed and the net is training for a short moment.</p>
<p>Torch version 1.5.0</p>

username: ptrblck
cooked: <p>This issue seems like a multiprocessing issue, if you are using multiple workers.<br>
Try to update to the latest PyTorch version and please create a new topic if it still doesn’t work.</p>

username: 3DTOPO
cooked: <p>I just want to follow up and report that I have not been able to capture a stack trace.</p>
<p>The last two times I’ve had any issue (after like a week of continuous use), instead of just hanging, the host became unresponsive.</p>
<p>I suspect that it is system stability issue related to possibly thermal overload.</p>