Random freezes and CUDA errors

username: arc144
cooked: <p>Hi guys, I have a code that I’ve been using for a while but recently I updated my Nvidia driver (I use Linux) to 440xx and I started experiencing random freezes. My training goes well but usually somewhere in the first epoch it hangs forever, some other times it runs 8 epochs than hangs, others it simply freezes the entire computer, others it gives random device assertion error or CUDA errors (different runs different errors)…</p>
<p>I’ve already tried going back to the older driver, updating PyTorch to 1.6, using all different combinations of PyTorch and Cuda (Torch 1.4 + Cuda 10.2 on conda, Torch 1.6 + Cuda 10.2 on conda, Torch 1.6 + Cuda 11 on ArchLinux’s Python) but the issue persists…</p>
<p>Any help is much appreciated. Thanks!</p>

username: ptrblck
cooked: <p>Could you check <code>dmesg</code> for some reported errors/warning which could explain the system freezes?<br>
Also, are you only seeing this behavior in PyTorch or any random CUDA program?</p>

username: arc144
cooked: <p>Sure. It seems to be PyTorch related because the other GPU usages are fine.</p>
<p>My <code>dmesg</code> is full of:</p>
<pre><code class="lang-auto">[  959.195648] pcieport 0000:00:03.0: AER: Multiple Corrected error received: 0000:00:03.0
[  959.195768] pcieport 0000:00:03.0: AER: can't find device of ID0018
[  959.195769] pcieport 0000:00:03.0: AER: Multiple Corrected error received: 0000:00:03.0
[  959.195898] pcieport 0000:00:03.0: AER: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID)
[  959.195902] pcieport 0000:00:03.0: AER:   device [8086:6f08] error status/mask=00001040/00002000
[  959.195903] pcieport 0000:00:03.0: AER:    [ 6] BadTLP                
[  959.195908] pcieport 0000:00:03.0: AER:    [12] Timeout               
[  959.195915] pcieport 0000:00:03.0: AER: Multiple Corrected error received: 0000:00:03.0
[  959.195986] pcieport 0000:00:03.0: AER: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
[  959.195987] pcieport 0000:00:03.0: AER:   device [8086:6f08] error status/mask=00000040/00002000
[  959.195989] pcieport 0000:00:03.0: AER:    [ 6] BadTLP                
[  959.195992] pcieport 0000:00:03.0: AER: Corrected error received: 0000:00:03.0
[  959.195996] pcieport 0000:00:03.0: AER: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
[  959.195998] pcieport 0000:00:03.0: AER:   device [8086:6f08] error status/mask=00000040/00002000
[  959.196000] pcieport 0000:00:03.0: AER:    [ 6] BadTLP                
[  959.196007] pcieport 0000:00:03.0: AER: Corrected error received: 0000:00:03.0
[  959.196148] pcieport 0000:00:03.0: AER: can't find device of ID0018
[  959.196150] pcieport 0000:00:03.0: AER: Multiple Corrected error received: 0000:00:03.0
[  959.196291] pcieport 0000:00:03.0: AER: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
[  959.196293] pcieport 0000:00:03.0: AER:   device [8086:6f08] error status/mask=00000040/00002000
[  959.196294] pcieport 0000:00:03.0: AER:    [ 6] BadTLP                
[  959.196300] pcieport 0000:00:03.0: AER: Multiple Corrected error received: 0000:00:03.0
</code></pre>
<p>-Edit: When it breaks I get a lot of <code>EDAC sbridge: Seeking for: PCI ID 8086:6f6d</code></p>
<p>I also noticed that the training runs well for a while and then when the errors start to appear the time to finish the epoch suddenly increases a lot until eventually the program freezes.</p>

username: ptrblck
cooked: <p>Since a driver update seems to have caused the issues, you could try to update again to the latest one (450+) and check, if it would help. If not, I would recommend to create a post in the <a href="https://forums.developer.nvidia.com/">NVIDIA developer forum</a>.</p>

username: arc144
cooked: <p>I’ve tried this and didn’t work. However, I also did some tests with tensorflow and it also froze. To make sure it’s not an issue with the hardware, I’ve booted in windows (I have dual boot) and did a benchmark with GFXBench. The test ran ok and the results are compatible with other RTX 2080ti (reported by other users). So I believe it’s an issue with Linux, driver or cuda.</p>

username: ptrblck
cooked: <p>Yes, I also think it’s not PyTorch-related, but seems to be caused by the CUDA toolkit, driver, hardware or any interaction between them, which is why I would suggest to create a post for the NVIDIA driver team.</p>