Random reboot during training

username: fdarmon
cooked: <p>Hello,</p>
<p>I am trying to train a model on a Ubuntu 14.04 with a Nvidia 1080 ti, Cuda 8.0 and Pytorch 0.4. I have never been able to train an 200 epochs. The computer reboots after a random time from 1 to 90 epochs.</p>
<p>I have investigated the problem:</p>
<ul>
<li>It is not a PSU max power problem because the computer can run other programs (with pytorch 0.3) with its 4 GPUs</li>
<li>Overheating: I have ran a script that monitor nvidia-smi every 0.05s and when the crash occurs, the temperature of the 4 GPUs is 78°, 51°, 43° and 34° and their power consumption is 90W, 13W, 11W and 9W (when training on GPU 0)</li>
<li>I have also monitored the CPU and GPU memory utilization and everything is stable</li>
<li>I have experienced the crash when training on GPU 0 and GPU 2 so it is not a specific hardware issue.</li>
<li>Similar to <a href="https://discuss.pytorch.org/t/automatically-reboot-when-set-cudnn-benchmark-to-true/2069" class="inline-onebox">Automatically reboot when set cudnn.benchmark to True</a> I set cudnn.benchmark = False and still got the crash.</li>
<li>I was able to complete the training multiple times on another computer with the same hardware (except a smaller PSU) and Ubuntu 16.04 and Cuda 9.1.</li>
</ul>
<p>One possible explanation is the one of <a href="https://github.com/pytorch/pytorch/issues/3022" class="inline-onebox" rel="noopener nofollow ugc">Reliably repeating pytorch system crash/reboot when using imagenet examples · Issue #3022 · pytorch/pytorch · GitHub</a> large power variation causing crashes. I ran my program with a finer image resolution in order to have a constant 100% utilization of the GPU and I got a huge increase in power consumption (still with large power variation) but I experienced no crash. Since the crash appears after a random time up to 6 hours, maybe I have just not waited long enough.</p>
<p>Do you think this is the explanation ? The more Powerful PSU on this computer is in fact more sensible to power variations and shuts off. Do you think it is a software problem with Cuda 8 and pytorch 0.4 ?</p>
<p>Thank you for your help</p>
<p><strong>EDIT</strong>: Add a plot the temperature for a training that crashed</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/b/b508819f671c0040a78255993d5a66392f631ed8.png" data-download-href="https://discuss.pytorch.org/uploads/default/b508819f671c0040a78255993d5a66392f631ed8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b508819f671c0040a78255993d5a66392f631ed8_2_581x500.png" alt="image" data-base62-sha1="pPuE6FNDSeOWE6kIKhoCRsncLKE" width="581" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b508819f671c0040a78255993d5a66392f631ed8_2_581x500.png, https://discuss.pytorch.org/uploads/default/original/2X/b/b508819f671c0040a78255993d5a66392f631ed8.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/b/b508819f671c0040a78255993d5a66392f631ed8.png 2x" data-dominant-color="F6F6F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">640×550 29.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: danieldlongo
cooked: <p>Did you ever solve this problem? I have a similar error where my computer restarts shortly after I start training. I have a 750w psu and only 1 gpu (1080ti) so I don’t think it is a power problem. Also, I did not see an increased wattage going to my gpu before it restarts.</p>

username: fdarmon
cooked: <p>No, I have stopped using that computer and used another one with Ubuntu 16.04 and Cuda 9.1. I never experienced a similar crash with this computer.</p>
<p>I think it was a specific hardware / driver / cuda compatibility issue.</p>

username: Alejandro_Castaneira
cooked: <p>I have a similar problem with a RTX 2080ti using cuda 10.1 and a 1100W power supply, it randomly crash after a couple seconds of training on the GPU.</p>

username: AravindDoss
cooked: <p>Hi,<br>
I have same issues. check this <a href="https://discuss.pytorch.org/t/cpu-is-at-100-even-when-training-is-on-gpu/53309">link</a></p>
<p>Did you find a fix?</p>
<p>Thanks</p>

username: Aldebaran
cooked: <p>Hi,</p>
<p>I have same issue with the same GPU and Cuda.<br>
Did you find a fix?</p>

username: Saurabh_Bagalkar
cooked: <p>Exact same thing! Any solution?</p>

username: omeryildirim
cooked: <p>I was facing similar issues. Even with small batch sizes in Pytorch the computer would reboot by itself. I removed a gpu but still no solution. I did the following and all my problem was solved.</p>
<pre><code>sudo nvidia-smi -pm 1
sudo nvidia-smi -lgc 1400
sudo nvidia-smi -lmc 6500
sudo nvidia-smi -gtt 65
sudo nvidia-smi -cc 1
sudo nvidia-smi -pl 165
</code></pre>
<p>These settings are for the RTX 2080TI. Edit according to your own gpus.</p>
<p>My system:</p>
<ul>
<li>HP Z800 Workstation</li>
<li>Intel(R) Xeon(R) CPU E5-2643 0 @ 3.30GHz</li>
<li>PSU 850W</li>
<li>Ubuntu 20.04</li>
<li>2x RTX 2080TI</li>
<li>NVIDIA-SMI 510.73.05 with CUDA Version: 11.6</li>
</ul>

username: ptrblck
cooked: <p>Since limiting the power limit and clocks seems to help, I would also suggest to check the PSU as it seems to be too weak or faulty.</p>

username: omeryildirim
cooked: <p>Thanks, I gonna check it.</p>