Multi GPU (2080 ti) training crashes PC

username: mstrfx
cooked: <p>My build:<br>
Asrock z390 extreme4<br>
Intel 8700k<br>
x2 2080 ti x2<br>
Cooler Master v1200 Platinum</p>
<p>Ubuntu 18.04</p>
<p>Cuda 10.0<br>
nccl 2.4.0-2<br>
pytorch was installed according to guide on <a href="http://pytorch.org" rel="nofollow noopener">pytorch.org</a></p>
<p>So I’ve got something interesting: pc crashes right after I try running imagenet script for multi gpu from official pytorch repository. It doesn’t crash pc if I start training with apex mixed precision. Training on a single 2080 also didn’t cause reboot.</p>
<p>What didn’t work:</p>
<ol>
<li>decreasing batch size</li>
<li>limiting power consumption of gpu’s via nvidia-smi</li>
<li>changing motherboard, cpu, power supply</li>
<li>changing 2080 ti vendor</li>
</ol>
<p>For some reason everything worked after I switched both 2080 ti’s with 1080 ti’s. So it seems pytorch (or some nvidia software) isn’t fully compatible with multiple 2080 ti’s? Has anyone encountered this?</p>

username: ptrblck
cooked: <p>Two 2080TIs should work, so I think it might be a hardware issue.<br>
However, it seems you’ve already changed a lot of parts of your system.<br>
Regarding point 3 and 4, it seems you completely rebuilt your system.</p>
<p>Does your code only crash using the ImageNet example or also a very small model, e.g. a single linear layer and <code>DataParallel</code>?</p>

username: hay
cooked: <p>I can corroborate. It happens with me too. torch distributed data parallel fails on 2080Ti with &gt;1 gpu, however works well with titan-x, titan-xp or 1080Ti.</p>

username: AljoSt
cooked: <p>Same here, fails on two 2080Ti, works with two 1080Ti. In the script, I just load a <code>resnet50</code> from <code>torchvision.models</code> and do inference on it. It does not crash however, if I run the same script twice at the same time, each using one GPU. So it does not seem to a power supply issue</p>

username: AljoSt
cooked: <p>Okay, so now I have done some runs with the following code:</p>
<pre><code class="lang-auto">import os

from tqdm import tqdm
from torchvision.models import resnet50, resnet18
from torch.utils.data import DataLoader
from torch.utils.data import Dataset
import torch
import torch.nn as nn

n = 1000000

class RandomDs(Dataset):
    def __init__(self, ):
        pass

    def __len__(self):
        return n

    def __getitem__(self, index):
        return torch.rand(3, 512, 256)


if __name__ == '__main__':

    dataset = RandomDs()
    data_loader = DataLoader(dataset, batch_size=128, shuffle=False, num_workers=4, pin_memory=False)


    model = resnet50()

    # os.environ["CUDA_VISIBLE_DEVICES"] = "0"
    device = torch.device('cuda:0')
    model = nn.DataParallel(model)
    
    model.to(device)

    with torch.no_grad():
        for batch in tqdm(data_loader):
            batch = batch.to(device)
            model(batch)

</code></pre>
<p>I changed the <code>input shape to the network</code>, <code>batch size</code>, <code>number of workers</code>, <code>model type</code>, <code>torch version</code>, and whether it runs with <code>DataParallel</code> or as two scripts (so two scripts, running at the same time, so that both GPUs are in use). These are the results:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>input shape</th>
<th>batch_size</th>
<th>num_workers</th>
<th>model</th>
<th>parallel</th>
<th>torch version</th>
<th>crash</th>
<th>memory (GB each)</th>
</tr>
</thead>
<tbody>
<tr>
<td>3, 256, 256</td>
<td>128</td>
<td>4</td>
<td>resnet50</td>
<td>yes</td>
<td>1.2.0</td>
<td>No</td>
<td>~2.3</td>
</tr>
<tr>
<td>3, 512, 256</td>
<td>128</td>
<td>4</td>
<td>resnet50</td>
<td>yes</td>
<td>1.2.0</td>
<td>Yes</td>
<td>~3.3</td>
</tr>
<tr>
<td>3, 256, 256</td>
<td>256</td>
<td>4</td>
<td>resnet50</td>
<td>yes</td>
<td>1.2.0</td>
<td>Yes</td>
<td>~3.3</td>
</tr>
<tr>
<td>3, 256, 256</td>
<td>256</td>
<td>4</td>
<td>resnet50</td>
<td>two scripts</td>
<td>1.2.0</td>
<td>No</td>
<td>~5.3 (single)</td>
</tr>
<tr>
<td>3, 256, 256</td>
<td>256</td>
<td>16</td>
<td>resnet50</td>
<td>yes</td>
<td>1.2.0</td>
<td>Yes</td>
<td>~3.3</td>
</tr>
<tr>
<td>3, 256, 256</td>
<td>256</td>
<td>4</td>
<td>resnet18</td>
<td>yes</td>
<td>1.2.0</td>
<td>Yes</td>
<td>~2.2</td>
</tr>
<tr>
<td>3, 512, 256</td>
<td>128</td>
<td>4</td>
<td>resnet50</td>
<td>yes</td>
<td>1.4.0</td>
<td>Yes</td>
<td>~3.3</td>
</tr>
</tbody>
</table>
</div><p>It seems to be memory related, but given that I am using 2*2080Ti and I have 64GB ram, it’s not an OOM. Any ideas what else I can test?</p>
<p>I am using cuda 10.2 and tried pytroch version 1.2 and 1.4</p>

username: ptrblck
cooked: <p>Does your machine restart or how does the crash look for you?<br>
If that’s the case, was PyTorch (or other libs) working before or is it a new setup?<br>
Did you run some stress tests on the devices?</p>

username: AljoSt
cooked: <p>The machine shuts down immediately and then restarts. It is more or less a new system: It has been used for other tasks before, but not for ML. I’ve run <a href="http://wili.cc/blog/gpu-burn.html" rel="nofollow noopener">GPU-burn</a> for an hour with Double precision and tensor cores enabled and it seemed to work fine.</p>
<p>The only times when it doesn’t crash is when I am using a small input (shape/bs) or when I’m not using <code>DataParallel</code>. Could this be related to the communication between the GPUs?</p>

username: ptrblck
cooked: <p>You could run e.g. <a href="https://github.com/NVIDIA/nccl-tests">nccl-test</a> to check the communication.</p>

username: AljoSt
cooked: <p>Seems fine:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf9c4c8d4cb09f9166f45a55a46312e16be8a9bc.png" data-download-href="https://discuss.pytorch.org/uploads/default/bf9c4c8d4cb09f9166f45a55a46312e16be8a9bc" title="Screenshot from 2020-04-21 13-45-54"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf9c4c8d4cb09f9166f45a55a46312e16be8a9bc_2_690x469.png" alt="Screenshot from 2020-04-21 13-45-54" data-base62-sha1="rl445EaG2uR7lCXBy5O3S2aG6ra" width="690" height="469" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf9c4c8d4cb09f9166f45a55a46312e16be8a9bc_2_690x469.png, https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf9c4c8d4cb09f9166f45a55a46312e16be8a9bc.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf9c4c8d4cb09f9166f45a55a46312e16be8a9bc.png 2x" data-dominant-color="0C0C0C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-04-21 13-45-54</span><span class="informations">957×651 101 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m really out of ideas on what to test. The only way I can replicate the crashes or find any abnormalities is by using PyTorch DataParallel. GPU burn runs fine, nccl-test runs fine, utilizing both GPUs at the same time individually is fine. What else could I do in order to identify the cause of the problem?</p>

username: pritamdamania87
cooked: <p>Can you check system logs to see if there is something there mentioning why the machine restarted? Usually on Linux there might be some information in <code>/var/log/messages</code>.</p>
<p>Another option might be to post on Nvidia forums to check if there might be some system issue.</p>

username: ptrblck
cooked: <p>That’s another good idea.<br>
<a class="mention" href="/u/aljost">@AljoSt</a> try to grab for <code>Xid</code> and post the result here, please.</p>

username: AljoSt
cooked: <p>Could not find anything interesting in kernlog, syslog or dmesg. But what I did find was this <a href="https://github.com/pytorch/pytorch/issues/3022" rel="nofollow noopener">github issue</a> where people have a similar problem. GPU burn etc cannot make the system crash and PSU is powerful enough (on paper). They say, that PyTorch causes a short big power surge which can cause the PSU to fail, even if it’s big enough to support the system on heavy load in other circumstances (gpu burn etc).</p>
<p>I put the 2 2080Tis into a system with a 1600w PSU and they seemed to work fine. The PSU in the other system had 850w. Obv. I cannot guarantee that this is the cause/solution due to many variables being changed but as other people have had similar experiences, I will take it for now (and will post again here if there are new developments). Thanks for your help!</p>
<p>(and btw, since many people have the same problem which seems to originate in PyTorch, I think it would be good if the relevant developers could have a look into that)</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="AljoSt" data-post="12" data-topic="52032">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aljost/48/23042_2.png" class="avatar"> AljoSt:</div>
<blockquote>
<p>(and btw, since many people have the same problem which seems to originate in PyTorch, I think it would be good if the relevant developers could have a look into that)</p>
</blockquote>
</aside>
<p>Since a 2080Ti can take &gt;300W, a 850W PSU might not be enough for the system.<br>
I’m not sure, if artificially limiting the GPU is a valid use case.</p>

username: AljoSt
cooked: <p>Ah interesting, I didn’t know that, thank you. I have difficulties finding the max power draw (the highest I saw in some random blog post was around 330w). Is it vendor specific (so would it differ say between MSI 2080Ti and EVGA 2080TI?) or is there a “fixed” number?</p>

username: ptrblck
cooked: <p>It should be vendor specific, as some vendors might increase the clock speeds and add a better cooling solution to the device. You won’t see much difference, but it’s not strictly a single number.</p>

username: AljoSt
cooked: <p>Thank you for your help!</p>

username: mstrfx
cooked: <p>hi guys <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>Just wanted to let you know it is not related to only pytorch framework. Happens also with tensorflow.</p>
<p>It feels like modern power supplies can’t handle 300w + 300w simultaneously during training. I got double 2080 ti with another power supply (with cheap 2$ synchronizer) and everything works like a charm.</p>

username: VincentWang
cooked: <p>Hi mstrfx, may I ask which PSU did you buy? I am facing the same issue with a 1200 watt PSU.</p>

username: VincentWang
cooked: <p>It must be the PSU…</p>
<p>So I have 2 2080Ti GPU (from EVGA) and CORSAIR - HX Series <strong>1200W</strong> ATX12V 2.4/EPS12V 2.92 80 Plus Platinum Modular Power Supply. It shuts down the PC sometimes when I run 2 gpu for training (separately or together) and it usually happens when one model reaches the epoch of that fold.</p>
<p>Now I switch to EVGA - GP Series SuperNOVA <strong>1000W</strong> ATX 80 Plus Gold Fully Modular Power Supply. It works well without any PC shuts down. I made the switch because my friend has the same setting as mine and he used this PSU. It turns out that it is indeed the case but it is still surprised to see 1200W cannot work but 1000W can.</p>
<p>Not sure if it was a defect for the 1200W PSU though.</p>

username: 3co
cooked: <p>Been trying to run 2 RTX 3090’s on a brand new Corsair 1500W PSU… with many sudden shutdowns.  I’m thinking there’s no way I don’t have enough watts, right?</p>
<p>I was worried it could be a temperature issue, so was obsessively logging temps for all devices, and running fans at max speed, right up to shutdown. GPUs were a “cool” ~60 C – not enough to warrant causing the shutdown.</p>
<p>What folks say above about potentially a single PSU “not being able to handle both 300w + 300w during training” due to sudden bursts of power then makes a lot of sense.</p>
<p>Thanks to <a class="mention" href="/u/aljost">@AljoSt</a> pointing to the <a href="https://github.com/pytorch/pytorch/issues/3022" rel="noopener nofollow ugc">relevant github issue</a>, where at the end @<a href="https://github.com/lfcnassif" class="inline-onebox" rel="noopener nofollow ugc">lfcnassif (Luis Filipe Nassif) · GitHub</a> says that his team did experiments, and found that what successfully worked was to limit clock speeds of both GPUs.</p>
<p>The trick was simply running:<br>
<code>sudo nvidia-smi -lgc 1395,1740</code></p>
<p>(where the -lgc sets min, max clock speeds)</p>
<p>It turns out that max clock speed is not much lower than the ~1900 MHz that the GPU would naturally run at, but basically I think we’re just preventing sudden synchronous large bursts of computational hunger…</p>
<p>And after training a run for a long time now, I’m posting “confirmation”, this works. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>