Server with dual 3090 GPU crashes

username: danohev
cooked: <p>Hi all!</p>
<p>I have a server with 2 x RTX 3090 GPUs.<br>
Installed Nvidia driver 460, CUDA 11.1, PyTorch nightly (1.8), on Ubuntu 20 and tried running deep learning benchmarks.</p>
<p>The problem is everything runs fine if I use a single GPU.<br>
But the moment when I run both of them, the PC just shuts off.</p>
<ol>
<li>I tried using a stress test that loaded both GPUs 100% utilization and it <strong>worked fine</strong> without crashing.</li>
<li>I tried limiting the power of GPUs to 200W (using ‘sudo nvidia-smi -pl 200’ command), started the pytorch training script and it <strong>crashed</strong> again</li>
</ol>
<p>so I guess it isn’t power supply issue (it’s a SilverStone 1500 watt power supply)</p>
<p>here are the code lines I use for using the 2GPUs:</p>
<p>model = models.resnet152(pretrained=False)<br>
model.conv1 = nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)<br>
model.fc = nn.Linear(2048, 2)</p>
<p>if torch.cuda.device_count() &gt; 1:<br>
model = nn.DataParallel(model)<br>
model.to(device)</p>
<p>for input_images, labels in dataloaders[‘train’]:<br>
# Enable CUDA: use GPUs for model computation<br>
input_images, labels = input_images.to(device), labels.to(device)</p>
<p>don’t know how to proceed with this…<br>
help needed.</p>
<p>thanks!</p>

username: J_Johnson
cooked: <p>Have you tried with CUDA 11.0 and the corresponding drivers?</p>

username: danohev
cooked: <aside class="quote no-group" data-username="danohev" data-post="1" data-topic="110150">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/danohev/48/33917_2.png" class="avatar"> danohev:</div>
<blockquote>
<p>Installed Nvidia driver 460, CUDA 11.1</p>
</blockquote>
</aside>
<p>no, just CUDA 11.1.<br>
I was told thig might still be PSU issue.<br>
even though the stress test with 100% utilization for both 2GPUs passes correctly, when running the pytorch training the GPUs+CPU might have big power spikes which the PSU cant handle (1500Wat, rated ‘80 PLUS Silver’).<br>
I got the recommendation to get a PSU with ‘80 platinum’ or ‘80 titanium’ rating</p>
<p>what do you think?</p>

username: J_Johnson
cooked: <p>Any PSU above 1200W usually requires that the incoming voltage be of a higher rating to achieve the max stated watt rating of the PSU. What is the AC input voltage rating coming into the PSU? (This can vary by country.)</p>

username: J_Johnson
cooked: <p>By way of example, <a href="https://www.newegg.com/corsair-ax1500i-cp-9020057-na-1500w/p/N82E16817139079" class="inline-onebox" rel="noopener nofollow ugc">CORSAIR AXi Series AX1500i Digital 1500W 80 PLUS TITANIUM Haswell Ready Full Modular ATX12V &amp; EPS12V SLI and Crossfire Ready Power Supply with C-Link Monitoring and Control - Newegg.com</a></p>
<p>This PSU is rated 1500W, but if the incoming power supply is 100-115V, it will only provide 1300W.</p>

username: danohev
cooked: <p>we supply 220V AC volatge</p>

username: Ludwig_Friborg
cooked: <p>Any follow up on this? I got a similar situation with pretty much the same rig. The PSU power draw is about 600W when the rig crashes.</p>

username: Hiroyuki_Onishi
cooked: <p>I’m running into the similar symptom.<br>
Did you find any solutions on this?</p>

username: danohev
cooked: <p>yes, changing the PSU as suggested solved the problem.<br>
it think it has ‘80 platinum’ or ‘80 titanium’ rating</p>