Machine reboot when running model in torch.nn.DataParallel

username: kirk86
cooked: <p>Hi folks, I’ve noticed that whenever I run multi-gpu training on my machine it suddenly reboots. There are 2 gpus on the machine a 1080Ti and a Titan X. 1080Ti is gpu:0  and is also used to power 2 monitors for video display. Usually never had issues before with multi-gpu training using tensorflow. Any ideas what might be causing this?</p>

username: ptrblck
cooked: <p>This sounds like your PSU might be too weak. What kind of PSU are you using at the moment?</p>

username: kirk86
cooked: <p>I don’t think that’s the case I have an 850W <a href="https://www.lc-power.com/en/product/pc-power-supply-units/metatron-gaming-series/lc8850iii-v23-arkangel-3/" rel="nofollow noopener">PSU</a></p>

username: ptrblck
cooked: <p>I think this might be indeed the problem, as the recommended system power for the <a href="https://www.geforce.com/hardware/desktop-gpus/geforce-gtx-titan-x/specifications" rel="nofollow noopener">Titan X</a> are 600W and the <a href="https://www.geforce.com/hardware/desktop-gpus/geforce-gtx-1080-ti/specifications" rel="nofollow noopener">1080 Ti</a> might need another 250W at max. performance. Your PSU might be maxed out with these two GPUs (of course it also depends on other hardware and its power consumption).</p>

username: kirk86
cooked: <p>My understanding of reading the linked paged to Titan X is that the actual card requires 250W and it’s recommended to have an overall system with 600W in which you’re gonna use the card? 600W for Titan X seems ridiculously high to me? If I max out the TX then it barely goes above 260W. In the worst case scenario both of them maxed out would have gotten 600W still not explains the reboots, also I’ve trained multi-gpu using tensorflow and maxing out both cards didn’t notice any of these issues?</p>

username: ptrblck
cooked: <p>The info for the Titan X states 600W are recommended for this single GPU and the whole system.<br>
So if that’s the max. power consumption of a single Titan X + rest of the system, the additional 1080Ti might need 250W extra as stated on the other page.</p>
<p>Maybe you were lucky in TF as the GPUs might not have been at their peak power consumption.<br>
You could try to set the power limits lower using <code>nvidia-smi</code>, maybe even create artificial bottlenecks so that your GPUs will only have a short burst of power consumption or test another PSU if available.</p>

username: kirk86
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="54492">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>set the power limits lower using <code>nvidia-smi</code></p>
</blockquote>
</aside>
<p>Oh nice, I didn’t know you could do that! Let me search around and see how to test those things. Thanks for the tip!</p>

username: kirk86
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
so I did a breakdown of the components and Watts consumption required on my setup.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/73bd3581bf17a89d6348dc877bc3686de94d7ad8.png" data-download-href="https://discuss.pytorch.org/uploads/default/73bd3581bf17a89d6348dc877bc3686de94d7ad8" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/7/73bd3581bf17a89d6348dc877bc3686de94d7ad8.png" alt="image" data-base62-sha1="gvShPioFSV6dAXufsZPzZ7kHnrG" width="690" height="429" data-dominant-color="E8E8E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">702×437 47.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And it seems that an 850W PSU should be able to handle that no?</p>

username: phuocnguyen2008
cooked: <p>Hi folks,</p>
<p>Sorry to bump this topic again but I have a problem with my server.<br>
Currently, I am using two A6000 GPUs. The PSU is 1200W, which is enough for the system as I investigated.<br>
Whenever I train my model using torch.nn.DataParallel, the system reboots after a few minutes.</p>
<p>I wonder about the reasons behind this situation.</p>