Process stuck in backward if multi-rtx3090 were enabled

username: huany11
cooked: <p>Hi Guys,<br>
When I run my AI train in multi-GPUs mode, process always stuck (hang) in backward. I traced it into torch interface torch/autograd/<strong>init</strong>.py line 145 Variable._execution_engine.run_backward. It was blocked process and never return. No matter what I have set up DataParallel or DistributedDataParallel.  The console nothing output. I guess something dead-loop in torch/_C.cpython-38-x86_64-linux-gnu.so or further. Have anyone know about this issue or workaround solution?<br>
I compare two mode train (the single GPU and both GPUs), my train is successfully passed when only one GPUs but failed in double. And I collect the arguments of both of them when pass it into Variable.<em>execution_engine.run_backward, they are same.<br>
rt_frames.T = {NameError}name ‘rt_frames’ is not defined<br>
create_graph = {bool} False<br>
grad_tensors = {NoneType} None<br>
grad_tensors</em> = {tuple: 1} tensor(1., device=‘cuda:0’)<br>
grad_variables = {NoneType} None<br>
inputs = {tuple: 0} ()<br>
retain_graph = {bool} False<br>
tensors = {tuple: 1} tensor(0.6899, device=‘cuda:0’, grad_fn=)<br>
And I also test GPU + CUDA with p2pBandwithLatencyTest and the result is OK.<br>
Would anyone do me a favor on having a look at this issue?<br>
My hardware environment is CPU Intel® Core™ i9-7900X CPU @ 3.30GHz, Memory 8G*8 DDR3,<br>
GPU rtx3090<br>
software environment is Ubuntu18.04TLS, RTX3090 Driver460.56 + cuda_11.1.TC455_06.29069683_0 + Tensorflow 2.4.1.+ Pytorch1.8.1(cu111).</p>

username: ptrblck
cooked: <p>You could check <code>dmesg</code> for Xid errors and see, if a GPU might be dropped due to some issues such as a PSU, which might be too weak to drive two 3090s.</p>

username: huany11
cooked: <p>Hi Ptrblck,<br>
dmesg show nothing about Xid error. and I check PSU power, all items are unknow <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<h1>dmidecode 3.1</h1>
<p>Getting SMBIOS data from sysfs.<br>
SMBIOS 3.2.0 present.</p>
<h1>SMBIOS implementations newer than version 3.1.1 are not</h1>
<h1>fully supported by this version of dmidecode.</h1>
<p>Handle 0x0048, DMI type 39, 22 bytes<br>
System Power Supply<br>
Power Unit Group: 1<br>
Location: To Be Filled By O.E.M.<br>
Name: To Be Filled By O.E.M.<br>
Manufacturer: To Be Filled By O.E.M.<br>
Serial Number: To Be Filled By O.E.M.<br>
Asset Tag: To Be Filled By O.E.M.<br>
Model Part Number: To Be Filled By O.E.M.<br>
Revision: To Be Filled By O.E.M.<br>
Max Power Capacity: Unknown<br>
Status: Present, OK<br>
Type: Switching<br>
Input Voltage Range Switching: Auto-switch<br>
Plugged: Yes<br>
Hot Replaceable: No<br>
Input Voltage Probe Handle: 0x0044<br>
Cooling Device Handle: 0x0046<br>
Input Current Probe Handle: 0x0047</p>

username: huany11
cooked: <p>I just did a test that AI training script copied in 2 and 1 cuda:0 and the other file is cuda:1. Then I launched them separately. They can successfully run themselves in their own GPU card. So I think the PSU is not a key problem.<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e74d31cd45909779e05f7e4c8c40991aa21a794c.png" alt="2process" data-base62-sha1="x0bGq6iPLtp6v0XI6cB0sUPJHVa" width="665" height="474"></p>

username: huany11
cooked: <p>Finally, I found the root cause, My PC is too hot to launch 2 GPUs. After I take off the chassis cover. It work!</p>