Multi-GPU Error: ProcessExitedException: process 0 terminated with signal SIGSEGV

chess2021: <p>I am trying to use DDP to do multi-GPU training of my model, however I am facing the following error:</p>
<blockquote>
<p>ProcessExitedException: process 0 terminated with signal SIGSEGV</p>
</blockquote>
<p>I am using PyTorch lightening. My code works perfectly for a single GPU machine.<br>
My system environment is as follows:<br>
Python: 3.10-6<br>
Cuda: 11.4<br>
Torch: 2.0.1+cu117</p>
<p>I searched in the internet and people were talking about downgrading Python version to 3.8 from 3.9 however all those posts are old back in 2021, and wondering if there is any solution to this problem (since downgrading python version may not be an option for me especially going to Python 3.8).</p>
<p>Following are some more info about this error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c8017d7849dd0b0baa510696d1c32e58d979a9b1.png" data-download-href="https://discuss.pytorch.org/uploads/default/c8017d7849dd0b0baa510696d1c32e58d979a9b1" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c8017d7849dd0b0baa510696d1c32e58d979a9b1.png" alt="image" data-base62-sha1="sxkGztX42fRaXXUZ2pir9SCULCx" width="690" height="287" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1122×468 31.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/0/c075e1c433bb80356b90d191c494b1e4c5d46d78.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/c075e1c433bb80356b90d191c494b1e4c5d46d78" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c075e1c433bb80356b90d191c494b1e4c5d46d78_2_524x500.jpeg" alt="image" data-base62-sha1="rsAezQBJcrAo5OaHRq3YdGj8PMs" width="524" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c075e1c433bb80356b90d191c494b1e4c5d46d78_2_524x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c075e1c433bb80356b90d191c494b1e4c5d46d78_2_786x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c075e1c433bb80356b90d191c494b1e4c5d46d78_2_1048x1000.jpeg 2x" data-dominant-color="F4F4F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1786×1702 189 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you provide the same debug information and stacktraces as discussed in [this topic[(<a href="https://discuss.pytorch.org/t/pytorch-multi-processes-error-on-sagemaker-gpu-instance/172484/8" class="inline-onebox">Pytorch multi-processes error on SageMaker GPU instance - #8 by chess2021</a>) you’ve cross-posted the question in?</p>