How to fix Signal 11 (SIGSEGV) problem when I use DDP?

LuoXin-s: <h3>
<a name="describe-the-bug-1" class="anchor" href="#describe-the-bug-1"></a><img src="https://discuss.pytorch.org/images/emoji/apple/bug.png?v=12" title=":bug:" class="emoji" alt=":bug:" loading="lazy" width="20" height="20"> Describe the bug</h3>
<p>I can run this <a href="https://github.com/pytorch/examples/blob/main/distributed/ddp/main.py" rel="noopener nofollow ugc">code</a> successfully on default gloo backend, and it will fail when I shift to nccl backend.</p>
<p>Note that I have a higher cuda version than pytorch built, however I expect it could work without problem as per <a href="https://discuss.pytorch.org/t/install-pytorch-with-cuda-12-1/174294">Install pytorch with Cuda 12.1 1</a>.</p>
<p>I have tried to use python 3.8, however the problem persists.</p>
<h3>
<a name="versions-2" class="anchor" href="#versions-2"></a>Versions</h3>
<p>Below is my environments:<br>
PyTorch version: 2.0.0+cu117<br>
CUDA used to build PyTorch: 11.7</p>
<p>OS: Ubuntu 22.04.2 LTS (x86_64)<br>
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0<br>
Libc version: glibc-2.35</p>
<p>Python version: 3.9.12 (main, Apr 5 2022, 06:56:58) [GCC 7.5.0] (64-bit runtime)<br>
Python platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35<br>
Is CUDA available: True<br>
CUDA runtime version: 12.0.140<br>
CUDA_MODULE_LOADING set to: LAZY<br>
GPU models and configuration:<br>
GPU 0: NVIDIA A100-SXM4-80GB<br>
GPU 1: NVIDIA A100-SXM4-80GB</p>
<p>Nvidia driver version: 525.60.13<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A<br>
Is XNNPACK available: True</p>

ptrblck: <aside class="quote no-group quote-modified" data-username="LuoXin-s" data-post="1" data-topic="178752">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/luoxin-s/48/35927_2.png" class="avatar"> LuoXin-s:</div>
<blockquote>
<p>Note that I use a higher cuda version than pytorch built</p>
</blockquote>
</aside>
<p>You are not using the locally installed CUDA toolkit as the PyTorch pip wheels ship with their own CUDA runtime and dependencies. Only the NVIDIA driver will be used.</p>
<p>To debug the issue you could try to get the stacktrace by launching the application from <code>gdb</code>.</p>

LuoXin-s: <p>Ok, I’ll try it. Though I guess the problem may come from python 3.9, I see some others downgrade to python 3.8 to fix this problem. see <a href="https://discuss.pytorch.org/t/how-to-fix-a-sigsegv-in-pytorch-when-using-distributed-training-e-g-ddp/113518/7" class="inline-onebox">How to fix a SIGSEGV in pytorch when using distributed training (e.g. DDP)? - #7 by Brando_Miranda</a> and <a href="https://discuss.pytorch.org/t/how-to-fix-sigsegv-in-distributed-training-i-e-ddp-2/120571/4" class="inline-onebox">How to fix SIGSEGV in distributed training (i.e. DDP) 2 - #4 by dsethz</a></p>

LuoXin-s: <p>I have run the same code on python 3.8, it fails also.</p>

LuoXin-s: <p>I can not use gdb, since my program works on a GPU cluster.</p>

LetterLiGo: <p>How about trying <code>gloo</code> instead of <code>nccl</code> for your backend communication? I fix the SIGSEGV error by it.</p>