Torch DDP with accelerate using torchrun cause failed ERROR with exitcode: -11

deJQK: <p>Hi, I am trying to use <code>accelerate</code> with <code>torchrun</code>, and inside the <code>accelerate</code> code they call <code>torch.nn.parallel.DistributedDataParallel</code> which causes ERROR with either 1GPU or multiple GPU. Here is a simple code example:</p>
<pre><code class="lang-auto">## ./debug.py
import os

from accelerate import Accelerator
from accelerate.utils import ProjectConfiguration

from diffusers import UNet2DConditionModel

import torch

def main():
        output_dir = './debug'
        accelerator_project_config = ProjectConfiguration(
            total_limit=10,
            automatic_checkpoint_naming=True,
            project_dir=output_dir,
            logging_dir=os.path.join(output_dir, 'logs'),
        )
          
        accelerator = Accelerator(
            gradient_accumulation_steps=2,
            mixed_precision='fp16',
            project_config=accelerator_project_config,
        )
        
        if accelerator.is_main_process:
            os.makedirs(output_dir, exist_ok=True)
            os.makedirs(os.path.join(output_dir, 'checkpoints'), exist_ok=True)
        
        unet = UNet2DConditionModel.from_pretrained(
            "runwayml/stable-diffusion-v1-5",
            subfolder="unet",
            revision=None
        )
        
        for d_idx in range(accelerator.state.num_processes):
            unet_idx = torch.nn.parallel.DistributedDataParallel(
                unet.cuda(), device_ids=[d_idx], output_device=d_idx,
            )
        unet= accelerator.prepare(unet)

if __name__ == '__main__':
    main()
</code></pre>
<p>When I run this code with the following script:</p>
<pre><code class="lang-auto">export TORCH_CPP_LOG_LEVEL=INFO NCCL_DEBUG=INFO
export CUDA_VISIBLE_DEVICES=0
torchrun --nproc_per_node=1 ./debug.py
</code></pre>
<p>the output will be something like the following:</p>
<pre><code class="lang-auto">[I ProcessGroupNCCL.cpp:842] [Rank 0] NCCL watchdog thread started!
**hostname**:70473:70473 [0] NCCL INFO Bootstrap : Using eth0:10.200.155.199&lt;0&gt;
**hostname**:70473:70473 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
**hostname**:70473:70473 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).

**hostname**:70473:70473 [0] misc/cudawrap.cc:112 NCCL WARN cuDriverGetVersion failed with 34
NCCL version 2.14.3+cuda11.7
**hostname**:70473:70611 [0] NCCL INFO NET/OFI Using aws-ofi-nccl 1.5.0aws
**hostname**:70473:70611 [0] NCCL INFO NET/OFI Selected Provider is tcp (found 1 nics)
**hostname**:70473:70611 [0] NCCL INFO Using network AWS Libfabric
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -11) local_rank: 0 (pid: 70473) of binary: /opt/conda/bin/python
Traceback (most recent call last):
  File "/opt/conda/bin/torchrun", line 33, in &lt;module&gt;
    sys.exit(load_entry_point('torch==2.0.1', 'console_scripts', 'torchrun')())
  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
    run(args)
  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
=======================================================
./debug.py FAILED
-------------------------------------------------------
Failures:
  &lt;NO_OTHER_FAILURES&gt;
-------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-09-24_20:02:11
  host      : **hostname**
  rank      : 0 (local_rank: 0)
  exitcode  : -11 (pid: 70473)
  error_file: &lt;N/A&gt;
  traceback : Signal 11 (SIGSEGV) received by PID 70473
=======================================================
</code></pre>
<p>Using 8 GPU with <code>--nproc_per_node=8</code> gives the same error.</p>
<p>The above code part is extracted based on <a href="https://github.com/huggingface/accelerate/blob/main/src/accelerate/accelerator.py#L1424-L1431" rel="noopener nofollow ugc">the code</a> in accelerate library.</p>
<p>The environment I am using is as follows:</p>
<pre><code class="lang-auto">accelerate==0.20.3
torch==2.0.1
diffusers==0.20.0
</code></pre>
<p>with <code>cuda_11.7.r11.7/compiler.31442593_0</code> and <code>A100</code> GPUs.</p>
<p>Does anyone know the reason and solution for this? Thanks!</p>

ptrblck: <aside class="quote no-group" data-username="deJQK" data-post="1" data-topic="188882">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dejqk/48/4965_2.png" class="avatar"> deJQK:</div>
<blockquote>
<p><code>misc/cudawrap.cc:112 NCCL WARN cuDriverGetVersion failed with 34</code></p>
</blockquote>
</aside>
<p>Your system seems to have trouble communicating with the NVIDIA driver. Was this system working before?</p>

deJQK: <p>Thanks. It should work fine before but I could double-check on this. How could I debug to see if the system communicate well with the NVIDIA driver by some nv-commands?</p>