Distributed Data Parallel single node maximum number of GPUs

bapi: <p>DistributedDataParallel imagenet training example breaks throwing the following error: <code>RuntimeError: NCCL error in: /tmp/pip-req-build-4baxydiv/torch/lib/c10d/ProcessGroupNCCL.cpp:400, unhandled cuda error</code> on running it on a single node with 10 GPUs. The same runs perfectly fine as soon as the number of GPUs in the environment is set to 8. For DataParallel, somewhere it is mentioned that at present it does not run on more than 8 GPUs; however, I could not find similar info about DDP (I may have missed it). Moreover, as all the processes load their own module locally on each of the devices without a broadcast during initialization, is not it unexpected?</p>

ptrblck: <p>Which NCCL version are you using?<br>
Could you rerun your script with <code>NCCL_DEBUG=DEBUG python ...</code> and post the log here?</p>

bapi: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="65485">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>NCCL_DEBUG=DEBUG</p>
</blockquote>
</aside>
<p>Thanks for the response.<br>
That flag does not generate much info other than the usual output. I reran the code with <code>NCCL_DEBUG=INFO</code>, below is the log (machine is named gpu123):</p>
<blockquote>
<p>$ NCCL_DEBUG=INFO python main.py -a resnet18 --dist-url ‘tcp://127.0.0.1:6840’ --dist-backend ‘nccl’ --multiprocessing-distributed --world-size 1 --local_rank 0 ~/tiny-imagenet-200<br>
Use GPU: 3 for training                                                                                                                                   Use GPU: 5 for training                                                                                                                                   Use GPU: 7 for training                                                                                                                                   Use GPU: 9 for training                                                                                                                                   Use GPU: 6 for training                                                                                                                                   Use GPU: 1 for training                                                                                                                                   Use GPU: 0 for training                                                                                                                                   Use GPU: 4 for training                                                                                                                                   =&gt; creating model ‘resnet18’                                                                                                                              Use GPU: 8 for training                                                                                                                                   =&gt; creating model ‘resnet18’                                                                                                                              Use GPU: 2 for training                                                                                                                                   =&gt; creating model ‘resnet18’                                                                                                                              =&gt; creating model ‘resnet18’                                                                                                                              =&gt; creating model ‘resnet18’                                                                                                                              =&gt; creating model ‘resnet18’                                                                                                                              =&gt; creating model ‘resnet18’                                                                                                                              =&gt; creating model ‘resnet18’                                                                                                                              =&gt; creating model ‘resnet18’                                                                                                                              =&gt; creating model ‘resnet18’                                                                                                                              gpu123:7932:7932 [0] NCCL INFO Bootstrap : Using [0]ib0:10.36.192.223&lt;0&gt;                                                                                  gpu123:7932:7932 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).                                                                             gpu123:7932:7932 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:10.36.192.223&lt;0&gt;                                                                   NCCL version 2.4.8+cuda10.0                                                                                                                               gpu123:7941:7941 [9] NCCL INFO Bootstrap : Using [0]ib0:10.36.192.223&lt;0&gt;                                                                                  gpu123:7936:7936 [4] NCCL INFO Bootstrap : Using [0]ib0:10.36.192.223&lt;0&gt;                                                                                  gpu123:7934:7934 [2] NCCL INFO Bootstrap : Using [0]ib0:10.36.192.223&lt;0&gt;                                                                                  gpu123:7933:7933 [1] NCCL INFO Bootstrap : Using [0]ib0:10.36.192.223&lt;0&gt;                                                                                  gpu123:7937:7937 [5] NCCL INFO Bootstrap : Using [0]ib0:10.36.192.223&lt;0&gt;                                                                                  gpu123:7935:7935 [3] NCCL INFO Bootstrap : Using [0]ib0:10.36.192.223&lt;0&gt;                                                                                  gpu123:7941:7941 [9] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).                                                                             gpu123:7936:7936 [4] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).                                                                             gpu123:7934:7934 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).                                                                             gpu123:7933:7933 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).                                                                             gpu123:7937:7937 [5] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).                                                                             gpu123:7935:7935 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).                                                                             gpu123:7940:7940 [8] NCCL INFO Bootstrap : Using [0]ib0:10.36.192.223&lt;0&gt;                                                                                  gpu123:7940:7940 [8] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).                                                                             gpu123:7941:7941 [9] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:10.36.192.223&lt;0&gt;                                                                   gpu123:7935:7935 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:10.36.192.223&lt;0&gt;                                                                   gpu123:7933:7933 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:10.36.192.223&lt;0&gt;                                                                   gpu123:7936:7936 [4] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:10.36.192.223&lt;0&gt;                                                                   gpu123:7934:7934 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:10.36.192.223&lt;0&gt;                                                                   gpu123:7937:7937 [5] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:10.36.192.223&lt;0&gt;                                                                   gpu123:7940:7940 [8] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:10.36.192.223&lt;0&gt;                                                                   gpu123:7932:8011 [0] NCCL INFO Setting affinity for GPU 0 to 0fff                                                                                         gpu123:7941:8013 [9] NCCL INFO Setting affinity for GPU 9 to 0fff                                                                                         gpu123:7940:8025 [8] NCCL INFO Setting affinity for GPU 8 to 0fff                                                                                         gpu123:7934:8022 [2] NCCL INFO Setting affinity for GPU 2 to 0fff                                                                                         gpu123:7937:8023 [5] NCCL INFO Setting affinity for GPU 5 to 0fff                                                                                         gpu123:7935:8017 [3] NCCL INFO Setting affinity for GPU 3 to 0fff                                                                                         gpu123:7936:8020 [4] NCCL INFO Setting affinity for GPU 4 to 0fff                                                                                         gpu123:7933:8018 [1] NCCL INFO Setting affinity for GPU 1 to 0fff                                                                                         gpu123:7939:7939 [7] NCCL INFO Bootstrap : Using [0]ib0:10.36.192.223&lt;0&gt;                                                                                  gpu123:7939:7939 [7] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).                                                                             gpu123:7939:7939 [7] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:10.36.192.223&lt;0&gt;                                                                   gpu123:7938:7938 [6] NCCL INFO Bootstrap : Using [0]ib0:10.36.192.223&lt;0&gt;                                                                                  gpu123:7938:7938 [6] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).                                                                             gpu123:7938:7938 [6] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:10.36.192.223&lt;0&gt;                                                                   gpu123:7939:8027 [7] NCCL INFO Setting affinity for GPU 7 to 0fff                                                                                         gpu123:7938:8029 [6] NCCL INFO Setting affinity for GPU 6 to 0fff                                                                                         gpu123:7932:8011 [0] NCCL INFO Channel 00 :    0   1   2   3   4   5   6   7   8   9                                                                      gpu123:7935:8017 [3] NCCL INFO Ring 00 : 3[3] → 4[4] via P2P/IPC                                                                                         gpu123:7938:8029 [6] NCCL INFO Ring 00 : 6[6] → 7[7] via P2P/IPC                                                                                         gpu123:7936:8020 [4] NCCL INFO Ring 00 : 4[4] → 5[5] via P2P/IPC                                                                                         gpu123:7937:8023 [5] NCCL INFO Ring 00 : 5[5] → 6[6] via P2P/IPC                                                                                         gpu123:7939:8027 [7] NCCL INFO Ring 00 : 7[7] → 8[8] via P2P/IPC                                                                                         gpu123:7941:8013 [9] NCCL INFO Ring 00 : 9[9] → 0[0] via P2P/IPC                                                                                         gpu123:7940:8025 [8] NCCL INFO Ring 00 : 8[8] → 9[9] via P2P/IPC                                                                                         gpu123:7933:8018 [1] NCCL INFO Ring 00 : 1[1] → 2[2] via P2P/IPC                                                                                         gpu123:7932:8011 [0] NCCL INFO Ring 00 : 0[0] → 1[1] via P2P/IPC                                                                                         gpu123:7934:8022 [2] NCCL INFO Ring 00 : 2[2] → 3[3] via P2P/IPC                                                                                                                                                                                                                                                   gpu123:7941:8013 [9] transport/p2p.cc:574 NCCL WARN failed to open CUDA IPC handle : 60 peer mapping resources exhausted                                  gpu123:7941:8013 [9] NCCL INFO init.cc:669 → 1                                                                                                           gpu123:7941:8013 [9] NCCL INFO init.cc:815 → 1                                                                                                           gpu123:7941:8013 [9] NCCL INFO init.cc:951 → 1                                                                                                           gpu123:7941:8013 [9] NCCL INFO misc/group.cc:69 → 1 [Async thread]                                                                                                                                                                                                                                                 gpu123:7932:8011 [0] transport/p2p.cc:604 NCCL WARN failed to open CUDA IPC handle : 60 peer mapping resources exhausted                                  gpu123:7932:8011 [0] NCCL INFO init.cc:679 → 1                                                                                                           gpu123:7932:8011 [0] NCCL INFO init.cc:815 → 1                                                                                                           gpu123:7932:8011 [0] NCCL INFO init.cc:951 → 1                                                                                                           gpu123:7932:8011 [0] NCCL INFO misc/group.cc:69 → 1 [Async thread]                                                                                       gpu123:7935:8017 [3] NCCL INFO comm 0x2abf58001e10 rank 3 nranks 10 cudaDev 3 nvmlDev 3 - Init COMPLETE                                                   gpu123:7934:8022 [2] NCCL INFO comm 0x2b8408001e10 rank 2 nranks 10 cudaDev 2 nvmlDev 2 - Init COMPLETE                                                   gpu123:7937:8023 [5] NCCL INFO comm 0x2b011c001e10 rank 5 nranks 10 cudaDev 5 nvmlDev 5 - Init COMPLETE                                                   gpu123:7936:8020 [4] NCCL INFO comm 0x2b4b28001e10 rank 4 nranks 10 cudaDev 4 nvmlDev 4 - Init COMPLETE                                                   gpu123:7938:8029 [6] NCCL INFO comm 0x2b58d8001e10 rank 6 nranks 10 cudaDev 6 nvmlDev 6 - Init COMPLETE                                                   gpu123:7933:8018 [1] NCCL INFO comm 0x2af070001e10 rank 1 nranks 10 cudaDev 1 nvmlDev 1 - Init COMPLETE                                                   gpu123:7939:8027 [7] NCCL INFO comm 0x2b2180001e10 rank 7 nranks 10 cudaDev 7 nvmlDev 7 - Init COMPLETE                                                   gpu123:7940:8025 [8] NCCL INFO comm 0x2b015c001e10 rank 8 nranks 10 cudaDev 8 nvmlDev 8 - Init COMPLETE                                                   /nfs/scistore08/alistgrp/bchatter/anaconda3/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 11 leaked semaphores to clean up at shutdown                                                                                                                   len(cache))                                                                                                                                             /nfs/scistore08/alistgrp/bchatter/anaconda3/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 11 leaked semaphores to clean up at shutdown                                                                                                                   len(cache))                                                                                                                                             /nfs/scistore08/alistgrp/bchatter/anaconda3/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 11 leaked semaphores to clean up at shutdown                                                                                                                   len(cache))                                                                                                                                             /nfs/scistore08/alistgrp/bchatter/anaconda3/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 11 leaked semaphores to clean up at shutdown                                                                                                                   len(cache))                                                                                                                                             /nfs/scistore08/alistgrp/bchatter/anaconda3/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 11 leaked semaphores to clean up at shutdown                                                                                                                   len(cache))                                                                                                                                             /nfs/scistore08/alistgrp/bchatter/anaconda3/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 11 leaked semaphores to clean up at shutdown                                                                                                                   len(cache))                                                                                                                                             Traceback (most recent call last):                                                                                                                          File “main.py”, line 425, in                                                                                                                        main()                                                                                                                                                  File “main.py”, line 109, in main                                                                                                                           mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args))                                                                               File “/nfs/scistore08/alistgrp/bchatter/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py”, line 171, in spawn                           while not spawn_context.join():                                                                                                                         File “/nfs/scistore08/alistgrp/bchatter/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py”, line 118, in join                            raise Exception(msg)                                                                                                                                  Exception:                                                                                                                                                                                                                                                                                                          – Process 9 terminated with the following error:                                                                                                         Traceback (most recent call last):                                                                                                                          File “/nfs/scistore08/alistgrp/bchatter/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py”, line 19, in _wrap                            fn(i, *args)                                                                                                                                            File “/nfs/scistore08/alistgrp/bchatter/workspace/async-opt/dist_data_parallel/imagenet_training/main.py”, line 151, in main_worker                         model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu])                                                                         File “/nfs/scistore08/alistgrp/bchatter/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/distributed.py”, line 298, in <strong>init</strong>                      self.broadcast_bucket_size)                                                                                                                             File “/nfs/scistore08/alistgrp/bchatter/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/distributed.py”, line 480, in _distributed_broadcast_coalesced                                                                                                                                                        dist._broadcast_coalesced(self.process_group, tensors, buffer_size)                                                                                   RuntimeError: NCCL error in: /tmp/pip-req-build-4baxydiv/torch/lib/c10d/ProcessGroupNCCL.cpp:400, unhandled cuda error</p>
</blockquote>

bapi: <p><code>torch.cuda.nccl.version()</code> outputs 2408.</p>

mrshenli: <p>Hey <a class="mention" href="/u/bapi">@bapi</a></p>
<blockquote>
<p>For DataParallel, somewhere it is mentioned that at present it does not run on more than 8 GPUs;</p>
</blockquote>
<p>Just curious, could you please point me to the doc with this claim? This is new to me, I wasn’t aware there is such a limitation in DP.</p>
<blockquote>
<p>however, I could not find similar info about DDP (I may have missed it).</p>
</blockquote>
<p>We recently tested DDP using 256 GPUs, and it runs fine. Could this error be sth specific to the imagenet example? cc <a class="mention" href="/u/fmassa">@fmassa</a> for vision questions</p>
<blockquote>
<p>Moreover, as all the processes load their own module locally on each of the devices without a broadcast during initialization, is not it unexpected?</p>
</blockquote>
<p>There is a broadcast in DDP ctor. Please see the link below:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/c71ec1c717e5b225f28ef3bacde416c69f3c4d77/torch/nn/parallel/distributed.py#L324-L329">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/c71ec1c717e5b225f28ef3bacde416c69f3c4d77/torch/nn/parallel/distributed.py#L324-L329" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/c71ec1c717e5b225f28ef3bacde416c69f3c4d77/torch/nn/parallel/distributed.py#L324-L329" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/c71ec1c717e5b225f28ef3bacde416c69f3c4d77/torch/nn/parallel/distributed.py#L324-L329</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="324" style="counter-reset: li-counter 323 ;">
          <li># Sync params and buffers</li>
          <li>module_states = list(self.module.state_dict().values())</li>
          <li>if len(module_states) &gt; 0:</li>
          <li>    self._distributed_broadcast_coalesced(</li>
          <li>        module_states,</li>
          <li>        self.broadcast_bucket_size)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


bapi: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a>, (sorry for such a late read/response to this message)</p>
<p>So, I figured out that using either the flag NCCL_P2P_LEVEL=0 or NCCL_P2P_DISABLE=1, DDP runs fine on a machine with &gt;8 GPUs. Here I am specifically talking about 10 GPUs in the same machine. I am not sure about the topology of the 256 GPUs that you mentioned.</p>
<p>Right now, I can not locate the doc page where I had seen that nn.dataparallel does not run (efficiently or something?) on more than 8 GPUs in a machine. Maybe it was in the previous version of the doc? In any case, I will confirm this one by testing it on a machine with 10 GPUs that I have access to.</p>
<p>Thanks.</p>

mrshenli: <aside class="quote no-group" data-username="bapi" data-post="7" data-topic="65485">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bapi/48/27077_2.png" class="avatar"> bapi:</div>
<blockquote>
<p>I am not sure about the topology of the 256 GPUs that you mentioned.</p>
</blockquote>
</aside>
<p>In that test, each node only has 8 GPUs.</p>
<blockquote>
<p>So, I figured out that using either the flag NCCL_P2P_LEVEL=0 or NCCL_P2P_DISABLE=1, DDP runs fine on a machine with &gt;8 GPUs.</p>
</blockquote>
<p>I see. We don’t have tests covering &gt; 8GPUs per node cases yet. This is an important message, thanks for sharing!</p>