DistributedDataParallel on multiple GPU nodes slower than one GPU node

Yinglei_Zhang: <p>Hi, i am using PyTorch DistributedDataParallel to train some models and surpisingly. It become slower when i move from one GPU node to 2 GPU nodes.</p>
<p>Here is the example i am running: <a href="https://github.com/huggingface/transformers/blob/v2.3.0/examples/run_lm_finetuning.py" rel="nofollow noopener">https://github.com/huggingface/transformers/blob/v2.3.0/examples/run_lm_finetuning.py</a></p>
<p>Here is my infrastructure:</p>
<p>Single Node: p3.16xlarge<br>
Training time: 36 mins</p>
<p>Two Nodes: p3.16xlarge<br>
Training time: 1 h 45 mins</p>
<p>In both cases, i am using PyTorch distributed data parallel and GPU utilization is almost always be 100%.</p>
<p>I have enabled NCCL_DEBUG=INFO<br>
I copied the nccl output from single node training and multiple node training in this link below.<br>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/YingleiZhang/a8df48eb534ba20ff8f26b5309094b55" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/YingleiZhang/a8df48eb534ba20ff8f26b5309094b55" target="_blank" rel="nofollow noopener">https://gist.github.com/YingleiZhang/a8df48eb534ba20ff8f26b5309094b55</a></h4>
<h5>NCCL debug info on multiple node:</h5>
<pre><code class="">ip-10-0-42-19:13243:13243 [2] NCCL INFO Bootstrap : Using [0]eth0:10.0.42.19&lt;0&gt;
ip-10-0-42-19:13282:13282 [7] NCCL INFO Bootstrap : Using [0]eth0:10.0.42.19&lt;0&gt;
ip-10-0-42-19:13243:13243 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ip-10-0-42-19:13282:13282 [7] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
 
ip-10-0-42-19:13243:13243 [2] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
 
ip-10-0-42-19:13282:13282 [7] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
ip-10-0-42-19:13282:13282 [7] NCCL INFO NET/Socket : Using [0]eth0:10.0.42.19&lt;0&gt;
ip-10-0-42-19:13243:13243 [2] NCCL INFO NET/Socket : Using [0]eth0:10.0.42.19&lt;0&gt;</code></pre>
This file has been truncated. <a href="https://gist.github.com/YingleiZhang/a8df48eb534ba20ff8f26b5309094b55" target="_blank" rel="nofollow noopener">show original</a>
<h5>NCCL debug info on single node:</h5>
<pre><code class="">ip-10-0-28-255:46114:46114 [0] NCCL INFO Bootstrap : Using [0]eth0:10.0.28.255&lt;0&gt;
ip-10-0-28-255:46114:46114 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
 
ip-10-0-28-255:46114:46114 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
ip-10-0-28-255:46114:46114 [0] NCCL INFO NET/Socket : Using [0]eth0:10.0.28.255&lt;0&gt;
NCCL version 2.5.6+cuda10.0
ip-10-0-28-255:46149:46149 [5] NCCL INFO Bootstrap : Using [0]eth0:10.0.28.255&lt;0&gt;
ip-10-0-28-255:46149:46149 [5] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
 
ip-10-0-28-255:46149:46149 [5] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]</code></pre>
This file has been truncated. <a href="https://gist.github.com/YingleiZhang/a8df48eb534ba20ff8f26b5309094b55" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>It seems on single node mode, NCCL are creating a lot of CHANNELs, but not in the multiple node mode. Could this slow speed due to the networking issues between different GPU nodes?</p>
<p>Any help is appreciated.</p>

mrshenli: <p>Two questions,</p>
<ol>
<li>Did you divide the the epoch size on each process by <code>world_size</code>?</li>
<li>Will there be any contention on the data loader?</li>
</ol>
<p>cc <a class="mention" href="/u/osalpekar">@osalpekar</a></p>
<p>Also cc <a class="mention" href="/u/zhangguanheng66">@zhangguanheng66</a> for transformer questions</p>

Yinglei_Zhang: <p>We are tracking this issue here: <a href="https://github.com/NVIDIA/nccl/issues/318" rel="nofollow noopener">https://github.com/NVIDIA/nccl/issues/318</a></p>
<p>It looks like the bottleneck is the network bandwidth between two GPU nodes. This example is fine tuning roberta-base model, which is about 500 MB. During the synchronization of two nodes cases, the amount of data would be 8G. NCCL folks mentioned that on single instances, this communication is supported by nvlink, and it is about 120GB/s, however on if this is tcp/ip sockets, even if the bandwidth is 100GB/s, the all reduce bandwidth would be 10GB/s. So like a order of magnitude smaller, therefore the communication time become the bottleneck.</p>
<p>In this example, the epoch is set to 1 for both single node and two nodes case.<br>
The training script uses distributedSampler, so the data partition on each node should be mutually exclusive.</p>
<p>if you feel there is anything else other than the network bandwidth would cause any issues, let me know.</p>