DistributedDataParallel: cuda init; forced parameter sync; pinning?

vadimkantorov: <p>I’m trying to figure out DistributedDataParallel (on a single machine; single GPU / process mode). I’ve got a few questions:</p>
<ol>
<li>
<p>Will all launched processes do CUDA init? Is it safe? Should we explicitly set CUDA_VISIBLE_DEVICES per launched process to ensure that it can see only one device? ( think fork/spawn global state issues with CUDA / OpenMP / pthreads etc…) Would it prevent it from rpc using nccl?</p>
</li>
<li>
<p>Is it sensible to do a forced parameter sync once in a while? Inherent GPU parallelism non-determinism can cause parameter divergence that could cause replica parameter divergence for some sensitive models (even if gradients are sychronized)? How does one do it?</p>
</li>
<li>
<p>When does it make sense to do NUMA node pinning? CPU affinity pinning?</p>
</li>
</ol>
<p>Thank you!</p>

mrshenli: <aside class="quote no-group" data-username="vadimkantorov" data-post="1" data-topic="95547">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vadimkantorov/48/365_2.png" class="avatar"> vadimkantorov:</div>
<blockquote>
<p>Will all launched processes do CUDA init?</p>
</blockquote>
</aside>
<p>CUDA is lazily initialized. So if one process is not touching a specific device, corresponding CUDA context shouldn’t be created on that device.</p>
<blockquote>
<p>Is it safe?</p>
</blockquote>
<p>Even if multiple processes create context on the same device, it won’t crash, but each context consumes about 500MB CUDA memory, which is not desired.</p>
<blockquote>
<p>Should we explicitly set CUDA_VISIBLE_DEVICES per launched process to ensure that it can see only one device?</p>
</blockquote>
<p>This is the recommended way, as it also rules out the possibility that some third-party library accidentally create tensors on other devices.</p>
<blockquote>
<p>Would it prevent it from rpc using nccl?</p>
</blockquote>
<p>Which rpc are you referring to? Are you using DDP in conjunction with <code>torch.distributed.rpc</code>?</p>
<blockquote>
<p>Is it sensible to do a forced parameter sync once in a while? Inherent GPU parallelism non-determinism can cause parameter divergence that could cause replica parameter divergence for some sensitive models (even if gradients are sychronized)?</p>
</blockquote>
<p>If there are drifts, then, yes, manually sync once a while would help.</p>
<blockquote>
<p>How does one do it?</p>
</blockquote>
<p>You can use broadcast, see the code linked below. If you would like to calculate average, you can also use <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.all_reduce" rel="noopener nofollow ugc"><code>all_reduce</code></a> and then divide by <code>world_size</code>.</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/3806c939bda0df4c0f38b9d356c004f384535ac1/torch/nn/parallel/distributed.py#L404">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/3806c939bda0df4c0f38b9d356c004f384535ac1/torch/nn/parallel/distributed.py#L404" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/3806c939bda0df4c0f38b9d356c004f384535ac1/torch/nn/parallel/distributed.py#L404" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/3806c939bda0df4c0f38b9d356c004f384535ac1/torch/nn/parallel/distributed.py#L404</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="394" style="counter-reset: li-counter 393 ;">
          <li>    self.bucket_bytes_cap = int(bucket_cap_mb * 1024 * 1024)</li>
          <li></li>
          <li>    # Sync params and buffers</li>
          <li>    self._sync_params_and_buffers(authoritative_rank=0)</li>
          <li></li>
          <li>    self._ddp_init_helper()</li>
          <li></li>
          <li>def _sync_params_and_buffers(self, authoritative_rank=0):</li>
          <li>    module_states = list(self.module.state_dict().values())</li>
          <li>    if len(module_states) &gt; 0:</li>
          <li class="selected">        self._distributed_broadcast_coalesced(</li>
          <li>            module_states,</li>
          <li>            self.broadcast_bucket_size,</li>
          <li>            authoritative_rank)</li>
          <li></li>
          <li>def _ddp_init_helper(self):</li>
          <li>    """</li>
          <li>    Initialization helper function that does the following:</li>
          <li></li>
          <li>    (1) replicating the module from device[0] to the other devices</li>
          <li>    (2) bucketing the parameters for reductions</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<blockquote>
<p>When does it make sense to do NUMA node pinning? CPU affinity pinning?</p>
</blockquote>
<p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a>, do you know what’s the best practice here?</p>

vadimkantorov: <p>Thanks a lot <a class="mention" href="/u/mrshenli">@mrshenli</a> for these responses! Look like it should be in some official guide <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p><a class="mention" href="/u/mrshenli">@mrshenli</a> I’m having troubles launching the most basic two-node distributed configuration (I checked, TCP connection with nc works ok). It doesn’t seem to respect the passed port. If you could take a look, it would be awesome!</p>
<p><strong>UPD:</strong> I created an issue to discuss this: <a href="https://github.com/pytorch/pytorch/issues/44544" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/44544</a></p>
<pre><code class="lang-python">import os
import torch
import argparse
import torch.distributed as dist

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--backend', default='gloo')
    parser.add_argument('--rank', type=int, default=0)
    parser.add_argument('--world-size', type=int, default=1)
    args = parser.parse_args()
    dist.init_process_group(args.backend, init_method="env://", rank=args.rank, world_size=args.world_size)
    print(f"Master node {os.environ['MASTER_ADDR']}:{os.environ['MASTER_PORT']}. Rank {args.rank}. World size: {args.world_size}")
    test_tensor = torch.tensor(args.rank+1)
    if args.backend == 'nccl':
        test_tensor = test_tensor.cuda()
    dist.all_reduce(test_tensor, op=dist.ReduceOp.SUM)
    print(f"Test value: {test_tensor.item()}, expected: {sum(range(args.world_size+1))}")
</code></pre>
<pre><code class="lang-auto">Test standalone

Node 1
Input: 
GLOO_SOCKET_IFNAME=team0 MASTER_ADDR=10.81.13.54 MASTER_PORT=12345 python distributed_example.py
Output: 
Master node 10.81.13.54:12345. Rank 0. World size: 1
Test value: 1, expected: 1

Node 2
Input: 
GLOO_SOCKET_IFNAME=team0 MASTER_ADDR=10.81.13.51 MASTER_PORT=12345 python distributed_example.py 
Output: 
Master node 10.81.13.51:12345. Rank 0. World size: 1
Test value: 1, expected: 1

Test disctibuted Gloo

Node 1
Input: 
GLOO_SOCKET_IFNAME=team0 MASTER_ADDR=10.81.13.54 MASTER_PORT=12345 python distributed_example.py --rank 0 --world-size 2
Output: 
Traceback (most recent call last):
  File "distributed_example.py", line 11, in &lt;module&gt;
    dist.init_process_group('gloo', init_method="env://", rank=args.rank, world_size=args.world_size)
  File "/miniconda3/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 425, in init_process_group
    _default_pg = _new_process_group_helper(
  File "/miniconda3/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 499, in _new_process_group_helper
    pg = ProcessGroupGloo(
RuntimeError: [/opt/conda/conda-bld/pytorch_1595629411241/work/third_party/gloo/gloo/transport/tcp/pair.cc:769] connect [10.81.13.51]:11169: No route to host

Node 2
Input:
GLOO_SOCKET_IFNAME=team0 MASTER_ADDR=10.81.13.54 MASTER_PORT=12345 python distributed_example.py --rank 1 --world-size 2
Output:

Test distributed NCCL

Node 1
Input:
NCCL_SOCKET_IFNAME=team0 NCCL_DEBUG=INFO MASTER_ADDR=10.81.13.54 MASTER_PORT=12345 python distributed_example.py --rank 0 --world-size 2 --backend nccl
Output:
Master node 10.81.13.54:12345. Rank 0. World size: 2
srs-ds11:64570:64570 [0] NCCL INFO Bootstrap : Using [0]team0:10.81.13.54&lt;0&gt;
srs-ds11:64570:64570 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).

srs-ds11:64570:64570 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
srs-ds11:64570:64570 [0] NCCL INFO NET/Socket : Using [0]team0:10.81.13.54&lt;0&gt;
NCCL version 2.4.8+cuda10.1
srs-ds11:64570:65266 [0] NCCL INFO Setting affinity for GPU 0 to 55,55555555

Node 2
Input:
NCCL_SOCKET_IFNAME=team0 NCCL_DEBUG=INFO MASTER_ADDR=10.81.13.54 MASTER_PORT=12345 python distributed_example.py --rank 1 --world-size 2 --backend nccl
Output:
Master node 10.81.13.54:12345. Rank 1. World size: 2
srs-ds8:192240:192240 [0] NCCL INFO Bootstrap : Using [0]team0:10.81.13.51&lt;0&gt;
srs-ds8:192240:192240 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).

srs-ds8:192240:192240 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
srs-ds8:192240:192240 [0] NCCL INFO NET/Socket : Using [0]team0:10.81.13.51&lt;0&gt;
srs-ds8:192240:192316 [0] NCCL INFO Setting affinity for GPU 0 to 55,55555555

srs-ds8:192240:192316 [0] include/socket.h:390 NCCL WARN Connect to 10.81.13.54&lt;34419&gt; failed : No route to host
srs-ds8:192240:192316 [0] NCCL INFO bootstrap.cc:100 -&gt; 2
srs-ds8:192240:192316 [0] NCCL INFO bootstrap.cc:326 -&gt; 2
srs-ds8:192240:192316 [0] NCCL INFO init.cc:695 -&gt; 2
srs-ds8:192240:192316 [0] NCCL INFO init.cc:951 -&gt; 2
srs-ds8:192240:192316 [0] NCCL INFO misc/group.cc:69 -&gt; 2 [Async thread]
Traceback (most recent call last):
  File "distributed_example.py", line 17, in &lt;module&gt;
    dist.all_reduce(test_tensor, op=dist.ReduceOp.SUM)
  File "/miniconda3/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 936, in all_reduce
    work = _default_pg.allreduce([tensor], opts)
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1595629411241/work/torch/lib/c10d/ProcessGroupNCCL.cpp:518, unhandled system error, NCCL version 2.4.8
</code></pre>

mrshenli: <p>For the glooo case, I noticed that the master address for two nodes are different, is this a typo?</p>
<pre><code class="lang-auto">node 1: MASTER_ADDR=10.81.13.54
node 2: MASTER_ADDR=10.81.13.51
</code></pre>

aabugaliev: <p>No it doesn’t. Addresses are different because it is standalone test (one node/one worker) for sanity check. Theare are three cases: single node test, multi node test with gloo backend and with nccl backend.</p>