Confused about Distributed data parallel behavior

subhashnerella: <p>I am trying to run my model using DDP on a single node with 3 GPUs. I only intend to use two GPUs so I used <code>os.environ["CUDA_VISIBLE_DEVICES"]="1,2"</code>.</p>
<p>I started running the  code with only one process<br>
<code>python -m  torch.distributed.launch --nproc_per_node=1 train.py</code></p>
<p>The code runs but when i check <code>nvidia-smi</code> i see a  processes running on two gpus.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2c202982ca7daf5cd0b2a715ff57c7795a62bab0.png" alt="image" data-base62-sha1="6ilUMj0hTzExvnVyL07HCE12rlK" width="554" height="158"><br>
Process 62966 corresponds to the DDP. Ignore process 2815. I did not understand why the process is running on GPU 2 but without using any resource. The code seems to run alright in this case.</p>
<p>When i run the command with two processes<br>
<code>python -m  torch.distributed.launch --nproc_per_node=2 train.py</code><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f25ec5fdec717438fea967ba65d7cc4dafd751c.png" alt="image" data-base62-sha1="90Dkwq7YEL7Dw1xdK6YSwQ1DApC" width="557" height="191"><br>
I see two processes are created but both the process run on both the gpus. The code does not run and stuck at <code>torch.distributed.barrier()</code>.</p>
<p>my code for init_process_group:</p>
<pre><code class="lang-auto">def init_distributed_mode(args):
    args.rank = int(os.environ["RANK"])
    args.world_size = int(os.environ['WORLD_SIZE'])
    args.gpu = int(os.environ['LOCAL_RANK'])
    args.distributed = True
    torch.cuda.device(args.gpu)
    args.dist_backend = 'nccl'
    torch.distributed.init_process_group(backend=args.dist_backend, init_method=args.dist_url,world_size=args.world_size, rank=args.rank)               
    torch.distributed.barrier()
</code></pre>
<p>How do I limit one process to one gpu and make the code not stuck at torch distributed barrier?</p>

agolynski: <p>Hi,</p>
<p>Could you try torch.cuda.set_device() instead, torch.cuda.device is a context manager, also see <a href="https://github.com/pytorch/pytorch/issues/1608" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/1608</a></p>

subhashnerella: <p>Thanks, the code started running, no longer stuck at the distributed barrier but the <code>nvidia-smi</code> output still doesn’t make sense to me. Two processes are running on both the GPUs.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f3e9c464bf6b8d393cc928dc6cad24b5a8ae133.png" alt="image" data-base62-sha1="91udOQc5iu68OKFTDS6pglANxth" width="555" height="189"><br>
Is this the expected behavior or something’s wrong?</p>

agolynski: <p>How do you initialize DDP, do you provide the correct device to it? e.g.</p>
<p><code>ddp_model = DDP(model, device_ids=[rank])</code></p>

subhashnerella: <pre><code class="lang-auto">args.gpu = int(os.environ['LOCAL_RANK'])
model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu])
</code></pre>

agolynski: <p>I just run a toy example on my machine with 2 processes and i got:</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0    882788      C   ...cal/miniconda3/envs/pytorch3/bin/python   945MiB |
|    1    882789      C   ...cal/miniconda3/envs/pytorch3/bin/python   945MiB |
+-----------------------------------------------------------------------------+
</code></pre>