Problem in deploying DDP

leonleeldc: <p>Hey, I am using DDP for model training. In the script, I add a line as,<br>
dist.init_process_group(backend=“nccl”) besides using model = DDP(model)</p>
<p>Then, I ran the script in command line as,<br>
python -m torch.distributed.launch --nproc_per_node=8 --nnode=1 --node_rank=0 paraphrase_simpletransformers.py</p>
<p>But then it will say OOM after it is running. I checked nvidia-smi, I found that each pid must have one gpu and also occupy the 0th gpu. That is why OOM happens. Not sure why this happens? May you give help here, thanks in advance.</p>
<p>±------------------------------±---------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------+<br>
| Processes:                                                       GPU Memory |<br>
|  GPU       PID   Type   Process name                             Usage      |<br>
|=============================================================================|<br>
|    0     82610      C   …e/anaconda3/envs/pytorch_p36/bin/python  3071MiB |<br>
|    0     82611      C   …e/anaconda3/envs/pytorch_p36/bin/python   409MiB |<br>
|    0     82612      C   …e/anaconda3/envs/pytorch_p36/bin/python   409MiB |<br>
|    0     82614      C   …e/anaconda3/envs/pytorch_p36/bin/python   403MiB |<br>
|    0     82615      C   …e/anaconda3/envs/pytorch_p36/bin/python   409MiB |<br>
|    0     82616      C   …e/anaconda3/envs/pytorch_p36/bin/python   411MiB |<br>
|    0     82617      C   …e/anaconda3/envs/pytorch_p36/bin/python   405MiB |<br>
|    0     82618      C   …e/anaconda3/envs/pytorch_p36/bin/python   409MiB |<br>
|    1     82611      C   …e/anaconda3/envs/pytorch_p36/bin/python  1517MiB |<br>
|    2     82612      C   …e/anaconda3/envs/pytorch_p36/bin/python  1517MiB |<br>
|    3     82614      C   …e/anaconda3/envs/pytorch_p36/bin/python  1497MiB |<br>
|    4     82615      C   …e/anaconda3/envs/pytorch_p36/bin/python  1497MiB |<br>
|    5     82616      C   …e/anaconda3/envs/pytorch_p36/bin/python  1537MiB |<br>
|    6     82617      C   …e/anaconda3/envs/pytorch_p36/bin/python  1537MiB |<br>
|    7     82618      C   …e/anaconda3/envs/pytorch_p36/bin/python  1537MiB |<br>
±----------------------------------------------------------------------------+</p>

leonleeldc: <p>I also tried to change the number of npro_per-node=4 as following,<br>
python -m torch.distributed.launch --nproc_per_node=4 --nnode=1 --node_rank=0 paraphrase_simpletransformers.py</p>
<p>Then, I did nvidia-smi, it becomes,<br>
±----------------------------------------------------------------------------+<br>
| Processes:                                                       GPU Memory |<br>
|  GPU       PID   Type   Process name                             Usage      |<br>
|=============================================================================|<br>
|    0     85910      C   …e/anaconda3/envs/pytorch_p36/bin/python  4925MiB |<br>
|    0     85911      C   …e/anaconda3/envs/pytorch_p36/bin/python  3175MiB |<br>
|    0     85912      C   …e/anaconda3/envs/pytorch_p36/bin/python  3175MiB |<br>
|    0     85913      C   …e/anaconda3/envs/pytorch_p36/bin/python  3175MiB |<br>
|    1     85910      C   …e/anaconda3/envs/pytorch_p36/bin/python  1421MiB |<br>
|    1     85911      C   …e/anaconda3/envs/pytorch_p36/bin/python  1393MiB |<br>
|    2     85910      C   …e/anaconda3/envs/pytorch_p36/bin/python  1431MiB |<br>
|    2     85912      C   …e/anaconda3/envs/pytorch_p36/bin/python  1465MiB |<br>
|    3     85910      C   …e/anaconda3/envs/pytorch_p36/bin/python  1433MiB |<br>
|    3     85913      C   …e/anaconda3/envs/pytorch_p36/bin/python  1465MiB |<br>
|    4     85910      C   …e/anaconda3/envs/pytorch_p36/bin/python  1431MiB |<br>
|    5     85910      C   …e/anaconda3/envs/pytorch_p36/bin/python  1433MiB |<br>
|    6     85910      C   …e/anaconda3/envs/pytorch_p36/bin/python  1433MiB |<br>
|    7     85910      C   …e/anaconda3/envs/pytorch_p36/bin/python  1433MiB |<br>
±----------------------------------------------------------------------------+</p>

osalpekar: <p>Can you set the env var <code>CUDA_VISIBLE_DEVICES</code> and specify which GPUs to use before launching training? This will at least clarify whether this is a problem with processes binding to GPUs or something else.</p>
<p>Here’s a good explanation of the env var: <a href="https://stackoverflow.com/questions/39649102/how-do-i-select-which-gpu-to-run-a-job-on" class="inline-onebox" rel="noopener nofollow ugc">cuda - How do I select which GPU to run a job on? - Stack Overflow</a></p>