Distributed training creates multiple processes in GPU0

Soumya_Sanyal: <p>Hi,</p>
<p>I’ve recently started using the distributed training framework for PyTorch and followed the <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" rel="nofollow noopener">imagenet</a> example. I’m using multi-node multi-GPU training. While running the code, during the 1st epoch itself, I see multiple processes starting at GPU 0 of both the servers. They are not present initially when I start the training. From the GPU memory usage, it seems that the other processes are some copy of the model (they all have a fixed usage like 571M). Since running an epoch takes ~12 hours for my use case, debugging step by step is not exactly a feasible solution. I’ve ensured that I pass <code>args.gpu </code> as argument whenever I do a <code>.cuda()</code> call. Also, the model loading/saving is done as suggested in the imagenet example.</p>
<p>Are there any pointers to the probable cause of the issue (or some intelligent ways to debug the code)? Thanks in advance.</p>

mrshenli: <p>Could you please share the cmd you used to launch the processes?</p>
<p>Does the problem disappear if you set <code>CUDA_VISIBLE_DEVICES</code> when launching the process and not passing in <code>--gpu</code> (let it use the default and only visible one)?</p>

Soumya_Sanyal: <p>Hi Shen,</p>
<p>I always set the <code>CUDA_VISIBLE_DEVICES</code> for each run using <code>export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7</code> for example. The code runs on all the 8 GPUs with full utilization, so multiprocessing is surely working. The command I use is as follows  on the two servers I’m using (with appropriate IP and port set):</p>
<pre><code class="lang-auto">python train_distributed.py --dist-url 'tcp://ip:port' --dist-backend 'gloo' --multiprocessing-distributed --world-size 2 --rank 0
python train_distributed.py --dist-url 'tcp://ip:port' --dist-backend 'gloo' --multiprocessing-distributed --world-size 2 --rank 1
</code></pre>

Soumya_Sanyal: <p>Found the bug. So we need to be careful with setting the right GPU context while calling <code>clear_cache()</code> function, otherwise it allocates fixed memory on GPU0 for the other GPUs. Relevant issue <a href="https://github.com/pytorch/pytorch/issues/25752" rel="nofollow noopener">here</a>.</p>

siaimes: <p>I’m having the same problem now, but the sample code doesn’t call clear_cache()</p>

siaimes: <p>Hi, what is your solution to this problem?</p>

Soumya_Sanyal: <p>I had to basically check all functions to see if they are passed the right GPU as the context. So, you should check any function that’s creating some tensors, etc. For my case, the issue was that in the <code>clear_cache()</code> function, I was not passing the GPU context correctly. I think any torch function that accepts a <code>device</code> parameter needs to be called in the right context, otherwise, it might default to GPU0.</p>

siaimes: <p>I also ran the example you mentioned <a href="https://github.com/pytorch/examples/blob/main/imagenet/main.py" rel="noopener nofollow ugc">https://github.com/pytorch/examples/blob/main/imagenet/main.py</a>, but I checked carefully and there was nothing suspicious.</p>