Strange number of processes per GPU

username: aleSuglia
cooked: <p>I have a problem very similar to <a href="https://discuss.pytorch.org/t/a-strange-problem-about-the-gpu-processes-and-num-workers/9689" class="inline-onebox">A strange problem about the gpu processes and num_workers</a></p>
<p>In particular the following happens and I don’t understand why there are extra processes per GPU that are consuming a lot of GPU ram:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0ced5adce72553c63d0d844ccbdbf7d32e41250a.png" data-download-href="https://discuss.pytorch.org/uploads/default/0ced5adce72553c63d0d844ccbdbf7d32e41250a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0ced5adce72553c63d0d844ccbdbf7d32e41250a.png" alt="image" data-base62-sha1="1Qmg8nxO2nr3cBqjIDFUcdUbpSa" width="346" height="500" data-dominant-color="131313"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">556×802 23.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Could you please advise what could be the problem here?</p>
<p>I’m using PyTorch-Lightning for training using multiple GPUs. I have 4 V100 GPUs and the following dependencies:</p>
<ol>
<li>torch==1.7.1</li>
<li>torchvision==0.8.2</li>
<li>CUDA 11.0</li>
</ol>

username: ptrblck
cooked: <p>I guess that you are using a distributed training setup (via <code>DDP</code>).<br>
If that’s the case, it seems that the setup is incorrect, as apparently each process creates a new CUDA context on each device. I’m not familiar with Lightning, but you could check, if your script only uses the specified GPU and doesn’t allocate tensors on all visible devices.</p>

username: aleSuglia
cooked: <p>How can I check this specifically? Shall I check whether the tensors are moved to specific devices? Any suggestions?</p>

username: ptrblck
cooked: <p>You could check, if all devices are visible in each process and check then for <code>to('cuda')</code> or <code>cuda()</code> calls.<br>
If these operations are used, the tensor or module would be moved to the default device (<code>GPU:0</code>) in each process and could thus create multiple CUDA contexts.</p>

username: Lanezzz
cooked: <p>I have met a similar problem.And I am learning how to use ‘DistributedDataParallel’.And it’s really strange that why there are extre processes per GPU that are consuming 0 GPU ram.</p>
<p>To reproduce:<a href="https://github.com/yangkky/distributed_tutorial/blob/master/src/mnist.py" class="inline-onebox" rel="noopener nofollow ugc">distributed_tutorial/src/mnist.py at master · yangkky/distributed_tutorial · GitHub</a><br>
I follow the code completely and only had three modifications.<br>
` parser.add_argument(‘-g’, ‘–gpus’, default=3, type=int,<br>
help=‘number of gpus per node’)</p>
<p>os.environ[‘MASTER_ADDR’] = ‘127.0.0.1’<br>
os.environ[‘MASTER_PORT’] = ‘23556’`</p>
<p>And my Command line argument was ‘python mnist.py -n 1 -g 3 -nr 0’<br>
PID 6283 6284 6285 are my processes.And GPU 3 is being used by others<br>
my dependencies:  pytorch == 1.6.0    torchvision == 0.7.0  CUDA 10.2           3 x RTX 2080Ti</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/7/67bfe4b1d8571c0e82c36b7c3b38cf130feb793b.png" data-download-href="https://discuss.pytorch.org/uploads/default/67bfe4b1d8571c0e82c36b7c3b38cf130feb793b" title="gpu222"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/7/67bfe4b1d8571c0e82c36b7c3b38cf130feb793b.png" alt="gpu222" data-base62-sha1="eNOj9WtTiDsyOyJA8QsCZKgAYor" width="521" height="500" data-dominant-color="1B1B1B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">gpu222</span><span class="informations">620×594 7.77 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>