DistributedDataParallel, got more than one processes in one GPU

username: Si_laopi
cooked: <p>I am trying to apply distributed training in my project, but got a strange problem. It seems all data were loaded to one gpu, please see the figure below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4c5dee23d6eebab04d4d5dd0ea9706f1e4fd3c16.png" data-download-href="https://discuss.pytorch.org/uploads/default/4c5dee23d6eebab04d4d5dd0ea9706f1e4fd3c16" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c5dee23d6eebab04d4d5dd0ea9706f1e4fd3c16_2_475x500.png" alt="image" data-base62-sha1="aTzx0mEKZDho3VXIjc08Yc1K1ZI" width="475" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c5dee23d6eebab04d4d5dd0ea9706f1e4fd3c16_2_475x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c5dee23d6eebab04d4d5dd0ea9706f1e4fd3c16_2_712x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/c/4c5dee23d6eebab04d4d5dd0ea9706f1e4fd3c16.png 2x" data-dominant-color="141414"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">802×843 104 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>By the way, I am using pytorch 1.5, Ubuntu 18.04.</p>

username: mrshenli
cooked: <p>Looks like all processes create some states on <code>cuda:5</code>. It’s possible that some part of the code is using default device. One option is to use <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.set_device" rel="nofollow noopener"><code>set_device</code></a> to set the current device before creating any context/data on the device. Another option is to set <code>CUDA_VISIBLE_DEVICES</code> to make sure that each process can only see one device.</p>
<p>BTW, could you please add a “distributed” tag for distributed-training related questions? Developers working on the distributed package actively monitors that tag.</p>

username: Si_laopi
cooked: <p>I did set <code>CUDA_VISIBLE_DEVICES</code> in my project, and it seems doesn’t work. And I couldn’t find a way to add tags afterwards.</p>

username: mrshenli
cooked: <p>How did you set <code>CUDA_VISIBLE_DEVICES</code>?  Did you set it in the very beginning of every spawned subprocess?</p>
<p>One way to verify is to add the following line right before where <code>init_process_group</code> is called in each subprocess.</p>
<pre><code class="lang-auto">print(os.environ['CUDA_VISIBLE_DEVICES'])
</code></pre>

username: Si_laopi
cooked: <p>Not the very beginning, but before <code>mp.spawn</code> was called. And I did what you say, the result is here (<code>mp.spaw</code>n is called in <code>launch.launch</code>). I think the correct result should be one single number at each time, but how to fix it?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/5/d5d6279b103da67cdfcd5209c2d5db419a23b05e.png" data-download-href="https://discuss.pytorch.org/uploads/default/d5d6279b103da67cdfcd5209c2d5db419a23b05e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d5d6279b103da67cdfcd5209c2d5db419a23b05e_2_690x334.png" alt="image" data-base62-sha1="uvGwAeyWPe2FjubreZSpkDwaI1w" width="690" height="334" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d5d6279b103da67cdfcd5209c2d5db419a23b05e_2_690x334.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d5d6279b103da67cdfcd5209c2d5db419a23b05e_2_1035x501.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d5d6279b103da67cdfcd5209c2d5db419a23b05e_2_1380x668.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1852×899 239 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: mrshenli
cooked: <p>Is the intention to let each process see 3 devices? If each process only works on one device, it’s better to let it see only one device, so that it won’t accidentally create states on other devices?</p>

username: Si_laopi
cooked: <p>My problem has been fixed. Thanks a lot!</p>