nn.DataParallel doesn't automatically use all GPUs

username: Rakshit_Kothari
cooked: <p>I have a cluster of 10 GPUs. The default code,</p>
<blockquote>
<pre><code>    if torch.cuda.device_count() &gt; 1:
        net = nn.DataParallel(net)
</code></pre>
</blockquote>
<p>Uses 8 GPUs. Does PyTorch have a max 8 GPU policy? Training number workers = 1 at the moment but this behavior doesn’t change when I tried 1, 2 or 10.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/496fc4106a38a7a3d2a0f1cebba85e96c7af52b7.png" data-download-href="https://discuss.pytorch.org/uploads/default/496fc4106a38a7a3d2a0f1cebba85e96c7af52b7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/4/496fc4106a38a7a3d2a0f1cebba85e96c7af52b7_2_570x500.png" alt="image" data-base62-sha1="atEjnJFTuZOHwolSeiaPAUJ72Yf" width="570" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/4/496fc4106a38a7a3d2a0f1cebba85e96c7af52b7_2_570x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/4/496fc4106a38a7a3d2a0f1cebba85e96c7af52b7_2_855x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/4/496fc4106a38a7a3d2a0f1cebba85e96c7af52b7.png 2x" data-dominant-color="131313"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">875×767 96.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: spanev
cooked: <p>What’s <code>torch.cuda.device_count()</code> return value?</p>
<p>Have you tried:</p>
<pre><code class="lang-auto">dev_count = torch.cuda.device_count()
if  dev_count &gt; 1:
    net = nn.DataParallel(net, device_ids=list(range(dev_count)))
</code></pre>
<p>Also <a href="https://github.com/pytorch/pytorch/blob/74fa53995debdb3c11f885c8dd960eb2bb0b0f27/torch/nn/parallel/data_parallel.py#L112" rel="nofollow noopener">looks like</a> 8+ GPUs is not handled very well. So it might from <code>torch.cuda.device_count()</code>. If that works for you (you know the number of GPUs) :</p>
<pre><code class="lang-auto">dev_count = 10
net = nn.DataParallel(net, device_ids=list(range(dev_count)))
</code></pre>

username: ptrblck
cooked: <p>Additionally to what has been said: how large is your batch size and is it divisible by the number of GPUs?</p>

username: Rakshit_Kothari
cooked: <p>Thanks for your reply. The device count is 10. I also tried what you suggested and that didn’t work. <a class="mention" href="/u/ptrblck">@ptrblck</a> was right, silly me, I was using a batchsize of 16. I increased it to 20 and it uses all the GPUs. Thanks guys <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Rakshit_Kothari
cooked: <p>I don’t want to make a new entry for this so just replying to this thread.<br>
MultiGPU keeps crashing with an OSError.</p>
<p>File “ExploreArch.py”, line 245, in <br>
for bt, data in enumerate(trainloader):<br>
File “/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py”, line 819, in <strong>next</strong><br>
return self._process_data(data)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py”, line 846, in _process_data<br>
data.reraise()<br>
File “/usr/local/lib/python3.6/dist-packages/torch/_utils.py”, line 369, in reraise<br>
raise self.exc_type(msg)<br>
OSError: Caught OSError in DataLoader worker process 1.<br>
Original Traceback (most recent call last):<br>
File “/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py”, line 178, in _worker_loop<br>
data = fetcher.fetch(index)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py”, line 44, in fetch<br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py”, line 44, in <br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “ExploreArch.py”, line 102, in <strong>getitem</strong><br>
OSError: [Errno 5] Input/output error: </p>
<p>This system has 10 P4 cards. I’ve made the following observations:<br>
With num_workers=0, there are no crashes. However, it slows down the processing.<br>
With num_workers=20, I get the exact same error but on a much earlier epoch (say, 2)<br>
With num_workers=10, the code ran smoothly until epoch 34 or so and then crashed.</p>
<p>I’ve seen a few links but the solution isn’t readily obvious? Any suggestions? If not, I’ll move this with num_workers = 0.</p>

username: ptrblck
cooked: <p>The error points to the file read operation, which is a bit weird.<br>
I assume you are reading some files in the <code>__getitem__</code> method and load the data.<br>
Which kind of files are you using and are you using any custom <code>collate_fn</code> or sampling?</p>

username: Rakshit_Kothari
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, sorry for the late response. It’s a simple imread of PNG files in <code>__getitem__</code>. I use the default collate function which PyTorch provides. An alternative explanation could be that the file IO speed is low as opposed to processor but I’m guessing PyTorch would have locking mechanisms to handle such situations.</p>
<p>It also seems the master GPU, default <code>device:0</code> has higher memory requirements because it is used to perform backprop(?). Would you advice a system with, say 8GB GPUsX9 and the 10th one as 16 GB. That way we can maximize the use across the rest of the GPUs. Any advice would be greatly appreciated!</p>

username: KIMBREL_DIAS
cooked: <p><a class="mention" href="/u/rakshit_kothari">@Rakshit_Kothari</a> did you get the solution to it? as I’m also facing the same issue</p>

username: Rakshit_Kothari
cooked: <p>Sorry for the late reply. This is a known issue and the google query you’re looking for is <em>GPU memory imbalance</em>. The correct answer is to choose a batchsize which is a multiple of 10. For instance, 10 GPUs should mean a batchsize of 10 or 20, 30 … so on.</p>
<p>Also, you’ll need to increase the number of workers to the capability of your system. Note that there is very little runtime benefit to having 10 GPUs (as my network admin learned after a gazillion hang ups). Data I/O will usually be the bottleneck. The only advantage in a 10 GPU system would be the insanely large batch size, i.e, higher learning rates.</p>