Decreasing GPU usage as increasing number of GPUs for training

username: JVGD
cooked: <p>Hi community.</p>
<p>I am writing you today because I have profiled different models in single GPU and multi GPU and I found that as I increase the number of GPUs for the model to train on the GPU usage of each GPU decreases. I would like to know if this is normal or if there is a way to increase GPU usage in the multi GPU set up.</p>
<p>The profiling I did comprises, different model complexity, different number of DataLoader workers, and different number of GPUs (with DataParallel):</p>
<ul>
<li>Models: Efficiendet-d0, d2, d4 (input resolutions of 512, 768, 1024 to give an idea)</li>
<li>
<span class="hashtag">#Workers:</span> Varying from 1 to <code>batch_size * 3</code>
</li>
<li>
<span class="hashtag">#GPUs:</span> Varying from 1, 2, and 4 RTX 2080 Ti</li>
</ul>
<p>These were my findings:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Model</th>
<th>GPU (x1)</th>
<th>GPU [%]</th>
<th>Workers</th>
<th>Batch size</th>
<th>Batch/s</th>
<th>Image/s</th>
</tr>
</thead>
<tbody>
<tr>
<td>d0</td>
<td>RTX 2080 Ti</td>
<td>90-93</td>
<td>4</td>
<td>8</td>
<td>4.48</td>
<td>35.8</td>
</tr>
<tr>
<td>d2</td>
<td>RTX 2080 Ti</td>
<td>91-92</td>
<td>1</td>
<td>3</td>
<td>3.02</td>
<td>9.06</td>
</tr>
<tr>
<td>d4</td>
<td>RTX 2080 Ti</td>
<td>93-96</td>
<td>1</td>
<td>1</td>
<td>1.85</td>
<td>1.85</td>
</tr>
</tbody>
</table>
</div><div class="md-table">
<table>
<thead>
<tr>
<th>Model</th>
<th>GPU (x2)</th>
<th>GPU [%]</th>
<th>Workers</th>
<th>Batch size</th>
<th>Batch/s</th>
<th>Image/s</th>
</tr>
</thead>
<tbody>
<tr>
<td>d0</td>
<td>RTX 2080 Ti</td>
<td>70-89</td>
<td>24</td>
<td>16</td>
<td>3.35</td>
<td>53.6</td>
</tr>
<tr>
<td>d2</td>
<td>RTX 2080 Ti</td>
<td>72-83</td>
<td>6</td>
<td>6</td>
<td>2.33</td>
<td>13.98</td>
</tr>
<tr>
<td>d4</td>
<td>RTX 2080 Ti</td>
<td>80-92</td>
<td>1</td>
<td>2</td>
<td>1.50</td>
<td>3</td>
</tr>
</tbody>
</table>
</div><div class="md-table">
<table>
<thead>
<tr>
<th>Model</th>
<th>GPU (x4)</th>
<th>GPU [%]</th>
<th>Workers</th>
<th>Batch size</th>
<th>Batch/s</th>
<th>Image/s</th>
</tr>
</thead>
<tbody>
<tr>
<td>d0</td>
<td>RTX 2080 Ti</td>
<td>45-62</td>
<td>8</td>
<td>32</td>
<td>2.28</td>
<td>72.96</td>
</tr>
<tr>
<td>d2</td>
<td>RTX 2080 Ti</td>
<td>58-70</td>
<td>6</td>
<td>12</td>
<td>1.78</td>
<td>21.36</td>
</tr>
<tr>
<td>d4</td>
<td>RTX 2080 Ti</td>
<td>65-82</td>
<td>2</td>
<td>4</td>
<td>1.16</td>
<td>4.64</td>
</tr>
</tbody>
</table>
</div><p>The conclusions are:</p>
<ul>
<li>
<span class="hashtag">#Workers:</span> increasing number of workers translates to increase GPU usage on simple model EfficientDet-d0, but when working with EfficientDet-d2,d4 the <span class="hashtag">#workers</span> had little to no influence</li>
<li>Model complexity: The more complex the model is the more GPU uses</li>
<li>
<span class="hashtag">#GPUs:</span> Increasing the number of GPUs available for a model to train translates in decreasing the GPU usage that the model uses on each of those GPU</li>
</ul>
<p>While the first two points make sense to me, I find it hard to understand the third point. Why increasing the number of GPUs decrease the usage? I know that when using DataParallel each splitted mini-batch needs to be synchronized to compute the loss, so it makes sense to see a reduction in GPU usage as this aggregation might be performed on just one GPU or in CPU. However I expected this GPU usage reduction was going to be constant (so a bit less GPU usage because of the DataParallel), but it turns out it inversely proportional to the number of GPU used…</p>
<p>Has anyone experienced this issue before? Can anyone explain this behavior?</p>
<p>Thanks!</p>

username: vibe2
cooked: <p>It looks like you’re not quite getting linear scaling with the number of GPUs…one idea is to try DistributedDataParallel(), which is supposed to be more performant than DataParallel()</p>
<p>I would also profile your code with torch.utils.bottleneck and see if any optimizations help</p>
<p>Finally I would try this multi-epoch dataloader if you observe slowness at the start of each epoch (which would reduce the average images/second in the multi-GPU case): <a href="https://github.com/pytorch/pytorch/issues/15849" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/15849</a></p>

username: JVGD
cooked: <p>Thank you I am going to review the code with the help of <code>torch.utils.bottleneck</code> and see if I can get the linear scaling</p>