Do DataParallel and DistributedDataParallel affect the batch size and GPU memory consumption?

username: mdc
cooked: <p>Do DataParallel and DistributedDataParallel affect the batch size and GPU memory consumption?<br>
(I use NCCL backend).</p>
<ol>
<li>
<p>If I set batch_size=4 and train with nn.DataParallel or nn.DistributedDataParallel on 8 GPUs, then what will be the batch-size and mini_batch_size: 4, 8, or 32?</p>
</li>
<li>
<p>Can I use batch_size lower than number of GPUs, batch_size=4 for 8xGPUs (will it lead to error, or will be used only 4 GPUs or will be batch_size increased to 8 or 32)?</p>
</li>
<li>
<p>I tried to train EfficientNet-L2 by using each of nn.DataParallel and nn.DistributedDataParallel, but with nn.DataParallel I can use batch_size 2x higher than with nn.DistributedDataParallel without CUDA Out of memory. Does nn.DistributedDataParallel spend 2x time more GPU memory than nn.DataParallel?</p>
</li>
</ol>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/mdc">@mdc</a></p>
<blockquote>
<ol>
<li>If I set batch_size=4 and train with nn.DataParallel or nn.DistributedDataParallel on 8 GPUs, then what will be the batch-size and mini_batch_size: 4, 8, or 32?</li>
</ol>
</blockquote>
<p>The <code>batch_size</code> var is usually a per-process concept. As <code>DataParallel</code> is single-process multi-threads, setting <code>batch_size=4</code> will make 4 the real batch size. The per-thread batch-size will be <code>4/num_of_devices</code>. However, as these threads accumulate grads into the same <code>param.grad</code> field, the per-threads batch-size shouldn’t make any differences.</p>
<p>For <code>DistributedDataParallel</code> (DDP), as it is multi-process training, if you set <code>batch_size=4</code> for each process, the real batch_size will be <code>4 * world_size</code>. One caveat is that, DDP uses <code>AllReduce</code> to calculate the average (instead of sum) gradients across processes.</p>
<blockquote>
<ol start="2">
<li>Can I use batch_size lower than number of GPUs, batch_size=4 for 8xGPUs (will it lead to error, or will be used only 4 GPUs or will be batch_size increased to 8 or 32)?</li>
</ol>
</blockquote>
<p>It should work, but will not fully utilize all devices. If <code>batch_size=4</code>, IIUC, it can at most use 4 GPUs.</p>
<blockquote>
<ol start="3">
<li>I tried to train EfficientNet-L2 by using each of nn.DataParallel and nn.DistributedDataParallel, but with nn.DataParallel I can use batch_size 2x higher than with nn.DistributedDataParallel without CUDA Out of memory. Does nn.DistributedDataParallel spend 2x time more GPU memory than nn.DataParallel?</li>
</ol>
</blockquote>
<p>DDP allocates dedicated CUDA buffer as communication buckets, so it will use more CUDA memory than DP. But it is not 2X compared to DP. The total comm bucket size is the same as the model size.</p>

username: AMellinger
cooked: <aside class="quote no-group quote-modified" data-username="mrshenli" data-post="2" data-topic="97194">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<blockquote>
<ol start="2">
<li>Can I use batch_size lower than number of GPUs, batch_size=4 for 8xGPUs (will it lead to error, or will be used only 4 GPUs or will be batch_size increased to 8 or 32)?</li>
</ol>
</blockquote>
<p>It should work, but will not fully utilize all devices. If <code>batch_size=4</code>, IIUC, it can at most use 4 GPUs.</p>
</blockquote>
</aside>
<p>Greetings!  I’d like some clarifications on this. Is this response referring to DP or DDP?  If DDP then isn’t batch_size per-process? Meaning if one sets <code>batch_size=4</code> in the DataLoader then isn’t that 4 sample per process/gpu? How does this turn into ‘it can at most use 4 GPUs?’</p>
<p>I guess I have always been confused by the DDP statement of “The batch size should be larger than the number of GPUs used locally.” because we are setting the batch_size per process/gpu not batch_size for the entire sets of gpus in aggregate. Or does “batch size” have two different meanings?</p>
<p>Thanks for any help!</p>

username: ishaan-mehta
cooked: <p>This is also something I have been confused about. I don’t understand how the number of GPUs should have any effect on batch size selection in DDP given that the specified batch size should be for each GPU/process. I would definitely appreciate some clarification if possible <a class="mention" href="/u/mrshenli">@mrshenli</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group quote-modified" data-username="AMellinger" data-post="3" data-topic="97194" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/amellinger/48/36934_2.png" class="avatar"> AMellinger:</div>
<blockquote>
<aside class="quote no-group quote-modified" data-username="mrshenli" data-post="2" data-topic="97194">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<blockquote>
<ol start="2">
<li>Can I use batch_size lower than number of GPUs, batch_size=4 for 8xGPUs (will it lead to error, or will be used only 4 GPUs or will be batch_size increased to 8 or 32)?</li>
</ol>
</blockquote>
<p>It should work, but will not fully utilize all devices. If <code>batch_size=4</code>, IIUC, it can at most use 4 GPUs.</p>
</blockquote>
</aside>
<p>Greetings!  I’d like some clarifications on this. Is this response referring to DP or DDP?  If DDP then isn’t batch_size per-process? Meaning if one sets <code>batch_size=4</code> in the DataLoader then isn’t that 4 sample per process/gpu? How does this turn into ‘it can at most use 4 GPUs?’</p>
<p>I guess I have always been confused by the DDP statement of “The batch size should be larger than the number of GPUs used locally.” because we are setting the batch_size per process/gpu not batch_size for the entire sets of gpus in aggregate. Or does “batch size” have two different meanings?</p>
<p>Thanks for any help!</p>
</blockquote>
</aside>