A question concerning batchsize and multiple GPUs in Pytorch

username: Shisho_Sama
cooked: <p>If I set batch-size to 256 and use all of the GPUs on my system (lets say I have 8), will each GPU get a batch of 256 or will it get <code>256//8</code> ?<br>
If my memory serves me correctly, in <code>Caffe</code>, all GPUs would get the same batch-size , i.e 256 and the effective batch-size would be 8*256 , 8 being the number of GPUs and 256 being the batch-size.<br>
Is the outcome/answer any different when using .<code>cuda()</code> (with no parameters) and when using <code>torch.nn.DataParallel(model, device_ids=args.gpus)</code> (specifying which gpu ids to use)?</p>
<p>Thanks a lot in advance</p>

username: ptrblck
cooked: <p><code>nn.DataParallel</code> splits the data along the batch dimension so that each specified GPU will get a chunk of the batch. If you just call <code>.cuda()</code> (or the equivalent <code>.to()</code> call), you will push the tensor or parameters onto the specified single device.</p>

username: AlexLuya
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>,if nn.DataParallel used and batch size is 256 in experimental single GPU machine,now moving to a machine with n gpus(same type of gpu),should batch size be changed to n*256?<br>
How about if DistributedDataParallel used?</p>

username: ptrblck
cooked: <p>I would generally recommend to use <code>DistributedDataParallel</code>, even for a single machine with multiple GPUs.</p>
<p>Yes, to get a potential performance increase, you should try to scale the batch size.</p>

username: 111204
cooked: <p>hello  ptrblck, are the chunks equal in each specified GPU ? If not, are there any solution to make the chunks for different GPUS be more flexible.</p>

username: ptrblck
cooked: <p>No, each GPU should get a part of the batch without repetition.</p>
<aside class="quote no-group" data-username="111204" data-post="6" data-topic="33767">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111204/48/18814_2.png" class="avatar"> 111204:</div>
<blockquote>
<p>If not, are there any solution to make the chunks for different GPUS be more flexible.</p>
</blockquote>
</aside>
<p>I’m not sure I understand this question properly. Could you explain a bit, what you mean by “more flexible”?</p>

username: 111204
cooked: <p>Sorry for the confusion. I mean could we give more chunks to one gpu while give less for another cause the available memory is not the same for all the gpus at some time. Thanks for your reply!</p>

username: ptrblck
cooked: <p>That should be generally possible, e.g. by using <a href="https://github.com/pytorch/pytorch/blob/1bb6c51421e904c7a5487544c21b13b5df75d1f3/torch/nn/parallel/parallel_apply.py#L22">parallel_apply</a> as the base method and changing the splits internally or try to create a manual data parallel approach.</p>
<p>However, I’m really unsure, if you’ll see any significant speedup and how large the workload would be.</p>

username: gmondaut
cooked: <p>Hi,<br>
I am having a similar issue here. If I use 1 GPU with batch_size 128, the job works fine. When I use 2 GPUs with batch_size 256 I am getting this error:<br>
‘RuntimeError: CUDA out of memory. Tried to allocate 192.00 MiB (GPU 1; 15.75 GiB total capacity…’</p>
<p>I am using DistributedDataParallel as per below:</p>
<p>‘torch.distributed.init_process_group(backend=‘nccl’)<br>
model = nn.parallel.DistributedDataParallel(model, device_ids = list(range(n_gpu))[::-1],find_unused_parameters=True)’</p>
<p>and I am running the job with : python3 -m torch.distributed.launch main.py</p>
<p>Thanks in advanced<br>
Giorgio</p>

username: ptrblck
cooked: <p>I guess the default device (GPU0) might run out of memory, as your work flow seems to be close to <code>nn.DataParallel</code> as described <a href="https://discuss.pytorch.org/t/distributeddataparallel-with-single-process-slower-than-sing-gpu/66495/2">here</a>.</p>
<p>Could you try to use the recommend use case of one device/replica per DDP process as described <a href="https://discuss.pytorch.org/t/data-splitting-in-distributeddataparallel/66682/3">here</a>?</p>

username: isalirezag
cooked: <p>can you provide a short example on how to do that? <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.parallel.DistributedDataParallel" rel="nofollow noopener">this</a> does not help on how to do use DistributedDataParallel.<br>
Using DataParallel alone is easy but then it has the problem that others mentioned. i.e if one gpu can run 256 batches, using Dataparallel for 2 gpus does not let us use 2*256.</p>

username: ptrblck
cooked: <p><a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html?highlight=distributeddataparallel">This</a> and <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html?highlight=distributeddataparallel">this</a> tutorial should give you a good starter.</p>