How to split the training batch into different sizes before feeding into multi GPUs?

username: jia_lee
cooked: <p>Science the device whose index is 0 will hold the network parameters, how to split the training batches differently to maximize the usage of GPUs? In my case, the GPUs except the device[0] has low GPU-Util.<br>
How to fix this problem? Thank you.</p>

username: jia_lee
cooked: <p>Any help is welcome.</p>

username: vabh
cooked: <p>Hi,<br>
Maybe you can use ‘nn.DataParallel’? <a href="https://pytorch.org/docs/stable/nn.html#dataparallel" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#dataparallel</a></p>

username: jia_lee
cooked: <p>Thank you for your advice. The problem is dividing the training samples into equal batches leading to unbalance GPU utility. The default GPU whose index is 0 usually has obvious more memory occupation. So, I think we should split the batches manually to maximize the utilities of multi GPUs.</p>

username: zzsunshine
cooked: <p>Hi <a class="mention" href="/u/jia_lee">@jia_lee</a> Have you solved this problem? Thanks.</p>

username: jia_lee
cooked: <p>Not totally. But I warped the loss into the network to alleviate this issue to some extent, look at this link:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/jialee93/Improved-Body-Parts/blob/e9868bbaed2114022bb3094d4a4fa220932e98e3/train_parallel.py#L145" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/jialee93/Improved-Body-Parts/blob/e9868bbaed2114022bb3094d4a4fa220932e98e3/train_parallel.py#L145" target="_blank" rel="nofollow noopener">jialee93/Improved-Body-Parts/blob/e9868bbaed2114022bb3094d4a4fa220932e98e3/train_parallel.py#L145</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="135" style="counter-reset: li-counter 134 ;">
<li>
</li>
<li># target tensor shape: [8,512,512,3], [8, 1, 128,128], [8,43,128,128], [8,36,128,128], [8,36,128,128]</li>
<li>images, mask_misses, heatmaps = target_tuple  # , offsets, mask_offsets</li>
<li># images = Variable(images)</li>
<li># loc_targets = Variable(loc_targets)</li>
<li># conf_targets = Variable(conf_targets)</li>
<li>
</li>
<li>optimizer.zero_grad()  # zero the gradient buff</li>
<li>
</li>
<li>loss_ngpu = posenet(target_tuple)  # reduce losses of all GPUs on cuda 0</li>
<li class="selected">loss = torch.sum(loss_ngpu) / opt.batch_size</li>
<li># print(loc_preds.requires_grad)</li>
<li># print(conf_preds.requires_grad)</li>
<li>if loss.item() &gt; 1e6:</li>
<li>    print("\nLoss is abnormal, drop this batch !")</li>
<li>    loss.zero_()</li>
<li>    continue</li>
<li># print(loss.requires_grad)</li>
<li>loss.backward()  # retain_graph=True</li>
<li># torch.nn.utils.clip_grad_norm(posenet.parameters(), args.max_grad_norm)</li>
<li>optimizer.step()  # TODO：可以使用累加的loss变相增大batch size，但对于bn层需要减少默认的momentum</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
Alternatively, I recommend to use distributed training.</p>

username: zzsunshine
cooked: <p>Thanks for your quick response. Yeah, improving the loss is one way to improve the first GPU memory utilization, however, i am not sure if it helps too much. I would prefer to split the batches manually.<img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>