How to decrease the weight of a mini-batch?

username: sunfishcc
cooked: <p>The problem comes from this paper: <a href="https://arxiv.org/abs/1706.02613" rel="noopener nofollow ugc">Decoupling “when to update” from “how to update”</a>.<br>
Regardless the detail of the paper, this implementation will resulting mini-batches has different sizes.<br>
For example, if we set the mini-batch size to 128. Some of the batches could have only 10 or even zero samples.<br>
If there’s zero sample in a batch, then we can just skip the update at this iteration.<br>
However, if the sample in a batch is much samller, let’s say 10% of the 128, then we have to scale down the update of the gradient on this iteration.<br>
The idea is that if there is a less important batch, it shouldn’t have the same impact as a full-size batch.<br>
If we reduce the learning rate 10 time on this paticular batch, we can achieve this effect, and use the normal learning rate on the full-size batches.<br>
But how can we achieve this in PyTorch?<br>
I hope my description is clear. Thank you so much for any suggestions.</p>

username: mMagmer
cooked: <p>you can reduce output of loss function yourself  or use <code>reduction='sum'</code>   instead of <code>reduction='mean'</code>. this way your orverall loss is proportional to number of samples in each batchs.<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">CrossEntropyLoss</a>:</p>
<blockquote>
<ul>
<li><strong>reduction</strong> (<em>string</em> <em>,</em> <em>optional</em>) – Specifies the reduction to apply to the output: <code>'none'</code> | <code>'mean'</code> | <code>'sum'</code>. <code>'none'</code>: no reduction will be applied, <code>'mean'</code>: the weighted mean of the output is taken, <code>'sum'</code>: the output will be summed. Note: <code>size_average</code> and <code>reduce</code> are in the process of being deprecated, and in the meantime, specifying either of those two args will override <code>reduction</code>. Default: <code>'mean'</code></li>
</ul>
</blockquote>

username: sunfishcc
cooked: <p>That’s a very intersting solution to the problem. I only saw the weight parameter in the loss func, but don’t think it fits this situation. If it works, it will scale down nicely. I give a try and let you know the result. Thank you.</p>
<p>When <code>nn.CrossEntropyLoss(reduction='sum')</code> is used, the model simple does not converge with <code>torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)</code>.<br>
Once swith back to <code>mean</code>, the same model converge within 5 epochs tested on MNIST.<br>
It seems the <code>sum</code> option is only used in evaluation for computing the overall loss, and nobody use it on training.</p>

username: mMagmer
cooked: <p><a class="mention" href="/u/sunfishcc">@sunfishcc</a> it’s because of large learning rate.<br>
When you’re using mean, you’re effectively dividing learning rate by batch size.<br>
use smaller learning rate.</p>

username: sunfishcc
cooked: <p><a class="mention" href="/u/mmagmer">@mMagmer</a> I see. It works once I changed <code>lr</code> to <code>0.0001</code>.</p>
<p>I also tried to manually adjust lr when then batch size is low: <code>optimizer.param_groups[0]['lr'] *= 0.1</code>, and then reset it after calling the <code>step()</code> function. This seems like a hack, not sure about if there’s any side effect.</p>
<p>Thank you so much for your suggestion.</p>