Batch norm training when batch size=1

username: edshkim98
cooked: <p>Hello everyone, I am currently facing a problem regarding a small GPU memory during my deep learning project. To handle this, I am currently training in batch size =4 but this requires a significant sampling from the initial data to be able to fit into my GPU. Hence, I think I have to use batch size = 1 which is a stochastic gd. However, I have read some posts saying that batch normalization is not good to be used in batch size =1. If it is true, what should I do with the BN in my model? Do I have to remove them?</p>

username: ptrblck
cooked: <p>The batch statistics might be noise with a single sample or your model might even raise an error, if no statistics can be computed from the input.<br>
However, as usual, it depends on your use case and I would recommend to run some experiments and play around with the <code>momentum</code> term in the batchnorm layers.</p>

username: edshkim98
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hi than you for your reply!<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/5/25fece8303a8d4b14682f2c6dc39fe168468e6c5.png" data-download-href="https://discuss.pytorch.org/uploads/default/25fece8303a8d4b14682f2c6dc39fe168468e6c5" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/5/25fece8303a8d4b14682f2c6dc39fe168468e6c5.png" alt="image" data-base62-sha1="5q7ApfLdEtNqYC2GwSd342CDepn" width="690" height="125" data-dominant-color="000000"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">770×140 11.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I realized that my momentum value is set in really low value currently, but apparently it is recommended to set high value when I am training in small batch.<br>
Using the equation above, could you please interpret to me how the statement is valid?</p>

username: ptrblck
cooked: <p>Be a bit careful about the <code>momentum</code> definition in batchnorm layers, as they might differ from other definitions.<br>
From the <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html?highlight=batchnorm2d#torch.nn.BatchNorm2d">docs</a>:</p>
<blockquote>
<p>This momentum argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is x_hat_new = (1 - momentum) * x_hat + momentum * x_t, where x_hat is the estimated statistic and x_t is the new observed value.</p>
</blockquote>
<p>Based on your posted formula (assuming <code>alpha</code> is the momentum) the definition differs as explained in the docs.</p>

username: edshkim98
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, thank you for the reply.<br>
Base on the document, it means that the momentum in pytorch is opposite to the equation I posted. Hence, it should be set really low like 0.1 during training with low batch.<br>
Is this correct?</p>

username: ptrblck
cooked: <p>The default value is already set to <code>0.1</code>, so you might want to decrease it even further.</p>

username: edshkim98
cooked: <p>Ok thank you for the hlep!!</p>

username: edshkim98
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, hi I just tried changing the momentum for batch norm, but unfortunately it did not work.<br>
The error is triggered during batchnorm1d with batch size 1.<br>
It seems that it is not possible to train in batch size 1 with batchnorm1d.<br>
Is this correct? If so, is there any alternative solution that I can do?</p>

username: ptrblck
cooked: <p>You cannot use batchnorm layers with a single sample, if the temporal dimension also contains only a single time step as seen here:</p>
<pre><code class="lang-python">bn = nn.BatchNorm1d(3)
x = torch.randn(1, 3, 10)
out = bn(x)

x = torch.randn(1, 3, 1)
out = bn(x) # error
</code></pre>
<p>since the mean would just be the channel values and the stddev cannot be calculated.<br>
I’m not sure, if any normalization layer would make sense in such a use case, but lets wait for others to chime in.</p>

username: googlebot
cooked: <aside class="quote no-group" data-username="edshkim98" data-post="8" data-topic="102979">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/edshkim98/48/29279_2.png" class="avatar"> edshkim98:</div>
<blockquote>
<p>If so, is there any alternative solution that I can do?</p>
</blockquote>
</aside>
<p>You can try LayerNorm as a substitute, but it is not always performing as well, being too different from batch norm. Or something like <a href="https://github.com/Cerebras/online-normalization" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Cerebras/online-normalization: Online Normalization for Training Neural Networks (Companion Repository)</a></p>

username: edshkim98
cooked: <p><a class="mention" href="/u/googlebot">@googlebot</a>  Thank you for recommending an excellent paper. I will have a read and try it!!</p>

username: ptrblck
cooked: <p>Wouldn’t <code>LayerNorm</code> calculate the stats from the single pixel and thus return a zero output or do I misunderstand the use case?</p>

username: googlebot
cooked: <p>It is applied to vectors in feature space, though I’ve read that layer norm doesn’t work well with convolutions, stats would be per image region. I think the problem is rather with channel importance equalization, as layer norm “ties” all dimensions; I guess that is bad for early vision filters.</p>
<p>PS: if that’s not clear, I meant group norm applied to a channels last permuted tensor</p>