[Module fixing] Batch Normalisation for batch size 1 and feature size 1 should be fixed for training

username: tlim
cooked: <p>Hi all,</p>
<p>I found that PyTorch currently does not have a solution to batch normalisation under batch size of 1 and feature size of 1. Example that will cause error:</p>
<pre><code class="lang-auto">    batch = 1
    channels = 3
    feature_size = 1
    in_ = torch.ones((batch,channels,feature_size,feature_size))
    norm  = nn.BatchNorm2d(channels)
    y = norm(in_)
</code></pre>
<p>Error will be raised (which I know makes sense as there is batch size of 1 has nothing to normalised with). However, there are many potential situation in training where you will load a single image as the last batch. It seems PyTorch has already allow batch size of 1 as long as we increase the feature size, example:</p>
<pre><code class="lang-auto">    batch = 1
    channels = 3
    feature_size = 2
    in_ = torch.ones((batch,channels,feature_size,feature_size))
    norm  = nn.BatchNorm2d(channels)
    y = norm(in_)
</code></pre>
<p>No error will be raised. Why don’t they do the same for <code>feature_size=1</code>?</p>
<p>If you put <code>norm.eval()</code>, there will be no issue. However, I am asking mainly for training.</p>
<p>Another way around it is to set training DataLoader <code>drop_last=True</code> to prevent this but some user with different GPU memory size will use different batch size and not realise what error is hitting them especially when they are just the user.</p>
<p>Edit: *user of another open-source repository, this is evident in a few pytorch open-source projects open issues.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="tlim" data-post="1" data-topic="136679">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tlim/48/32364_2.png" class="avatar"> tlim:</div>
<blockquote>
<p>It seems PyTorch has already allow batch size of 1 as long as we increase the feature size, example:</p>
</blockquote>
</aside>
<p>Because a larger feature size would allow the batchnorm layer to calculate the <code>stddev</code> from these values, while a batch size and feature dimension of 1 would have an undefined (or invalid) <code>stddev</code>.</p>
<p>How would you calculate the <code>stddev</code> in the case of a single value? Would you rather return NaNs in the forward pass?</p>

username: tlim
cooked: <p>Could the internal batch norm function adds an <code>if</code> clause for feature size of 1 to forward pass like nn.Identity()?</p>

username: ptrblck
cooked: <p>I would be a bit skeptical about this workaround. While it would avoid the runtime error, I would expect to see a hard runtime error as a user if I want to use a layer, which cannot perform the mathematical operation internally as the input shape doesn’t allow it.</p>
<p>The overall use case in a batchnorm layer is:</p>
<pre><code class="lang-python">out = (input - mean) / stddev * weight + bias
</code></pre>
<p>Using your suggestion you would also skip the affine parameters, which would thus also not be trained.<br>
On the other hand, you could also skip the mean subtraction and division by <code>stddev</code>, which would then however yield a different activation range and could potentially diverge your model training.</p>

username: tlim
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>I am not very familiar on what will hold the pytorch codebase back from just literally having batch normalisation do nothing (no trainable parameter) when <code>batch_size=1</code> and <code>feature_map_size=1</code> for training.</p>
<p>I think users will be unaware of the problem as the training will work most of the time and then suddenly fail (ex: DataLoader loads one final image for training).  This is also painful for users trying to train on a big dataset as they have to go through a lot of iterations just to get hit by the batch normalisation error caused by the final data batch being 1 (assuming in the networks, there is a module that pools to a <code>feature size=1</code>).</p>

username: ptrblck
cooked: <p>I would claim a valid runtime error raised by an invalid mathematical operation is way less painful than an unexpected behavior by the framework, which would change the behavior of the model and could potentially diverge your training.<br>
Debugging such divergence issues can take many more weeks than to fix a shape issue.<br>
As described before, I would see the workaround using the <code>nn.Identity</code> module as a silent numerical error, since the output would not be normalized at all and could have a different range and thus cause divergence.</p>
<p>Here is a small example:</p>
<pre><code class="lang-python">bn = nn.BatchNorm1d(3)

x = torch.randn(2, 3, 1) * 1000.
out = bn(x)
print(out.min(), out.max(), out.mean())
# &gt; tensor(-1., grad_fn=&lt;MinBackward1&gt;) tensor(1., grad_fn=&lt;MaxBackward1&gt;) tensor(0., grad_fn=&lt;MeanBackward0&gt;)

# workaround with Identity 
bn = nn.Identity()
out = bn(x)
print(out.min(), out.max(), out.mean())
# &gt; tensor(-1368.4515) tensor(1302.6108) tensor(75.8487)
</code></pre>
<p><code>x</code> is the incoming activation with a large value range (<code>randn * 1000.</code>). In the working case the batchnorm layer would normalize it and the following layer (e.g. a linear layer) would expect to see these normalized values. If you are now skipping the norm the following layer would get the unnormalized values with an entirely different range.</p>
<p>You could see similar effects by using a pretrained model (e.g. use <code>torchvision.models.resnet18</code>) and by comparing the achieved accuracy using normalized input images (expected) vs. unnormalized inputs with values in the range <code>[0, 255]</code>. In the latter case the model would not perform well even though it’s a proper pretrained model.</p>