Batch Normalization of Linear Layers

WR01: <p>Is it possible to perform batch normalization in a network that is only linear layers?</p>
<p>For example:</p>
<pre><code>class network(nn.Module):
    def __init__(self):
        super(network, self).__init__()
        self.linear1 = nn.Linear(in_features=40, out_features=320)
        self.linear2 = nn.Linear(in_features=320, out_features=2)

    def forward(input):  # Input is a 1D tensor
        y = F.relu(self.linear1(input))
        # Would it be possible to do a batch normalization of y overhere? If so how?
        y = F.softmax(self.linear2(input))
        return y</code></pre>

ptrblck: <p>Sure! You could just use <code>nn.BatchNorm1d</code>.<br>
There are some minor issues in your code, so here is a working example:</p>
<pre><code class="lang-python">class network(nn.Module):
    def __init__(self):
        super(network, self).__init__()
        self.linear1 = nn.Linear(in_features=40, out_features=320)
        self.bn1 = nn.BatchNorm1d(num_features=320)
        self.linear2 = nn.Linear(in_features=320, out_features=2)

    def forward(self, input):  # Input is a 1D tensor
        y = F.relu(self.bn1(self.linear1(input)))
        y = F.softmax(self.linear2(y), dim=1)
        return y
    
model = network()
x = torch.randn(10, 40)
output = model(x)
</code></pre>
<p>You can also put the <code>BatchNorm</code> after the <code>relu</code>, if you like.</p>

WR01: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I tried that but I received “ValueError: expected a 2D or 3D input (got 1D input).”</p>

ptrblck: <p>Are you sure you are passing your input as <code>[batch_dim, num_features]</code>?<br>
The error sounds like you’ve passed just <code>[num_features]</code> to your model.</p>

WR01: <p>When I do that I get a different error.: “ValueError: Expected more than 1 value per channel when training, got input size [1, 320].” This is for Q network, so it only receives one state at a time, hence the batch size of 1.</p>

ptrblck: <p>Then <code>nn.BatchNorm</code> probably won’t work very well.<br>
Have a look at the <a href="https://pytorch.org/docs/stable/nn.html#normalization-layers">normalization layers</a>. Maybe <code>LayerNorm</code> or another one will fit your needs.</p>

shirui-japina: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="20989">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>You can also put the <code>BatchNorm</code> after the <code>relu</code> , if you like.</p>
</blockquote>
</aside>
<p>Is it the same effect that put the <code>BatchNorm</code> <em><strong>before</strong></em> or <em><strong>after</strong></em> the <code>ReLU</code>?<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>You will most likely see a different performance depending on where you place the batchnorm layer, since the input activation will have a different distribution.</p>

shirui-japina: <p>So…where should I place the <code>BatchNorm</code> layer, to train a great performance model?<br>
(Not only linear layers model, but like CNN or RNN)<img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=9" title=":flushed:" class="emoji" alt=":flushed:"><img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=9" title=":flushed:" class="emoji" alt=":flushed:"></p>
<ol>
<li>
<p>Between each layer?<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>
</li>
<li>
<p>Just before or after the activation function layer?<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>
</li>
<li>
<p>Should <strong>before</strong> or <strong>after</strong> the activation function layer?<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>
</li>
</ol>
<p>And where I shouldn’t place the  <code>BatchNorm</code>  layer?</p>

mailcorahul: <p><a class="mention" href="/u/shirui-japina">@shirui-japina</a> In general, Batch Norm layer is usually added before ReLU(as mentioned in the Batch Normalization paper). But there is no real standard being followed as to where to add a Batch Norm layer. You can experiment with different settings and you may find different performances for each setting.</p>
<p>As far as I know, generally you will find batch norm as part of the feature extraction branch of a network and not in its classification branch(<code>nn.Linear</code>).</p>

shirui-japina: <p>Thanks for your reply.<img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>So the place of <code>BatchNorm</code> layer in CNN is like this:<br>
CNN(<br>
convolution-layer-1,<br>
batch-norm-layer-1,<br>
activate-layer(ReLU),</p>
<p>convolution-layer-2,<br>
batch-norm-layer-2,<br>
activate-layer(ReLU),</p>
<p>fully-connection-layer,<br>
)</p>
<p>How about the pooling layer?<br>
Should we place <code>BatchNorm</code> layer before the pooling layer?<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"><img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>

mailcorahul: <p>If you ask me, I would place it after the pooling layer. But you can check out how vision models are implemented in pytorch to get clarity.</p>

shirui-japina: <p>Got it, thanks for your help.<img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

saba: <p>Hi Ptrblck</p>
<p>Sorry to take your time. I have a question, I normalized my patch before training, and my ANN is 2CNN layer with 2 fully connected layer. Is it necessary to do batch normalization or since the layers are not very deep it is not necessary?</p>

ptrblck: <p>Oh, my best advice is to try out both approaches and compare the validation accuracy with and without batchnorm layers.<br>
I don’t have a specific advice on when to use them with respect to the number of layers. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<p>Let us know, which model worked better! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>PS: Also, compare the training and validation accuracy to pick the right model, <em>not</em> the test accuracy, as you would leak the test data information into your model selection process.</p>

Shisho_Sama: <p>You most likeley will not see a drastic change in the network performance (get higher acc,etc). however, batchnorml incur around 30% overhead to your network runtime. it will affect your training as well as inference unless at inference you fuse them.<br>
All in all BatchNorm shines when you have a very deep architecture, what you have there is not really considered deep that much.<br>
You may very well update us with the result you get though<br>
Cheers.</p>

saba: <p>Thanks for your answer</p>

Kale-ab_Tessera: <p>Shouldn’t we set the bias to False for the linear and conv layers when using batch norm?</p>

Anshumaan_Dash: <p>BatchNorm was introduced to distribute the data uniformly across a mean that the network sees best, before squashing it by the activation function. Without the BN, the activations could over or undershoot, depending on the squashing function though.</p>
<p>Hence, even in practice, BN before the activation function gives better performance.</p>
<p>I mean, for the sake of putting it, one can put a dropout as the very first layer, or even with Conv layers, and the network will still train. But, that doesn’t make any sense.</p>

STU: <p>Hi, if my input is [batch_dim, temporal_length, channel_dim], how to do batch normalization?</p>