Initialization and batch normalization

username: surojit_sengupta
cooked: <p>Hello Moderators,</p>
<p>Do we use any of the initialization techniques for weights like Glorot, He et al.; along with batch normalization in practice? or it’s a case of choosing either one?</p>
<p>Regards</p>

username: vabh
cooked: <p>Hi,</p>
<p>The reference ResNet implementation uses the initialisation strategy from He et al. and it’s a network with batch norm layers.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L118" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L118" target="_blank" rel="nofollow noopener">pytorch/vision/blob/master/torchvision/models/resnet.py#L118</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="108" style="counter-reset: li-counter 107 ;">
<li>    self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)</li>
<li>    self.layer1 = self._make_layer(block, 64, layers[0])</li>
<li>    self.layer2 = self._make_layer(block, 128, layers[1], stride=2)</li>
<li>    self.layer3 = self._make_layer(block, 256, layers[2], stride=2)</li>
<li>    self.layer4 = self._make_layer(block, 512, layers[3], stride=2)</li>
<li>    self.avgpool = nn.AdaptiveAvgPool2d((1, 1))</li>
<li>    self.fc = nn.Linear(512 * block.expansion, num_classes)</li>
<li>
</li>
<li>    for m in self.modules():</li>
<li>        if isinstance(m, nn.Conv2d):</li>
<li class="selected">            nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')</li>
<li>        elif isinstance(m, nn.BatchNorm2d):</li>
<li>            nn.init.constant_(m.weight, 1)</li>
<li>            nn.init.constant_(m.bias, 0)</li>
<li>
</li>
<li>def _make_layer(self, block, planes, blocks, stride=1):</li>
<li>    downsample = None</li>
<li>    if stride != 1 or self.inplanes != planes * block.expansion:</li>
<li>        downsample = nn.Sequential(</li>
<li>            conv1x1(self.inplanes, planes * block.expansion, stride),</li>
<li>            nn.BatchNorm2d(planes * block.expansion),</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: Hmrishav_Bandyopadhy
cooked: <p>Hi,<br>
Sorry for opening up this old thread. Can you please explain why initialization is necessary when we are using batch normalization? Batch normalization seems to normalize along the batches and reduces the problem of the “Mean length scale in final layer” as described in <a href="https://papers.nips.cc/paper/7338-how-to-start-training-the-effect-of-initialization-and-architecture.pdf" rel="nofollow noopener">How to Start Training:The Effect of Initialization and Architecture</a> by Hanin and Rolnick. Also, by intuition, batch norm should actually reduce the mean to zero and standard deviation to one before the layer output is fed to later layers. As such, this should prevent any kind of slowing down due to zigzagging as described in <a href="http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf" rel="nofollow noopener">Efficient Backprop</a> by LeCun</p>

username: blade
cooked: <p>There are still a shift and scale parameter applied to the normalized mini-batch input. These parameters are learned in the Batch normalization layer.</p>

username: Hmrishav_Bandyopadhy
cooked: <p>Ok but batch norm actually has a flag that can be set to false in order to avoid getting the shift and scale values. If we use that, then?</p>