BatchNorm Initialization

MeowLady: <p>Recently I rebuild my caffe code with pytorch and got a much worse performance than original ones. Also I find the converge speed is slightly slower than before. When I check the initialization of model, I notice that in caffe’s BN(actually scale layer) layer parameter gamma is initialized with 1.0 while the default initialization in pytorch seems like random float numbers. In addition, when I initialize BN’s weight with 1.0 the training loss can drop faster. So I want to know what the default initialization is in pytorch’s BN layer. Anyone could please answer my question? Thanks a lot!</p>
<p>Regards<br>
MeowLady</p>

SimonW: <p>It’s this <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/batchnorm.py#L33-L39">https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/batchnorm.py#L33-L39</a></p>
<p>w ~ U[0, 1]<br>
b = 0</p>

MeowLady: <p>Wow, got it! Thanks a lot for your quick and accurate reply! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

r0mer0m: <p>I might be understanding something wrong but I also want to see how the weights and bias of Batchnorm are initialised. When looking at the source code I see the following in the <code>__init__</code> for the default parameters:</p>
<pre><code class="lang-auto">            self.weight = Parameter(torch.Tensor(num_features))
            self.bias = Parameter(torch.Tensor(num_features))
</code></pre>
<p>However when I run <code>torch.Tensor(num_features)</code> on my computer I get some numbers <code>&lt;0</code> and  <code>&gt;1</code>. Even some <code>nan</code> for num_features. Am I doing something wrong or missing something?</p>
<p>Thanks!</p>

ptrblck: <p>You are right and <code>torch.Tensor</code> creates an uninitialized tensor, so you would have to make sure to properly initialize it afterwards.<br>
For batchnorm layers, it is done in the <a href="https://github.com/pytorch/pytorch/blob/15be5483c0222ca9fbb596e011eec41ae4061bcc/torch/nn/modules/batchnorm.py#L54" rel="nofollow noopener">reset_parameters()</a> method.</p>

r0mer0m: <p>Thank you very much <a class="mention" href="/u/ptrblck">@ptrblck</a>, that makes a lot of sense!</p>
<p>But then it does not match with w ~ U[0, 1], does it? Looking into that function and <code>init.ones_</code> the method fills weights with ones, not with weights following iid U[0, 1]. Therefore it is initialized with ones and not with iid U[0, 1].</p>

ptrblck: <p>Yeah, you are right and I posted the implementation of the current master branch.<br>
<a href="https://github.com/pytorch/pytorch/pull/13774" rel="nofollow noopener">This PR</a> was only merged recently, so that this new behavior is documented in the <a href="https://pytorch.org/docs/master/nn.html?highlight=batchnorm#torch.nn.BatchNorm2d" rel="nofollow noopener">master docs</a>.<br>
Before that, you’ll see that the weights were initialized using a uniform distribution (<a href="https://github.com/pytorch/pytorch/blob/fb6347d9372f2dfc8015003fe315d840f0709037/torch/nn/modules/batchnorm.py#L53" rel="nofollow noopener">line of code for 1.1.0</a>).</p>
<p>Sorry for the confusion.</p>

r0mer0m: <p>Yay! Thank you very much for the complete explanation and pointing to the doc,  code and reference. It’s been extremely helpful.</p>