Clarity on default initialization in pytorch

Taylor_Webb: <p>According to the <a href="https://pytorch.org/docs/stable/nn.html#linear-layers" rel="nofollow noopener">documentation</a> for torch.nn, the default initialization uses a uniform distribution bounded by 1/sqrt(in_features), but <a href="https://github.com/pytorch/pytorch/blob/3b1c3996e1c82ca8f43af9efa196b33e36efee37/torch/nn/modules/linear.py#L79" rel="nofollow noopener">this code</a> appears to show the default initialization as Kaiming uniform. Am I correct in thinking these are not the same thing? And if so, perhaps the documentation can be updated?</p>
<p>Does anyone know the motivation for this choice of default? In particular, the code appears to define a negative slope for the subsequent nonlinearity of sqrt(5), whereas the default negative slope for a leaky relu in pytorch is 0.01. Also, does anyone know how this negative slope is actually incorporated into the initialization?</p>

Nikronic: <p>Hi,</p>
<p>Let me explain it step by step.</p>
<ol>
<li>
<p>Here is <a href="https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.kaiming_uniform_" rel="noopener nofollow ugc">kaiming_uniform_</a>.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e734d7ffa1228e1394abcb8aa3b4db688b6bcc8.png" data-download-href="https://discuss.pytorch.org/uploads/default/2e734d7ffa1228e1394abcb8aa3b4db688b6bcc8" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e734d7ffa1228e1394abcb8aa3b4db688b6bcc8.png" alt="image" data-base62-sha1="6CUZUEtTb9VD7ROLTq2WBtEcCTu" width="690" height="135" data-dominant-color="232525"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">764×150 15.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a5811a62ab07cb15bb9419fa1f5c19d59e0ac408.png" data-download-href="https://discuss.pytorch.org/uploads/default/a5811a62ab07cb15bb9419fa1f5c19d59e0ac408" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a5811a62ab07cb15bb9419fa1f5c19d59e0ac408.png" alt="image" data-base62-sha1="nC7nQG2Hl3OF5PhSl253mT4V9lS" width="690" height="401" data-dominant-color="1B1D1E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">731×425 8.69 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Where <code>negative_slope=sqrt(5)</code> so the <code>gain=sqrt(2/6)=1/sqrt(3)</code> for kaiming.<br>
If we replace this in <code>bound</code> formula, we get <code>bound = [1/sqrt(3) ] * [sqrt(3/ fan_in)]</code> which with a little simplification, it will be <code>bound = 1/sqrt(fan_in)</code> which can be represented by <code>bound^2 = 1 / fan_in</code>.</p>
</li>
<li>
<p>In <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Linear" rel="noopener nofollow ugc">linear implementation</a> code you referenced:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2d67360b59665908a9ee8b4916e926f6ad7e97d.png" alt="image" data-base62-sha1="newO8FpuIhWKGnFcb591Ak9A4B7" width="381" height="28"><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4ad2915ff2923e2096f20eb067b59d51495d8bb5.png" data-download-href="https://discuss.pytorch.org/uploads/default/4ad2915ff2923e2096f20eb067b59d51495d8bb5" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4ad2915ff2923e2096f20eb067b59d51495d8bb5.png" alt="image" data-base62-sha1="aFUthSSOio8P8nbO2eMceTnxpGt" width="690" height="165" data-dominant-color="262627"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">710×170 15.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</li>
</ol>
<p>So what we have here is that <code>k= 1/in_feautres</code> which in case of kaming it can be represented <code>k=1/fan_in</code>. Also, we want a boundary of <code>[-sqrt(k), sqrt(k)]</code> where <code>k = bound^2= 1 / fan_in</code> from step 1.</p>
<p>For simplcity, just replace <code>sqrt(5)</code> in <code>gain</code> formula then optain <code>bound</code> in <code>kaiming_uniform_</code> and replace the <code>bound</code> as <code>k</code> in linear.</p>
<p>Edit: Add some related posts</p>
<ol>
<li><a href="https://github.com/pytorch/pytorch/issues/15314" class="inline-onebox" rel="noopener nofollow ugc">Kaiming init of conv and linear layers, why gain = sqrt(5) · Issue #15314 · pytorch/pytorch · GitHub</a></li>
<li><a href="https://discuss.pytorch.org/t/why-the-default-negative-slope-for-kaiming-uniform-initialization-of-convolution-and-linear-layers-is-5/29290" class="inline-onebox">Why the default negative_slope for kaiming_uniform initialization of Convolution and Linear layers is √5?</a></li>
</ol>
<p>Bests</p>

Taylor_Webb: <p>Thanks so much for this very thorough explanation. So, if I understand correctly, this achieves what is described in the documentation (parameters drawn from uniform distribution bounded by 1/sqrt(in_features)), but in a kind of circuitous way. Although this approach uses the init.kaiming_uniform_ function, it is not actually Kaiming initialization (in the scenario where the subsequent nonlinearity is a ReLU). To get Kaiming initialization for a ReLU layer, one would need to re-initialize the weights using init.kaiming_uniform_ with nonlinearity set to ‘relu’. init Is that correct?</p>

Nikronic: <p>You are welcome.<br>
I think so, actually if you read the paper that introduces <em><strong>kaiming</strong></em> or <em><strong>he</strong></em>, its main advantage is that defines uniform distribution in the way that enables deep NNs with many layers to converge faster, so it is kind of super class of basic uniform as it can be obtained from <code>kaiming_uniform</code>. So, that is why I think if you define a small model using this approach, it won’t hurt the model’s performance and maybe because of this logic, it is the default init method(simplicity purposes without loosing performance).<br>
But as you mentioned, logically gain for different activation functions are different and need to be incorporated.</p>

tom: <p>So what happened is that there were initializations in Torch7.<br>
These have later been expressed as calls to kaiming_uniform.<br>
But so there is not as much relation to the ideas of He’s paper as the use of the function might suggest.<br>
At some point people thought <a href="https://github.com/pytorch/pytorch/issues/18182" rel="nofollow noopener">changing the init</a> were a good idea. But it seems that that’s something everyone wants to have done but noone wants to actually do.</p>
<p>Best regards</p>
<p>Thomas</p>

Hmrishav_Bandyopadhy: <p>Hi,</p>
<p>As i understand from this, linear layers are initialized by kaiming initialization. This initialization is supposed to give output tensors of mean 0 and std 1.<br>
However, I noticed some contradicting stuff–</p>
<pre><code class="lang-auto">&gt;&gt;&gt; k=nn.Linear(512,512)
&gt;&gt;&gt; k1=torch.rand(512)
&gt;&gt;&gt; out=k(k1)
&gt;&gt;&gt; print(out.std(),out.mean())
tensor(0.3505, grad_fn=&lt;StdBackward0&gt;) tensor(0.0181, grad_fn=&lt;MeanBackward0&gt;)
</code></pre>
<p>Now, when i force kaiming initialization on the layer by –</p>
<pre><code class="lang-auto"> &gt;&gt;&gt; nn.init.kaiming_uniform_(k.weight,mode='fan_in')
Parameter containing:
tensor([[-0.0134,  0.0247, -0.0817,  ..., -0.0807, -0.0330, -0.0881],
        [-0.0641,  0.0447,  0.0645,  ...,  0.1048, -0.0307,  0.0989],
        [ 0.0745, -0.0076,  0.0161,  ...,  0.0252,  0.0285, -0.0527],
        ...,
        [ 0.0190, -0.0529, -0.0549,  ..., -0.0369,  0.0331,  0.0136],
        [ 0.0347,  0.0516,  0.0108,  ..., -0.0772,  0.0027,  0.0584],
        [-0.0236,  0.0565,  0.0082,  ...,  0.0717, -0.0619, -0.0772]],
       requires_grad=True)
&gt;&gt;&gt; out=k(k1)
&gt;&gt;&gt; print(out.std(),out.mean())
tensor(0.8420, grad_fn=&lt;StdBackward0&gt;) tensor(-0.0287, grad_fn=&lt;MeanBackward0&gt;)
</code></pre>
<p>I get the tensors nearing the actual mean and std values. What am i missing here?<br>
Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

tom: <aside class="quote no-group" data-username="Hmrishav_Bandyopadhy" data-post="6" data-topic="84696">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hmrishav_bandyopadhy/48/24065_2.png" class="avatar"> Hmrishav_Bandyopadhy:</div>
<blockquote>
<p>What am i missing here?</p>
</blockquote>
</aside>
<p>So the thing we discussed above is that while the default init is <em>expressed as</em> kaiming init times some gain factor, it <em>is not</em> kaiming init, as the gain factor is “bogus” w.r.t. kaiming init and the activation function and only serves to reproduce ancient behaviour.<br>
Happily, <a class="mention" href="/u/kushaj">@Kushaj</a> started work on it.</p>