What is the theoretical reason for the default way of Pytorchs weight initialization?

ColdWinter: <p>I was wondering what the reason is that weights are initialized the way they are? For torch.nn.Linear the weights are initialized with samples from Uniform(-sqrt(k), sqrt(k)) with k = 1/no_features. <a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#torch.nn.Linear" class="inline-onebox" rel="noopener nofollow ugc">Linear — PyTorch 1.7.0 documentation</a></p>
<p>In the source code “kaiming_uniform” appears. However, to my understanding Kaiming He at al. [1] suggest sampling from a normal distribution. If I take the condition they state that 0.5 * no_features * VAR[w_l] = 1 and plug in the variance for the uniform distribution which should be (b - a)^2 / 12, with a= - sqrt(k) and b=sqrt(k), this does not add up either.</p>
<p>So what is the “Kaiming Uniform” and/or what is the reason that the weights are "initialized from  initialized from U}(-sqrt(k)), sqrt(k))?</p>
<p>Thanks for any help</p>
<p>[1] <a href="https://arxiv.org/pdf/1502.01852.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1502.01852.pdf</a></p>

Nikronic: <p>Hi,</p>
<p>I think this thread and its references can answer your question.</p>
<aside class="quote quote-modified" data-post="1" data-topic="84696">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/taylor_webb/48/16355_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/clarity-on-default-initialization-in-pytorch/84696">Clarity on default initialization in pytorch</a> 
  </div>
  <blockquote>
    According to the <a href="https://pytorch.org/docs/stable/nn.html#linear-layers" rel="noopener nofollow ugc">documentation</a> for torch.nn, the default initialization uses a uniform distribution bounded by 1/sqrt(in_features), but <a href="https://github.com/pytorch/pytorch/blob/3b1c3996e1c82ca8f43af9efa196b33e36efee37/torch/nn/modules/linear.py#L79" rel="noopener nofollow ugc">this code</a> appears to show the default initialization as Kaiming uniform. Am I correct in thinking these are not the same thing? And if so, perhaps the documentation can be updated? 
Does anyone know the motivation for this choice of default? In particular, the code appears to define a negative slope for the subsequent nonlinearity of sqrt(5), whereas the default…
  </blockquote>
</aside>

<p>Feel free to ask if it did not help.</p>
<p>Bests</p>

ColdWinter: <p>Thank you so much, this brought a lot of clarify!</p>
<p>With the Uniform distributions variance being Var[W] = (1/12)* (b-a)^2 the way the weights are initialized for nn.Linear seems to fulfill Eq. 15 in [Xavier] which states that it should be n*Var[W] = 1/3<br>
Isn’t the weight initialization thus Xavier?</p>
<p>[Xavier] <a href="http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf" rel="noopener nofollow ugc">http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf</a></p>

Nikronic: <p>In Xavier uniform weight init which you can find in eq.16 of the paper you have referenced and also PyTorch docs:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a488e6fcd52ed58ffe227745016eb33a72b651c8.png" data-download-href="https://discuss.pytorch.org/uploads/default/a488e6fcd52ed58ffe227745016eb33a72b651c8" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a488e6fcd52ed58ffe227745016eb33a72b651c8.png" alt="image" data-base62-sha1="ntxCfCWXTX34N1CrZ9jnI5aSkje" width="690" height="430" data-dominant-color="1F2021"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">796×497 28.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Where in the paper, the target activation function was <code>ReLU</code> which has <code>gain=1</code>. Based on this, default weight init of Linear or Conv layers in PyTorch where follows Kaiming He uniform but the gain and negative slope has been defined in a way that it is a simple uniform as the way it has been explained in the thread I posted first.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d20d7540af9abc0e59794b97923ec22289f677e3.png" data-download-href="https://discuss.pytorch.org/uploads/default/d20d7540af9abc0e59794b97923ec22289f677e3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d20d7540af9abc0e59794b97923ec22289f677e3.png" alt="image" data-base62-sha1="tYd6lSor48QiKZdchyUZjZ93qmf" width="690" height="438" data-dominant-color="212324"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">777×494 41.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>One of the differences is that in Xavier you incorporate both <code>fan_in</code> and <code>fan_out</code> in the equation but Kaiming only uses one. Note that you can obtain Xavier from Kaiming by setting <code>negative_slope=0</code> and <code>fan_out=0</code>. But it does not work as <code>fan_out=0</code> does not make sense in neural networks.<br>
In the same way that you can get Xavier from Kaiming, you can get standard uniform from Kaiming too.</p>