What's the default initialization methods for layers?

Nick_Young: <p>what’s the default initialization methods for layers? Like conv, fc, and RNN layers? are they just initialized to all zeros?</p>

albanD: <p>All the layer are implemented in this folder: <a href="https://github.com/pytorch/pytorch/tree/master/torch/nn/modules">https://github.com/pytorch/pytorch/tree/master/torch/nn/modules</a><br>
The initialization depend on the layer, for example, the linear one is <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/linear.py#L44-L48">here</a></p>

Nick_Young: <p>Thank you so much!<img src="//discuss.pytorch.org/images/emoji/apple/+1.png?v=5" title=":+1:" class="emoji" alt=":+1:"></p>

Konpat_Ta_Preechakul: <p>I see only <code>torch.Tensor(...)</code> without any specific initialization method. I wonder what would that be?</p>

ptrblck: <p>In <code>reset_parameters()</code> the weights are set/reset.</p>

Brando_Miranda: <p>Are these initialization basically He init or Xavier init?</p>

klory: <p>Tanh --&gt; Xavier<br>
ReLU --&gt; He</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stats.stackexchange.com/questions/319323/whats-the-difference-between-variance-scaling-initializer-and-xavier-initialize/319849#319849" target="_blank" rel="nofollow noopener">stats.stackexchange.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stats.stackexchange.com/users/130598/maxim" target="_blank" rel="nofollow noopener">
    <img alt="Maxim" src="https://i.stack.imgur.com/Kt7gc.png?s=128&amp;g=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stats.stackexchange.com/questions/319323/whats-the-difference-between-variance-scaling-initializer-and-xavier-initialize/319849#319849" target="_blank" rel="nofollow noopener">What's the difference between variance scaling initializer and xavier initializer?</a>
</h4>

<div class="tags">
  <strong>machine-learning, neural-networks, deep-learning, conv-neural-network</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stats.stackexchange.com/users/130598/maxim" target="_blank" rel="nofollow noopener">
    Maxim
  </a>
  on <a href="https://stats.stackexchange.com/questions/319323/whats-the-difference-between-variance-scaling-initializer-and-xavier-initialize/319849#319849" target="_blank" rel="nofollow noopener">09:03AM - 21 Dec 17</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


Brando_Miranda: <p>So Pytorch uses He when its ReLU? Im confused what pytorch does.</p>

Brando_Miranda: <p>Sorry ptrblck, Im confused…pytorch uses Xavier or He depending on the activation?  Thats what klory seems to imply but the code looks as follows:</p>
<pre><code>def reset_parameters(self):
    stdv = 1. / math.sqrt(self.weight.size(1))
    self.weight.data.uniform_(-stdv, stdv)
    if self.bias is not None:
        self.bias.data.uniform_(-stdv, stdv)</code></pre>

rasbt: <blockquote>
<p>Tanh → Xavier<br>
ReLU → He</p>
</blockquote>
<p>No that’s not correct, PyTorch’s initialization is based on the layer type, not the activation function (the layer doesn’t know about the activation upon weight initialization).</p>
<p>For the linear layer, this would be somewhat similar to He initialization, but not quite:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/9e2f2cab94027c1be1860b9b5e98ac13c6b0516e/torch/nn/modules/linear.py#L48-L52">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/9e2f2cab94027c1be1860b9b5e98ac13c6b0516e/torch/nn/modules/linear.py#L48-L52" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/9e2f2cab94027c1be1860b9b5e98ac13c6b0516e/torch/nn/modules/linear.py#L48-L52" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/9e2f2cab94027c1be1860b9b5e98ac13c6b0516e/torch/nn/modules/linear.py#L48-L52</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="48" style="counter-reset: li-counter 47 ;">
          <li>def reset_parameters(self):</li>
          <li>    stdv = 1. / math.sqrt(self.weight.size(1))</li>
          <li>    self.weight.data.uniform_(-stdv, stdv)</li>
          <li>    if self.bias is not None:</li>
          <li>        self.bias.data.uniform_(-stdv, stdv)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<pre><code>def reset_parameters(self):
    stdv = 1. / math.sqrt(self.weight.size(1))
    self.weight.data.uniform_(-stdv, stdv)
    if self.bias is not None:
        self.bias.data.uniform_(-stdv, stdv)
</code></pre>
<p>Ie., when I remember correctly, He init is “sqrt(6 / fan_in)” whereas in PyTorch Linear layer it’s “1. / sqrt(fan_in)”</p>

klory: <p>Yeah, you’re correct, I just check their code for linear.py and conv.py, it seems they’re all using Xavier right? (I got the Xavier explanation from <a href="http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization" rel="nofollow noopener">here</a>)</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/099a6d5e083d78e04437a22e7cf963c4e0a1fa18/torch/nn/modules/linear.py#L48-L52" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/099a6d5e083d78e04437a22e7cf963c4e0a1fa18/torch/nn/modules/linear.py#L48-L52" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/099a6d5e083d78e04437a22e7cf963c4e0a1fa18/torch/nn/modules/linear.py#L48-L52</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="48" style="counter-reset: li-counter 47 ;">
<li>def reset_parameters(self):</li>
<li>    stdv = 1. / math.sqrt(self.weight.size(1))</li>
<li>    self.weight.data.uniform_(-stdv, stdv)</li>
<li>    if self.bias is not None:</li>
<li>        self.bias.data.uniform_(-stdv, stdv)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/099a6d5e083d78e04437a22e7cf963c4e0a1fa18/torch/nn/modules/conv.py#L40-L47" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/099a6d5e083d78e04437a22e7cf963c4e0a1fa18/torch/nn/modules/conv.py#L40-L47" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/099a6d5e083d78e04437a22e7cf963c4e0a1fa18/torch/nn/modules/conv.py#L40-L47</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="40" style="counter-reset: li-counter 39 ;">
<li>def reset_parameters(self):</li>
<li>    n = self.in_channels</li>
<li>    for k in self.kernel_size:</li>
<li>        n *= k</li>
<li>    stdv = 1. / math.sqrt(n)</li>
<li>    self.weight.data.uniform_(-stdv, stdv)</li>
<li>    if self.bias is not None:</li>
<li>        self.bias.data.uniform_(-stdv, stdv)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


Brando_Miranda: <p>are they all using Xavier?</p>

rasbt: <blockquote>
<p>it seems they’re all using Xavier right?</p>
</blockquote>
<p>Doesn’t xavier also include fan_out though? Here, I can only see input channels, not output channels.</p>

Brando_Miranda: <p>so is it just a unpublished made up pytorch init?</p>

rasbt: <p>Maybe I am overlooking sth or don’t recognize it, but I think so</p>

Brando_Miranda: <p>A validation from someone in the pytorch team would be nice calling for call for master <a class="mention" href="/u/simonw">@SimonW</a> <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

SimonW: <p>no, they are from respective well-established published papers actually. e.g., linear init is from “Efficient Backprop”, LeCun’99.</p>

Brando_Miranda: <p>Thanks! I appreciate the it.</p>

Brando_Miranda: <aside class="quote no-group" data-username="SimonW" data-post="17" data-topic="3157">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/simonw/48/1702_2.png" class="avatar"> SimonW:</div>
<blockquote>
<p>“Efficient Backprop”, LeCun’99</p>
</blockquote>
</aside>
<p>this one I assume:</p>
<aside class="onebox pdf" data-onebox-src="https://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf">
  <header class="source">

      <a href="https://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf" target="_blank" rel="noopener nofollow ugc">yann.lecun.com</a>
  </header>

  <article class="onebox-body">
    <a href="https://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>

<h3><a href="https://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf" target="_blank" rel="noopener nofollow ugc">lecun-98b.pdf</a></h3>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


rasbt: <p>Maybe it would be worthwhile adding comments to the docstrings? Would make it easier for the next person to find plus more convenient to refer to in papers.</p>