What is the default initialization of a conv2d layer and linear layer?

insomnia250: <p>Hey guys, when I train models for an image classification task, I tried replace the pretrained model’s last fc layer with a <code>nn.Linear</code> layer and a <code>nn.Conv2d</code> layer(by setting kernel_size=1 to act as a fc layer) respectively and found that two models performs differently. Specifically the conv2d one always performs better on my task.  I wonder  if it is because the different initialization methods for the two layers and what’s the default initialization method for a conv2d layer and linear layer in PyTorch. Thank you in advance.</p>

richard: <p>This is the initialization for linear:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/linear.py#L48-L52" target="_blank">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/linear.py#L48-L52" target="_blank">pytorch/pytorch/blob/master/torch/nn/modules/linear.py#L48-L52</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="48" style="counter-reset: li-counter 47 ;">
<li>def reset_parameters(self):</li>
<li>    stdv = 1. / math.sqrt(self.weight.size(1))</li>
<li>    self.weight.data.uniform_(-stdv, stdv)</li>
<li>    if self.bias is not None:</li>
<li>        self.bias.data.uniform_(-stdv, stdv)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>And this is the initialization for conv:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/08891b0a4e08e2c642deac2042a02238a4d34c67/torch/nn/modules/conv.py#L40-L47" target="_blank">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/08891b0a4e08e2c642deac2042a02238a4d34c67/torch/nn/modules/conv.py#L40-L47" target="_blank">pytorch/pytorch/blob/08891b0a4e08e2c642deac2042a02238a4d34c67/torch/nn/modules/conv.py#L40-L47</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="40" style="counter-reset: li-counter 39 ;">
<li>def reset_parameters(self):</li>
<li>    n = self.in_channels</li>
<li>    for k in self.kernel_size:</li>
<li>        n *= k</li>
<li>    stdv = 1. / math.sqrt(n)</li>
<li>    self.weight.data.uniform_(-stdv, stdv)</li>
<li>    if self.bias is not None:</li>
<li>        self.bias.data.uniform_(-stdv, stdv)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

insomnia250: <p>Thank you richard. It seems that they are initialized samely when acting as fc layer. But I’m more confused about why one model performs better than the other.<img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=5" title=":confused:" class="emoji" alt=":confused:"></p>

ptrblck: <p>Could you give some information regarding the input and output shape of the linear and conv layer?</p>

insomnia250: <p>The pretrained model’s feature map(after a avgpooling layer) is of shape <strong>(bs, 512, 1, 1)</strong>,<br>
<strong>so for nn.Conv2d layer,</strong> it should be<br>
<code>self.fc = nn.Conv2d(512, num_classes, kernel_size=1,stride=1, padding=0, bias=True)</code><br>
when forward,</p>
<pre><code class="lang-auto"># input x is the feature map
x = self.fc(x)
out = x.view(x.size(0), -1)
</code></pre>
<p><strong>for <code>nn.Linear</code>,</strong><br>
<code>self.fc = nn.Linear(512, num_classes, bias=True)</code><br>
when forward,<br>
<code>x = self.fc(x.view(x.size(0), -1))</code></p>

ptrblck: <p>This looks good. Is the “conv model” performing better in every run?<br>
You could use different seeds and check if it’s a random issue or systematic.</p>
<p>At the moment I don’t see any reason the conv layer should perform better than the linear layer.</p>

insomnia250: <p>Yeah, I also think it’s by coincidence. Anyway  I’ll do several further experiment and double check my code.<img src="https://discuss.pytorch.org/images/emoji/apple/neutral_face.png?v=5" title=":neutral_face:" class="emoji" alt=":neutral_face:"></p>

Ariel_biubiu: <p>Hi, is the initialization for conv2d xavier initialization?<br>
Thanks.</p>

ayumiymk: <p>Hi, have you solved this question?</p>

zyc: <aside class="quote no-group" data-username="richard" data-post="2" data-topic="16055">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/richard/48/1861_2.png" class="avatar"> Richard Zou:</div>
<blockquote>
<p><code>reset_parameters</code></p>
</blockquote>
</aside>
<p>It seems kaiming init is used for Convnd now:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/main/torch/nn/modules/conv.py#L110-L115">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/main/torch/nn/modules/conv.py#L110-L115" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/main/torch/nn/modules/conv.py#L110-L115" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/main/torch/nn/modules/conv.py#L110-L115</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="110" style="counter-reset: li-counter 109 ;">
          <li>self.output_padding = output_padding</li>
          <li>self.groups = groups</li>
          <li>self.padding_mode = padding_mode</li>
          <li># `_reversed_padding_repeated_twice` is the padding to be passed to</li>
          <li># `F.pad` if needed (e.g., for non-zero padding types that are</li>
          <li># implemented as two ops: padding + conv). `F.pad` accepts paddings in</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


julianolm: <p>I did not understand the initial assumption that a Conv2d with 1x1 filter size would act as a fully connected layer. I thought these were very different. Thinking in a single image: a fc layer would treat the image as an 1d vector and hece assign different weights for each pixel, while a 1x1 convolution would use the same filter in the whole image, corresponding to multiply all of the pixels for a same value. So this is not that the way things work?</p>

ptrblck: <p>Convolution and linear layers look quite different if you compare how the parameters are used in these operations, but note that you could use a matrix multiplication in a conv layer by unfolding the input and transforming the kernels appropriately (the input transformation is often called <code>im2col</code> in the literature).<br>
For a <code>1x1</code> kernel, you could also view and permute the input to get the same results, as seen here:</p>
<pre><code class="lang-python">x = torch.randn(2, 3, 24, 24)
conv = nn.Conv2d(3, 6, 1)
out_conv = conv(x)

lin = nn.Linear(3, 6)
with torch.no_grad():
    lin.weight.copy_(conv.weight.squeeze()) # remove spatial size
    lin.bias.copy_(conv.bias)
    
# permute x so that linear layer is executed on spatial dimension repeatedly
x = x.view(x.size(0), x.size(1), -1)
x = x.permute(0, 2, 1)
out_lin = lin(x)

# permute back to be able to compare results
out_lin = out_lin.permute(0, 2, 1)
out_lin = out_lin.view(out_lin.size(0), out_lin.size(1), 24, 24)

# compare
print((out_conv - out_lin).abs().max())
&gt; tensor(4.7684e-07, grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
<p>In particular these two layers are quite similar to each other. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

aoot: <p>According to (Liu et al., 2021)'s last sentence on page 7, all PyTorch networks defaults to the Xavier uniform initialization.</p>
<p>Liu, S., Li, X., Zhai, Y., You, C., Zhu, Z., Fernandez-Granda, C., &amp; Qu, Q. (2021). Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training. <em>Advances in Neural Information Processing Systems</em>, <em>34</em>, 28919–28928. <a href="https://proceedings.neurips.cc/paper_files/paper/2021/hash/f23d125da1e29e34c552f448610ff25f-Abstract.html" class="inline-onebox" rel="noopener nofollow ugc">Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training</a></p>