Layer with modified input size shouldn't work, but it does. Is this a bug?

Valerio_Biscione: <p>I have found a strange behaviour in a model I was writing, so I coded this little snippet to replicate it.</p>
<p>Basically I create a new linear layer, <strong>self.my_fc</strong>, deep-copying another layer, <strong>self.fc1</strong>. I change the input size of this layer to 3200 or whatever. However, I apply a tensor to both <strong>self.my_fc</strong> and <strong>self.fc1</strong>. I expect my new layer to fail, as the number of input doesn’t much the expected ones. Instead, they both work.<br>
How is this possible?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
import copy

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 5) 
        self.conv2 = nn.Conv2d(6, 16, 5) 
        self.fc1 = nn.Linear(16 * 20 * 20, 120) 
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

        self.my_fc = copy.deepcopy(self.fc1)
        self.my_fc.in_features = 3200

    def forward(self, x):
        x1 = F.relu(self.conv1(x))
        x2 = F.relu(self.conv2(x1))
        x3 = x2.view(-1, 16 * 20 * 20) #  x3 has shape [1, 6400]
        x4 = F.relu(self.fc1(x3)) #  fc1 is Linear(in_features=6400, out_features=120, bias=True)
        # I run the same tensor x3 to my layer, which expects a different number of features.
        # It should fail (x3 is size [1, 6400]), but it works! 
        xxx = self.my_fc(x3) ## &lt;---- Linear(in_features=3200, out_features=120, bias=True)
        # xxx has shape [1, 120] as expected
        x5 = F.relu(self.fc2(x4))
        x6 = self.fc3(x5)
        return x6

net = Net()
net(torch.rand(1, 1, 28, 28))
</code></pre>
<p>I also found that, sometime, by changing the input feature to THE SAME input feature as the deep-copied layer, the operation DOESN’T WORK and it actually complain about mismatch! I think the problems are related, but it would be a bit more tricky to create a minimal reproducible example so let’s start investigate this one <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>How course it is totally possible that I am doing something wrong!</p>

KFrank: <p>Hello Valerio!</p>
<aside class="quote no-group quote-modified" data-username="Valerio_Biscione" data-post="1" data-topic="78531" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/valerio_biscione/48/22759_2.png" class="avatar"> Valerio_Biscione:</div>
<blockquote>
<p>Basically I create a new linear layer, <strong>self.my_fc</strong>, deep-copying another layer, <strong>self.fc1</strong>. I change the input size of this layer to 3200 or whatever. However, I apply a tensor to both <strong>self.my_fc</strong> and <strong>self.fc1</strong>. I expect my new layer to fail, as the number of input doesn’t much the expected ones. Instead, they both work.</p>
<pre><code class="lang-auto">        self.fc1 = nn.Linear(16 * 20 * 20, 120) 

        self.my_fc = copy.deepcopy(self.fc1)
        self.my_fc.in_features = 3200
</code></pre>
</blockquote>
</aside>
<p>I think that <code>self.my_fc.in_features = 3200</code> is, in effect, lying to<br>
your <code>Linear</code> layer.  You already constructed it with an input<br>
dimension of 6400.</p>
<p>Here’s an example script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import copy

torch.__version__

fc1 = torch.nn.Linear (10, 2)
my_fc = copy.deepcopy (fc1)
my_fc.in_features = 5

fc1.in_features
fc1.weight
my_fc.in_features
my_fc.weight
</code></pre>
<p>Here is the output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; import copy
&gt;&gt;&gt;
&gt;&gt;&gt; torch.__version__
'0.3.0b0+591e73e'
&gt;&gt;&gt;
&gt;&gt;&gt; fc1 = torch.nn.Linear (10, 2)
&gt;&gt;&gt; my_fc = copy.deepcopy (fc1)
&gt;&gt;&gt; my_fc.in_features = 5
&gt;&gt;&gt;
&gt;&gt;&gt; fc1.in_features
10
&gt;&gt;&gt; fc1.weight
Parameter containing:
 0.0767  0.1974  0.0051  0.1165  0.0270 -0.2915 -0.0038 -0.2985  0.2314  0.2616
 0.0099 -0.0060  0.1231  0.1886  0.2331  0.0675 -0.1929 -0.1226 -0.1118 -0.2078
[torch.FloatTensor of size 2x10]

&gt;&gt;&gt; my_fc.in_features
5
&gt;&gt;&gt; my_fc.weight
Parameter containing:
 0.0767  0.1974  0.0051  0.1165  0.0270 -0.2915 -0.0038 -0.2985  0.2314  0.2616
 0.0099 -0.0060  0.1231  0.1886  0.2331  0.0675 -0.1929 -0.1226 -0.1118 -0.2078
[torch.FloatTensor of size 2x10]
</code></pre>
<p>As you can see, setting <code>my_fc.in_features</code> doesn’t actually change<br>
the <code>weight</code> tensor, so (presumably) doesn’t actually change the<br>
expected shape of the input tensor.</p>
<p>You could argue that this is a bug in that it shouldn’t be possible<br>
to create such an “inconsistent” <code>Linear</code> for which <code>in_features</code><br>
and <code>weight.shape</code> don’t match.</p>
<aside class="quote no-group">
<blockquote>
<p>I also found that, sometime, by changing the input feature to THE SAME input feature as the deep-copied layer, the operation DOESN’T WORK and it actually complain about mismatch!</p>
</blockquote>
</aside>
<p>I would guess that this is the same issue.  You’re trying to change<br>
<code>in_features</code> to match the shape of your input tensor, but this<br>
doesn’t actually change the shape of the <code>weight</code> tensor, so you<br>
do get a mismatch.</p>
<p>Good luck.</p>
<p>K. Frank</p>

Valerio_Biscione: <p>Thanks! That was extremely clear and it makes total sense <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I would suggest them to put an underscore  at the beginning of the parameter name, to indicate internal use, or to set it only for reading, not writing.</p>