Weight initialization shape question

username: 111242
cooked: <p>I have a neural network with input shape = 1000, hidden = 500, and output = 2 neurons for classification. I am trying to do transfer learning, and my pretrained model is shaped shape = 1000, hidden = 500, and output = 200 neurons. However, when I try to initialize the classification model parameters with the learnt parameters, there is no error about different output layer neuron numbers.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8ec612fa1f042a4b2f3c3de6ceafacb82c4db934.png" data-download-href="https://discuss.pytorch.org/uploads/default/8ec612fa1f042a4b2f3c3de6ceafacb82c4db934" title="Screen Shot 2021-05-27 at 1.11.50 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8ec612fa1f042a4b2f3c3de6ceafacb82c4db934_2_653x500.png" alt="Screen Shot 2021-05-27 at 1.11.50 PM" data-base62-sha1="kn2aHJDkUsBliQtgQgba3tC2hHm" width="653" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8ec612fa1f042a4b2f3c3de6ceafacb82c4db934_2_653x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8ec612fa1f042a4b2f3c3de6ceafacb82c4db934_2_979x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8ec612fa1f042a4b2f3c3de6ceafacb82c4db934_2_1306x1000.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-05-27 at 1.11.50 PM</span><span class="informations">1642×1256 187 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b3020bd27f0b350a5b35167ee0469fe6f6568b94.png" data-download-href="https://discuss.pytorch.org/uploads/default/b3020bd27f0b350a5b35167ee0469fe6f6568b94" title="Screen Shot 2021-05-27 at 1.11.58 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3020bd27f0b350a5b35167ee0469fe6f6568b94_2_690x114.png" alt="Screen Shot 2021-05-27 at 1.11.58 PM" data-base62-sha1="pxzQLKE0AzeOJrAKuYYrE25aWXy" width="690" height="114" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3020bd27f0b350a5b35167ee0469fe6f6568b94_2_690x114.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3020bd27f0b350a5b35167ee0469fe6f6568b94_2_1035x171.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3020bd27f0b350a5b35167ee0469fe6f6568b94_2_1380x228.png 2x" data-dominant-color="DFDDDD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-05-27 at 1.11.58 PM</span><span class="informations">1630×270 58.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>No error above when trying to initialize by directly setting the data. However, when I initialize with load_state_dict(), the correct behavior occurs. Below:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/3/e3ea4b87538a49f711c5f07ad6174e4d88b46a70.png" data-download-href="https://discuss.pytorch.org/uploads/default/e3ea4b87538a49f711c5f07ad6174e4d88b46a70" title="Screen Shot 2021-05-27 at 1.12.09 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/3/e3ea4b87538a49f711c5f07ad6174e4d88b46a70_2_690x245.png" alt="Screen Shot 2021-05-27 at 1.12.09 PM" data-base62-sha1="wwemm5R0i5goq7m4eKgN9mSumzK" width="690" height="245" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/3/e3ea4b87538a49f711c5f07ad6174e4d88b46a70_2_690x245.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/3/e3ea4b87538a49f711c5f07ad6174e4d88b46a70_2_1035x367.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/3/e3ea4b87538a49f711c5f07ad6174e4d88b46a70_2_1380x490.png 2x" data-dominant-color="F2F2F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-05-27 at 1.12.09 PM</span><span class="informations">2998×1066 381 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>You are manually overriding the <code>.data</code> attribute, which won’t trigger any shape checks, so it would be your responsibility to make sure the parameters are assigned properly using this approach.<br>
With that being said, note that you should generally not use the <code>.data</code> attribute, as it could yield unwanted side effects. Instead, if you want to manually assign the parameters, wrap the assignment in a <code>with torch.no_grad()</code> block and use the <code>.weight</code> and <code>.bias</code> attributes.</p>

username: 111242
cooked: <p>Thank you for your response. When I initialize the weights with the incorrect number of neurons, that does not change the current classification network’s shape. For example, the classification network is shaped 1000-500-2 and the weights that I am initializing the model with is 1000-500-200. When I use <code>.data</code> to initialize, the resulting shape remains 1000-500-2. According to your response, shouldn’t it change the shape to 1000-500-200? Or am I understanding it vice versa?</p>

username: ptrblck
cooked: <p>The model will change, but since you’ve manipulated the internal parameters manually, the <code>in_features</code> and other attributes won’t be changed as seen here:</p>
<pre><code class="lang-python"># default setup
model = nn.Linear(10, 10, bias=False)
print(model)
&gt; Linear(in_features=10, out_features=10, bias=False)
x = torch.randn(1, 10)
out = model(x)
print(out.shape)
&gt; torch.Size([1, 10])

# manual manipulation
with torch.no_grad():
    model.weight = nn.Parameter(torch.randn(1, 10))
print(model)
&gt; Linear(in_features=10, out_features=10, bias=False) # wrong, as you've manually manipulated the parameter

out = model(x)
print(out.shape)
&gt; torch.Size([1, 1]) # new, expected output shape
</code></pre>
<p>I would generally not recommend to manipulate internals manually unless you are sure that’s the right approach.</p>

username: 111242
cooked: <p>Thank you so much for your clarification!</p>