Unexpected output when extending nn.Linear with new output units

Tobias_Kalb: <p>Hi, I am trying to extend an exisiting trained nn.Linear() layer by new output units, as I want to train it for new classes as well. So the idea is to simply add output units to the existing ones. However, in my current implementation (see below) the output for the existing classes is not exactly the same as from the initial model, although the weights and biases are copied from before.<br>
When extending the network from 10 to 20 classes, it only ever effects the classes 9 and 10 not any other of the old classes 1 to 8.<br>
The difference between outputs is very small like 1e-8,  but still unexpected.<br>
Is this just some float precision error, or is my approach not working as intended?</p>
<pre><code class="lang-auto">import torch

in_channels = 64
n_classes_a = 10
n_classes_b = 20
random_input = torch.randn(2, 64)
a_layer = torch.nn.Linear(in_features=in_channels, out_features=n_classes_a, bias=True)
b_layer = torch.nn.Linear(in_features=in_channels, out_features=n_classes_b, bias=True)

# copying weight and bias for old classes
new_state_dict = b_layer.state_dict()
old_state_dict = a_layer.state_dict()
new_state_dict["weight"][:n_classes_a,] = old_state_dict["weight"]
new_state_dict["bias"][:n_classes_a,] = old_state_dict["bias"]
b_layer.load_state_dict(new_state_dict)

random_input = torch.randn(2, 64)
out_a = a_layer(random_input)
out_b = b_layer(random_input)
print(out_b[:,:n_classes_a] - out_a)
</code></pre>
<p>Example output:</p>
<pre><code class="lang-auto">tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,
          0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00, -5.9605e-08],
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,
          0.0000e+00,  0.0000e+00,  0.0000e+00,  8.9407e-08,  1.4901e-08]],
       grad_fn=&lt;SubBackward0&gt;)
</code></pre>
<p>Any help is really appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

KFrank: <p>Hi Tobias!</p>
<aside class="quote no-group" data-username="Tobias_Kalb" data-post="1" data-topic="142339" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tobias_kalb/48/36129_2.png" class="avatar"> Tobias_Kalb:</div>
<blockquote>
<p>Is this just some float precision error,</p>
</blockquote>
</aside>
<p>As you suspect, this is very likely floating-point round-off error.</p>
<p>As the size of a tensor changes, pytorch can change the details of<br>
how it pipelines tensor operations, both on the cpu and gpu.  This<br>
can cause operations to be reordered in a way that is mathematically<br>
equivalent, but that produces differing round-off errors.</p>
<p>You could check this by repeating your experiment in double-precision.<br>
I would then expect the discrepancy to be reduced by several orders of<br>
magnitude.</p>
<p>Note that the details of when and how the pipelining changes with<br>
tensor size will differ between the single-precision and double-precision<br>
cases, so you might have to play around with the tensor sizes to see<br>
a similar effect.</p>
<p>Best.</p>
<p>K. Frank</p>

Tobias_Kalb: <p>Thank you very much for the answer and explanation!<br>
I just tested it with double-precision and indeed the error is much smaller now, only in the range of 1e-17.<br>
So this is just neglectable precision error and not an error with my code.</p>