Manual weight reset differs from first initialization

ArneNx: <p>Hey all,<br>
I narrowed my problem down to the following:</p>
<ol>
<li>I set a random seed and create a new model (this should call the standard init for every parameter)</li>
<li>I save this model (to look at those parameters later on)</li>
<li>I again set the original random seed</li>
<li>I reset the model parameters via</li>
</ol>
<pre><code class="lang-auto">def weight_reset(m):
    if (
        isinstance(m, nn.Conv1d)
        or isinstance(m, nn.Conv2d)
        or isinstance(m, nn.Linear)
        or isinstance(m, nn.Conv3d)
        or isinstance(m, nn.ConvTranspose1d)
        or isinstance(m, nn.ConvTranspose2d)
        or isinstance(m, nn.ConvTranspose3d)
        or isinstance(m, nn.BatchNorm1d)
        or isinstance(m, nn.BatchNorm2d)
        or isinstance(m, nn.BatchNorm3d)
        or isinstance(m, nn.GroupNorm)
    ):
        m.reset_parameters()
model.apply(weight_reset)
</code></pre>
<ol start="5">
<li>I compare the parameters for the original and the re-initialized model</li>
</ol>
<p>Result: Most parameters are equal, but some conv-layers differ (not all!)<br>
The model is mostly identical to the ResNet18 in torchvision.  (To simplify things, I removed the custom initialization from both the ResNet constructor and the weight-reset method.)<br>
Here is the output I get when I compare the parameters:</p>
<pre><code class="lang-auto">conv1.weight tensor(False)
bn1.weight tensor(True)
bn1.bias tensor(True)
layer1.0.conv1.weight tensor(False)
layer1.0.bn1.weight tensor(True)
layer1.0.bn1.bias tensor(True)
layer1.0.conv2.weight tensor(False)
layer1.0.bn2.weight tensor(True)
layer1.0.bn2.bias tensor(True)
layer1.1.conv1.weight tensor(False)
layer1.1.bn1.weight tensor(True)
layer1.1.bn1.bias tensor(True)
layer1.1.conv2.weight tensor(False)
layer1.1.bn2.weight tensor(True)
layer1.1.bn2.bias tensor(True)
layer2.0.conv1.weight tensor(False)
layer2.0.bn1.weight tensor(True)
layer2.0.bn1.bias tensor(True)
layer2.0.conv2.weight tensor(False)
layer2.0.bn2.weight tensor(True)
layer2.0.bn2.bias tensor(True)
layer2.0.downsample.0.weight tensor(False)
layer2.0.downsample.1.weight tensor(True)
layer2.0.downsample.1.bias tensor(True)
layer2.1.conv1.weight tensor(False)
layer2.1.bn1.weight tensor(True)
layer2.1.bn1.bias tensor(True)
layer2.1.conv2.weight tensor(False)
layer2.1.bn2.weight tensor(True)
layer2.1.bn2.bias tensor(True)
layer3.0.conv1.weight tensor(False)
layer3.0.bn1.weight tensor(True)
layer3.0.bn1.bias tensor(True)
layer3.0.conv2.weight tensor(False)
layer3.0.bn2.weight tensor(True)
layer3.0.bn2.bias tensor(True)
layer3.0.downsample.0.weight tensor(False)
layer3.0.downsample.1.weight tensor(True)
layer3.0.downsample.1.bias tensor(True)
layer3.1.conv1.weight tensor(False)
layer3.1.bn1.weight tensor(True)
layer3.1.bn1.bias tensor(True)
layer3.1.conv2.weight tensor(False)
layer3.1.bn2.weight tensor(True)
layer3.1.bn2.bias tensor(True)
layer4.0.conv1.weight tensor(False)
layer4.0.bn1.weight tensor(True)
layer4.0.bn1.bias tensor(True)
layer4.0.conv2.weight tensor(False)
layer4.0.bn2.weight tensor(True)
layer4.0.bn2.bias tensor(True)
layer4.0.downsample.0.weight tensor(False)
layer4.0.downsample.1.weight tensor(True)
layer4.0.downsample.1.bias tensor(True)
layer4.1.conv1.weight tensor(False)
layer4.1.bn1.weight tensor(True)
layer4.1.bn1.bias tensor(True)
layer4.1.conv2.weight tensor(False)
layer4.1.bn2.weight tensor(True)
layer4.1.bn2.bias tensor(True)
fc.weight tensor(False)
fc.bias tensor(False)
</code></pre>
<p>When I do the manual reset twice and compare the outputs, I can see that the output is identical.<br>
Does anyone have an idea what I am missing here?</p>

ptrblck: <p>The order of the layer initialization seems to be different, so that resetting the seed won’t yield the same results.<br>
After removing the <a href="https://github.com/pytorch/vision/blob/f9ef235c402f48a335293c626e17bd8504d3af87/torchvision/models/resnet.py#L160-L175">init methods</a> and resetting the seed before each layer creation (in the <code>ResNet</code> class as well as in <code>BasicBlock</code>, <code>conv1x1</code>, and <code>conv3x3</code>) I get the same results.</p>
<p>That being said, I wouldn’t rely on the seed to yield the exact same results in such a setup, where the order of operations might be different.<br>
I.e. the order in which each modules are passed to <code>model.apply()</code> is apparently different from the order these layers are created in the <code>ResNet</code> class.</p>

ArneNx: <p>Thank you for the quick answer!<br>
Then this must be the explanation.<br>
So as long as I want to perfectly reproduce the initial weights it’s safest to construct a new model or re-load the initial state-dict, right?<br>
I’m just glad there is no bug in the <code>weight_reset</code> method, so I can still use it as long as I don’t need a perfect reproduction of the initial weights.</p>

ptrblck: <aside class="quote no-group" data-username="ArneNx" data-post="3" data-topic="79570">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/9f8e36/48.png" class="avatar"> ArneNx:</div>
<blockquote>
<p>So as long as I want to perfectly reproduce the initial weights it’s safest to construct a new model or re-load the initial state-dict, right?</p>
</blockquote>
</aside>
<p>Yes, I would recommend to set the seed once in a script, initialize the model, and store the <code>state_dict</code>.<br>
This <code>state_dict</code> is then the reference for this particular seed.<br>
Making sure that all operations are executed in the same order is often too complicated.</p>