Changing state dict value is not changing model

Sudarshan_Babu: <p>I am trying to change the value in my model’s state dict, but even after updating the state dict, the value does not change, any help would be appreciated.</p>
<pre><code class="lang-auto">sd = model.state_dict() 
sd['encoder.layer.11.output.LayerNorm._running_mean'] = layer_norm_stats['encoder.layer.11.output.LayerNorm._running_mean']  # Layer norm stats is a dict containing mean of the layer norm 
print(model.state_dict()['encoder.layer.11.output.LayerNorm._running_mean'])  # prints all zero tensor, which is the original value
print(layer_norm_stats['encoder.layer.11.output.LayerNorm._running_mean'] )  # prints the running mean, which is not the zero vector
</code></pre>

chetan_patil: <p>Shouldn’t you look at<br>
<code>sd['encoder.layer.11.output.LayerNorm._running_mean'] </code> in the second print statement if it has changed ?</p>

Sudarshan_Babu: <p>Thanks for your reply, that is my question, should not changing a copy of the state dict also directly change the model values</p>

ptrblck: <p>The values in the model parameters won’t be changed, if you assign a new tensor to the key in the <code>state_dict</code>.<br>
You could either load the manipulated <code>state_dict</code> afterwards or change the parameter’s value inplace as shown here:</p>
<pre><code class="lang-python">model = nn.Linear(1, 1)
print(model.weight)
&gt; Parameter containing:
tensor([[0.8777]], requires_grad=True)

sd = model.state_dict()
sd['weight'] = torch.tensor([[1.]])
print(model.weight)
&gt; Parameter containing:
tensor([[0.8777]], requires_grad=True)

model.load_state_dict(sd)
print(model.weight)
&gt; Parameter containing:
tensor([[1.]], requires_grad=True)


# or
model = nn.Linear(1, 1)
print(model.weight)
&gt; Parameter containing:
tensor([[-0.8112]], requires_grad=True)

with torch.no_grad():
    sd = model.state_dict()
    sd['weight'].fill_(1.)
print(model.weight)
&gt; Parameter containing:
tensor([[1.]], requires_grad=True)
</code></pre>

Sudarshan_Babu: <p>Thank you that was helpful!</p>

bvk: <p>could you please explain why is the design like this? because if one does deepcopy and then re-assign, it gets updated.</p>

ptrblck: <p>Could you explain your concern a bit more, please?<br>
I don’t understand which object you are creating a <code>deepcopy</code> of and what your use case is.</p>

bvk: <p>Okay, I am sorry for not being clear the last time. Here is something that is weird for me:</p>
<p>as you have shown above one will have to run <code>model.load_state_dict(sd)</code> again in order to see the change in the weights in the <code>model</code> variable. But the following sequence of commands also changes the value in the model variable, and it is confusing:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; model = nn.Linear(1, 1)
&gt;&gt;&gt; sd = model.state_dict()
&gt;&gt;&gt; sd['weight'].copy_(torch.tensor([[1.]]))
</code></pre>
<p>On printing <code>model.weight</code> after the above sequence of instructions one will find out that the <code>model</code>’s weights have been updated! With whatever I understand <code>Tensor.copy_()</code> behaves something similar to a <code>deepcopy</code> - but this behavior is counterintuitive.</p>

ptrblck: <p>Thanks for the explanation. As explained in <a href="https://discuss.pytorch.org/t/changing-state-dict-value-is-not-changing-model/88695/4">this post</a> references are stored in the <code>state_dict</code> and thus inplace operations are visible in the original model, too.</p>
<aside class="quote no-group" data-username="bvk" data-post="8" data-topic="88695">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bvk/48/51182_2.png" class="avatar"> bvk:</div>
<blockquote>
<p>With whatever I understand <code>Tensor.copy_()</code> behaves something similar to a <code>deepcopy</code> - but this behavior is counterintuitive.</p>
</blockquote>
</aside>
<p>No, that’s wrong as <code>.copy_()</code> is an inplace operation and does not create a separate copy, but manipulates the underlying data directly inplace.</p>

bvk: <p>Thanks for the quick response and for clarifying that <code>.copy_()</code> is an in-place operation. Any reason it is designed this way? It is weird that when I am using the normal assignment, the <code>sd</code> variable we defined does not behave as a reference, but while using <code>.copy_()</code> it does!.</p>

ptrblck: <p>The <code>state_dict</code> uses references e.g. to avoid wasting memory. Otherwise each <code>model.state_dict()</code> call would create completely new tensors, which will increase the memory by the model’s parameter and buffer size. Assigning a new tensor to a <code>dict</code> will break the reference, so this behavior is also expected.</p>

bvk: <aside class="quote no-group" data-username="ptrblck" data-post="11" data-topic="88695">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Assigning a new tensor to a <code>dict</code> will break the reference, so this behavior is also expected.</p>
</blockquote>
</aside>
<p>Then this should be true for all types of assignments right, not just in-place assignments?<br>
Thanks again for the quick response.</p>

ptrblck: <p>I don’t know which use case you are thinking about but generally yes: creating a new tensor and assigning it to any attribute/key/etc. will overwrite the existing object and will not manipulate it inplace.</p>

bvk: <p>NP. Thanks for the clarification!</p>

andy_sherlock: <p>Sorry to bother you, I am still confused about this. Does “The state_dict uses references” mean that when I call the .state_dict() every time, it will return the parameters’ references? So that even I let .state_dict()[“weight”]=1 which breaks the references and makes the value of the dict change, the return of the .state_dict() will not change.</p>
<pre><code class="lang-auto">for param in model.state_dict():
    model.state_dict()[param] =1

for param in model.state_dict():
    print(param, model.state_dict()[param])
</code></pre>

ptrblck: <p>Yes, as seen in your code. You will replace the stored parameter reference via the <code>param</code> key with any object that also does not have to be a valid tensor as seen in your example. If you want to manipulate parameters through the <code>state_dict</code>, use inplace operations:</p>
<pre data-code-wrap="python"><code class="lang-python">model = nn.Linear(1, 1)

for param in model.state_dict():
    model.state_dict()[param].fill_(1.)

for param in model.state_dict():
    print(param, model.state_dict()[param])

# weight tensor([[1.]])
# bias tensor([1.])
</code></pre>

andy_sherlock: <p>There are no other problems, and I sincerely thank you for your generous and clear reply! Wish you a good day <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>