Computing gradients of cloned parameters

Rouarez: <p>In the following code I clone a state dict of my model parameters and load the clone into the model.<br>
I would like to perform the autograd of the model with regards of the original state dict.<br>
I have created  the following code to test it:</p>
<pre><code class="lang-auto">import torch
from collections import OrderedDict
from torch import optim
import itertools

    

if __name__ == "__main__":
    #Initialising the model
    mlp = torch.nn.Sequential(torch.nn.Linear(1,2,bias=False))
    mlp[0].weight=torch.nn.Parameter(torch.tensor([1.0,2.0], requires_grad = True))

    # Taking storing dict of the model
    original_state_dict = OrderedDict([(k,v) for (k,v) in mlp.state_dict().items()])
    for item in original_state_dict.items():
        item[1].requires_grad= True

    # loading clone of dict of model
    mlp_bis_sttdict = OrderedDict([(k,torch.clone(v)) for (k,v) in original_state_dict.items()])
    mlp.load_state_dict(mlp_bis_sttdict)
    print(next(mlp.parameters()))


    # inference of model without stepping the parameters
    x=torch.tensor([10.0,100.0],requires_grad=True)
    optimizer = optim.Adam(
            mlp.parameters(),
            0.5)
    y=mlp(x)
    y.backward() # to change with grad = torch.autograd.grad(y, original_state_dict.values())

    

    # displaying grad
    print("## No step")
    print(list(iter(original_state_dict.values()))[0].grad)
    print(list(iter(mlp_bis_sttdict.values()))[0].grad)
    print(next(mlp.parameters()).grad)

    # stepping and displaying parameter changes
    print("## Step")
    optimizer.step()
    print(mlp_bis_sttdict)
    print(original_state_dict)
    print(mlp.state_dict())

</code></pre>
<p>I obtain the following results:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/1/8151ba9e90dde92b7b4317a9fd63726db4667e4b.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8151ba9e90dde92b7b4317a9fd63726db4667e4b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/8151ba9e90dde92b7b4317a9fd63726db4667e4b_2_690x94.jpeg" alt="image" data-base62-sha1="is0He85aVnafIaMsymgKogm0inh" width="690" height="94" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/8151ba9e90dde92b7b4317a9fd63726db4667e4b_2_690x94.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/8151ba9e90dde92b7b4317a9fd63726db4667e4b_2_1035x141.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/8151ba9e90dde92b7b4317a9fd63726db4667e4b_2_1380x188.jpeg 2x" data-dominant-color="1F1F1F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1580×216 74.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>why does the original state dict does not store the gradients but when stepping through the optimizer it is correctly updated?</p>

Rouarez: <p>I think I found an answer to my question:</p><aside class="quote quote-modified" data-post="1" data-topic="56676">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/evan_vogelbaum/48/13250_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/loading-a-state-dict-seems-to-erase-grad/56676">Loading a state dict seems to erase grad</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi Ya’ll, 
I am experiencing a somewhat spooky error, in which the simple act of loading a new state dictionary into my model seems to erase the graph tying parameters to an external matrix. 
Here is the code, and I will explain below. 
    for grad, (name, param) in zip(recovered, model.named_parameters()):
        updated_param = param - stepsize * grad  # This is 100% dependent on M
        new_state_dict[name] = updated_param

    model.load_state_dict(new_state_dict)
    l = list(model.para…
  </blockquote>
</aside>
<p>
WHen loading the graph, we don’t load the computational graph. Therefore the gradients are cannot be backpropagated to the dict object loaded into it.</p>