Updating the parameters in torch, delattr, setattr do not work as expected

username: Dara_Vahidi
cooked: <p>Hi<br>
I need to modify the parameters of a model, I am doing this</p>
<pre><code class="lang-auto">  delattr(base, localname)
  setattr(base, localname, param)
</code></pre>
<p>but when I update a parameter with setattr, this is already deleted from the state_dict, and I require to keep the value in the state_dict as well when I do setattr, so if I print</p>
<pre><code class="lang-auto">print("length of state_dict ", len(module.state_dict().keys()))
</code></pre>
<p>after the first operation this is 1 less, which I want this to be the same length after the second line, thanks a lot for your help</p>
<p>More context of what I am doing: I need to define all parameters of the model based on 1 parameter, I therefore define a parameter, then freeze all parameters and need to train only that one parameter, and compute and substitute all the rest of the parameters based on it.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> you are my hope.</p>

username: Dara_Vahidi
cooked: <p>Please note I cannot update paramerters like this as I said, I compute this parameter based on 1 shared parameter and that shared param is the only param</p>
<pre><code class="lang-auto">model.weight = nn.Parameter(torch.cat((model.weight, torch.randn(10, 1)), 1))
</code></pre>

username: ptrblck
cooked: <p>I’m not completely sure why are are unable to use the last code snippet, but also the first one seems to work:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.w0 = nn.Parameter(torch.randn(1))
        
    def forward(self, x):
        return x + self.w0

model = MyModel()
print(model.state_dict())
OrderedDict([('w0', tensor([0.4342]))])

delattr(model, 'w0')
print(model.state_dict())
&gt; OrderedDict()

setattr(model, 'w1', nn.Parameter(torch.randn(1)))
print(model.state_dict())
&gt; OrderedDict([('w1', tensor([0.9621]))])
</code></pre>

username: Dara_Vahidi
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
thanks for the response, the example you gave can only work if you define ‘w1’ as Parameter, lets see if this is not:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from collections import OrderedDict

class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.w0 = nn.Parameter(torch.randn(1))

    def forward(self, x):
        return x + self.w0

model = MyModel()
print(model.state_dict())
OrderedDict([('w0', tensor([0.4342]))])

delattr(model, 'w0')
print(model.state_dict())
&gt; OrderedDict()

setattr(model, 'w0', torch.randn(1)) #nn.Parameter(torch.randn(1)))
print(model.state_dict())
&gt; OrderedDict()
</code></pre>
<p>Basically, here is what I need to do, I freeze each parameter of the network, then I need to project it to a random space, like<br>
theta = theta + PX</p>
<p>then X is the shared parameter between all parameters of the network and the only one which gets updated and theta is the initial value for the parameter.</p>

username: albanD
cooked: <p>Only the <code>nn.Parameter</code> will show up in the state dict (or in the model.parameters()). This is expected.<br>
And this is also what you want right? As you want it to be shared across multiple models.</p>
<p>If you want something to be a regular Tensor and saved in the state dict as well, you can use buffers via <code>model.register_buffer("my_buffer", my_tensor)</code>.</p>

username: Dara_Vahidi
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/alband">@albanD</a><br>
Thank you for the response, I need to keep these other parameters in CPU, while state_dict with buffers push them to GPU when one do model.cuda(), I greatly appreciate assisting me how I can add them with buffers, while keeping the values into CPU, I am dealing with large-scale and this is crucial for me to keep them in CPU thanks</p>