How to assign an arbitrary tensor to model's parameter?

HX-idiot: <p>I try like this , it didn’t  report bug, but the value of parameter is not changed.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/4ca0dda842847b6664b3c59a311196e38293b60b.png" data-download-href="https://discuss.pytorch.org/uploads/default/4ca0dda842847b6664b3c59a311196e38293b60b" title="40"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4ca0dda842847b6664b3c59a311196e38293b60b_2_690x202.png" alt="40" data-base62-sha1="aVSWm98EbwgYt08LEnajQOsXyT9" width="690" height="202" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4ca0dda842847b6664b3c59a311196e38293b60b_2_690x202.png, https://discuss.pytorch.org/uploads/default/optimized/2X/4/4ca0dda842847b6664b3c59a311196e38293b60b_2_1035x303.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/4/4ca0dda842847b6664b3c59a311196e38293b60b_2_1380x404.png 2x" data-dominant-color="F2F2F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">40</span><span class="informations">1832×538 70.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>But using torch.nn.init.xavier_normal_(model_0.state_dict()[‘classifier.weight’]) , it works, why?</p>

HX-idiot: <p>Also, I try to use named_parameter to select the parameter I want to initialize:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/a6a3229281791a793b4e0b413399e5d153773bf8.png" data-download-href="https://discuss.pytorch.org/uploads/default/a6a3229281791a793b4e0b413399e5d153773bf8" title="18"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/a6a3229281791a793b4e0b413399e5d153773bf8_2_690x123.png" alt="18" data-base62-sha1="nM8M4ladHXurZc0Iq5z4nuClIU0" width="690" height="123" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/a6a3229281791a793b4e0b413399e5d153773bf8_2_690x123.png, https://discuss.pytorch.org/uploads/default/original/2X/a/a6a3229281791a793b4e0b413399e5d153773bf8.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/a/a6a3229281791a793b4e0b413399e5d153773bf8.png 2x" data-dominant-color="EAEAE9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">18</span><span class="informations">862×154 15.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
But it didn’t work, the value of parameter is not changed.</p>
<p>But when I use:   v.data=regression_fc.data   It works.</p>
<p>Why? What’s the difference between v=regression_fc and v.data=regression_fc.data ?<br>
I alsways assign value to Variable by simply using variable=xxx , when should I need to add .data?</p>

ptrblck: <p>You could modify the <code>state_dict</code> and reload it afterwards:</p>
<pre><code class="lang-python">state_dict = model.state_dict()
state_dict['classifier.weight'] = torch.randn(10, 10)
model.load_state_dict(state_dict)
</code></pre>
<p>Also, if you are using the loop over <code>named_parameters</code>, make sure to manipulate the parameters inplace and with a <code>torch.no_grad()</code> guard:</p>
<pre><code class="lang-python">with torch.no_grad():
    for name, param in model.named_parameters():
        if 'classifier.weight' in name:
            param.copy_(torch.randn(10, 10))
</code></pre>
<p>I wouldn’t recommend the usage of the <code>.data</code> attribute, as Autograd cannot track this operations and you might create silent bugs in your code.</p>

asoin29: <p>How about something like:</p>
<pre><code class="lang-auto">model.classifier.weight = torch.nn.Parameter(torch.randn(10, 10))
</code></pre>
<p>Could this work <a class="mention" href="/u/ptrblck">@ptrblck</a>?  Modifying the state_dict and reloading it essentially gives the same torch size mismatch error.</p>

ptrblck: <p>Your approach could work, if you want to change the parameter of the model instead of the <code>state_dict</code>.</p>
<aside class="quote no-group" data-username="asoin29" data-post="4" data-topic="44082">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/asoin29/48/28966_2.png" class="avatar"> asoin29:</div>
<blockquote>
<p>Modifying the state_dict and reloading it essentially gives the same torch size mismatch error.</p>
</blockquote>
</aside>
<p>That should not be the case, if you make sure the parameter in the <code>state_dict</code> has the same shape as the parameter in the model.</p>

asoin29: <p>The parameter in the <code>state_dict</code> that I’m trying to load is from a checkpoint where the classifier.weight and classifier.bias have different Tensor sizes from those of the model. I think that’s why it doesn’t work?</p>
<pre><code class="lang-auto">size mismatch for classifier.bias: copying a param with shape torch.Size([42]) from checkpoint, the shape in current model is torch.Size([1000]).
</code></pre>

ptrblck: <p>Manipulating the <code>state_dict</code> works for me:</p>
<pre><code class="lang-python">class MyModelA(nn.Module):
    def __init__(self):
        super(MyModelA, self).__init__()
        self.lin = nn.Linear(10, 10)
        self.classifier = nn.Linear(10, 42)
        
    def forward(self, x):
        x = F.relu(self.lin(x))
        x = self.classifier(x)
        return x


class MyModelB(nn.Module):
    def __init__(self):
        super(MyModelB, self).__init__()
        self.lin = nn.Linear(10, 10)
        self.classifier = nn.Linear(10, 1000)
        
    def forward(self, x):
        x = F.relu(self.lin(x))
        x = self.classifier(x)
        return x

modelA = MyModelA()
modelB = MyModelB()

sd = modelA.state_dict()
modelB.load_state_dict(sd) # your error

# replace tensors creating the shape mismatch with random tensors
sd['classifier.weight'] = torch.randn(1000, 10)
sd['classifier.bias'] = torch.randn(1000)
 
modelB.load_state_dict(sd) # works
</code></pre>

julianolm: <aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="44082">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>That should not be the case, if you make sure the parameter in the <code>state_dict</code> has the same shape as the parameter in the model.</p>
</blockquote>
</aside>
<p>What if this is not what I want? I mean, if I want to give the model’s parameter a new one with a different shape. How could I do it?</p>

ptrblck: <p>You can manually assign the new parameter to the model’s parameter:</p>
<pre><code class="lang-python">lin = nn.Linear(10, 10, bias=False)
print(lin)
&gt; Linear(in_features=10, out_features=10, bias=False)

x = torch.randn(1, 10)
out = lin(x)
print(out.shape)
&gt; torch.Size([1, 10])

with torch.no_grad():
    lin.weight = nn.Parameter(torch.randn(1, 10)) # out_features, in_features

out = lin(x)
print(out.shape)
&gt; torch.Size([1, 1])

print(lin) # wrong information
&gt; Linear(in_features=10, out_features=10, bias=False)
</code></pre>
<p>but note that the attributes won’t be automatically changes as well, so you might want to change them also manually.</p>

cheeseprata: <p>Thank you for the answer.</p>
<p>What should I do instead if I wish to backprop through the variables which I have copied to the param? i.e. without using “torch.no_grad()”?</p>

farah: <p>In</p>
<p>model.classifier.weight = torch.nn.Parameter(torch.randn(10, 10))</p>
<p>Should we assign it torch.no_grad() block? If we do it, would  model.classifier.weight still have requires_grad = true?.</p>

neva: <p>I tried it it dosent work for some reason</p>

ptrblck: <p>Do you see some errors when using this approach or unexpected results?</p>

zzx: <p>This question is really interesting! I am also working on the similar things. But I find another problem when using the function “copy_” (also other functions like “set_” with “_” at the end).</p>
<p>param.set_(param.data - lr * (a number))<br>
It will bring an error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6d6fd30c7e01d32f2ac8a62c73ab9a0d594c4b3.png" data-download-href="https://discuss.pytorch.org/uploads/default/b6d6fd30c7e01d32f2ac8a62c73ab9a0d594c4b3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6d6fd30c7e01d32f2ac8a62c73ab9a0d594c4b3.png" alt="image" data-base62-sha1="q5tvI989NZllfN2CqaUZu9XsXxp" width="690" height="50" data-dominant-color="FAF5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1450×107 3.94 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
However, if I use param.data, then it works well.<br>
param.data = param.data - lr * (a number))</p>

ptrblck: <p>PyTorch rightfully raises the error since inplace operations on leaf variables are disallowed in a differentiable context. Using the deprecated <code>.data</code> attribute will skip valid checks and you should wrap the inplace operation into a <code>with torch.no_grad()</code> context instead.</p>