Layer weight vs. weight.data

username: James_McCaffrey
cooked: <p>I’m new to PyTorch. Suppose you want to set layer weights to specific values. Is it better to set mynet.layer.weight[I][j] or set mynet.layer.weight.data[I][j] or does it matter?</p>
<p>For example, mynet.layer.weight[0][0] = 9.9999 and mynet.layer.weight.data[0][0] = 9.9999 both seem to have the same effect. The weight is a Parameter object and weight.data is a Tensor object but I don’t know what the implications are.</p>
<p>This is just investigation on my part. Thanks to some previous feedback from user ‘ptrblck’ I understand the Torch.nn.init functions for practical use.</p>

username: ptrblck
cooked: <p>Manipulating the weights directly would most likely give you an error when you try to call backward:</p>
<pre><code class="lang-python">lin = nn.Linear(10, 2)
lin.weight[0][0] = 1.
x = torch.randn(1, 10)
output = lin(x)
output.mean().backward()
&gt; RuntimeError: leaf variable has been moved into the graph interior
</code></pre>
<p>Using <code>.data</code> on the other side would work, but is generally not recommended, as changing it after the model was used would yield weird results and autograd cannot throw an error.<br>
I would recommend to use <code>with torch.no_grad():</code></p>
<pre><code class="lang-python">lin = nn.Linear(10, 2)
with torch.no_grad():
    lin.weight[0][0] = 1.

x = torch.randn(1, 10)
output = lin(x)
output.mean().backward()
</code></pre>

username: James_McCaffrey
cooked: <p>Ah! That makes perfect sense. I noticed that manipulating the weights changed the ‘requires_grad=True’ to ‘grad_fn=(CopySlices)’. Thank you!  JM</p>

username: 32c22ec2e75125e74133
cooked: <p>it works！！！！！！！！<br>
Thank you very much:grin:</p>

username: Santosh-Gupta
cooked: <p>Thanks for the answer ptrblck!!</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="24271">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Manipulating the weights directly would most likely give you an error when you try to call backward:</p>
</blockquote>
</aside>
<p>I am wondering why this is?</p>
<p>" but is generally not recommended, as changing it after the model was used would yield weird results and autograd cannot throw an error."</p>
<p>Is there any other reasons for this besides the autograd thing?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Santosh-Gupta" data-post="5" data-topic="24271">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/santosh-gupta/48/30716_2.png" class="avatar"> Santosh-Gupta:</div>
<blockquote>
<p>I am wondering why this is?</p>
</blockquote>
</aside>
<p>I think because an arbitrary manipulation is not a valid operation, which would yield valid gradients.<br>
Have a look at this dummy example:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(1)
w = torch.randn(1, requires_grad=True)

y = x*w
#w[0] = 1.
y.backward()
print(w.grad)
</code></pre>
<p>If you uncomment the particular line, you’ll get an error.</p>
<aside class="quote no-group" data-username="Santosh-Gupta" data-post="5" data-topic="24271">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/santosh-gupta/48/30716_2.png" class="avatar"> Santosh-Gupta:</div>
<blockquote>
<p>Is there any other reasons for this besides the autograd thing?</p>
</blockquote>
</aside>
<p>Note that manipulating the <code>.data</code> might yield a wrong result in your calculations, which I would consider a strong argument against using this approach.</p>

username: franguzmang1
cooked: <p>thanks it worked for me!</p>