What is the recommended way to re-assign/update values in a variable (or tensor)?

Brando_Miranda: <p>why is the following:</p>
<pre><code>w.data = w.data - eta*w.grad.data
</code></pre>
<p>or</p>
<pre><code>w= w- eta*w.grad
</code></pre>
<p>is not recommended?</p>
<p>I actually got from the official tutorials:</p>
<p><a href="http://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-nn" class="onebox" target="_blank" rel="nofollow noopener">http://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-nn</a></p>
<pre><code>for param in model.parameters():
    param.data -= learning_rate * param.grad.data
and
</code></pre>
<p>Or</p>
<pre><code>w1.data -= learning_rate * w1.grad.data
w2.data -= learning_rate * w2.grad.data</code></pre>

fmassa: <p>Note that there is a difference from doing</p>
<pre><code class="lang-python">param.data -= learning_rate * param.grad.data
</code></pre>
<p>and</p>
<pre><code class="lang-python">param.data = param.data - learning_rate * param.grad.data
</code></pre>
<p>In the first case, the operation is performed in-place, so the python object is still the same, while in the second one you create a new object.</p>
<p>To give an example:</p>
<pre><code class="lang-python">a = torch.rand(3)
print(id(a))
b = a  # same python object
print(id(b))
b = b - 1
print(id(b)) # object reference changed!
a -= 1  # in-place operation doesn't change object
print(id(a))  # still the same object
</code></pre>

Brando_Miranda: <p>Thanks for the help! Just for completeness I will try to address my question with the best best solution I know so far:</p>
<pre><code>W.data.copy(new_value.data)
</code></pre>
<p>not sure if this is good or if there are advantages and disadvantages to it but Im going to leave it here for future people to benefit (and or discuss).</p>
<hr>
<p>I guess its a little sad that:</p>
<pre><code>W = W - eta*W.grad
</code></pre>
<p>doesn’t work cuz now it looks less like maths and a bit harder to read but eh, Im being a bit pedantic…</p>

Brando_Miranda: <p>now this is just me being curious, is the fact that <code>x=x+x</code> re-assigns namespace ids vs <code>x+=x</code> does inplace, a feature of python or a feature of pytorch? Like could <code>x=x+x</code> been made equivalent to <code>x+=x</code> if the developers of pytorch wanted? Just curious.</p>

Brando_Miranda: <p>sorry for being so insistent. But right now I am just immensely curious why:</p>
<pre><code>W.data = W.data - eta*W.grad.data
</code></pre>
<p>would be a bad idea. It seems it works fine when I try it (maybe cuz its not in place?)</p>

fmassa: <p>If you were to do <code>W = W - eta * W.grad</code>, you would be still storing the history of the computations for the update, which is not normally what you want to do.</p>

fmassa: <p>This is something from python, not specific to pytorch. The same is present in numpy.</p>

Brando_Miranda: <p>you mean if u did:</p>
<pre><code>W.data = W.data - eta*W.grad.data
</code></pre>
<p>if you did <code>W = W - eta * W.grad</code> that would create no variables so the previous history would be thrown away, no?</p>

fmassa: <aside class="quote no-group" data-username="Brando_Miranda" data-post="5" data-topic="6125" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/48/14355_2.png" class="avatar"> Brando_Miranda:</div>
<blockquote>
<p>sorry for being so insistent. But right now I am just immensely curious why:</p>
<p>W.data = W.data - eta*W.grad.data</p>
<p>would be a bad idea. It seems it works fine when I try it (maybe cuz its not in place?)</p>
</blockquote>
</aside>
<p>The fact that it works fine is a feature (as mentioned by <a class="mention" href="/u/apaszke">@apaszke</a> in slack), but there are reasons why it wouldn’t necessarily be the case.<br>
<code>W</code> is a <code>Variable</code> that holds a tensor in <code>W.data</code>. Now, what happens if you change the tensor that <code>W</code> originally points to, by doing <code>W.data = new_tensor</code>? <code>W</code> should now point to <code>new_tensor</code>, but <code>W</code> is a Variable that was supposed to represent the original tensor present.</p>

Brando_Miranda: <p>sorry last question. Whats wrong with storing the history of the computations? (you also mentioned that in the slack didn’t quite catch what was wrong with that)</p>

fmassa: <p>If you store indefinitely the history of computations, your computation graph will grow bigger at every iteration, and you will never free memory, leading to out of memory issues.</p>

alexandre-blanc: <p>I want to add for anyone getting to this place that the code</p>
<aside class="quote no-group" data-username="Brando_Miranda" data-post="3" data-topic="6125">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/48/14355_2.png" class="avatar"> Brando_Miranda:</div>
<blockquote>
<p><code>W.data.copy(new_value.data)</code></p>
</blockquote>
</aside>
<p>doesn’t work, since Tensors do not have a <code>copy</code> method.</p>
<p>Instead, use :<br>
<code>W.data.copy_(new_value.data)</code></p>
<p>Notice the underscore !</p>

mahmood: <p>I have a similar question. My model is trained I want to multiply the values of a particular conv layer how do that during testing?</p>