Upgrading parameters (form scratch)

username: Gilles_Jack
cooked: <p>I thought that in python x = x - a was the exactly the same as x-=a. But apparently not with tensor :</p>
<p>In the following code the gradient is not set to None after updating the parameter (x-=a case):</p>
<blockquote>
<p>import torch<br>
x1 = torch.tensor([2.0], requires_grad=True)<br>
x2 = torch.tensor([3.0], requires_grad=True)<br>
y = (x1 * x2).pow(2)<br>
y.backward()<br>
print(“x1.grad=”,x1.grad)<br>
with torch.no_grad():<br>
x1 -= 0.01 * x1.grad      # &lt;=============================</p>
<p>print(“x1.grad=”,x1.grad)<br>
x1.grad.zero_()<br>
print(“x1.grad=”,x1.grad)</p>
</blockquote>
<p>output:</p>
<blockquote>
<p>x1.grad= tensor([36.])<br>
x1.grad= tensor([36.])<br>
x1.grad= tensor([0.])</p>
</blockquote>
<p>but the following code the gradient is set to None (x = x - a) :</p>
<blockquote>
<p>import torch<br>
x1 = torch.tensor([2.0], requires_grad=True)<br>
x2 = torch.tensor([3.0], requires_grad=True)<br>
y = (x1 * x2).pow(2)<br>
y.backward()<br>
print(“x1.grad=”,x1.grad)<br>
with torch.no_grad():<br>
x1 = x1 - 0.01 * x1.grad     # &lt;=============================</p>
<p>print(“x1.grad=”,x1.grad)<br>
x1.grad.zero_()</p>
</blockquote>
<p>output:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “D:/gCloud/GoogleDrive/colabai/notes/torch/grad/zforum.py”, line 14, in <br>
x1.grad.zero_() # AttributeError: ‘NoneType’ object has no attribute ‘zero_’<br>
AttributeError: ‘NoneType’ object has no attribute ‘zero_’<br>
x1.grad= tensor([36.])<br>
x1.grad= None</p>
</blockquote>
<p>Both solutions are ok for me. I just would like to understand this behaviour? Thanks.</p>

username: JuanFMontesinos
cooked: <p>Because in python when you use <code>x-=</code> you are using in-place operations<br>
I don’t know how do the rules work in these case but basically when you do <code>x-=lr*grad</code> you are still pointing to the original tensor whose data has been in-place modified.</p>
<p>In the 2nd case you are overwritting a local variable <code>x1</code> so that when you try to print x1 it no longer points to the one you defined at the beggining but a “totally different variable” and the one at the beggining is out of scope.</p>
<pre><code class="lang-auto">import torch
x1 = torch.tensor([2.0], requires_grad=True)
x2 = torch.tensor([3.0], requires_grad=True)
y = (x1 * x2).pow(2)
y.backward()
print(x1._version)
print(x1.is_leaf)
x1 -= 0.01 * x1.grad
print(x1._version)
print(x1.is_leaf)

import torch
x1 = torch.tensor([2.0], requires_grad=True)
x2 = torch.tensor([3.0], requires_grad=True)
y = (x1 * x2).pow(2)
y.backward()
print(x1._version)
print(x1.is_leaf)
x1 =x1- 0.01 * x1.grad

print(x1._version)
print(x1.is_leaf)
</code></pre>
<p>if you run this, you will find that it’s not possible to run the 1st case</p>
<pre><code class="lang-auto">    x1 -= 0.01 * x1.grad
RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.
</code></pre>
<p>but you can run the second case in which x1 won’t be a leaf variable</p>
<p>if you add torch.no_grad() it will disable autograd engine and first case will be runable because there is no longer grad tracking and second case will be a leaf variable as cos of the same reason</p>

username: Gilles_Jack
cooked: <p>Thanks.</p>
<blockquote>
<p>print(id(x1))</p>
</blockquote>
<p>also shows your point clearly.</p>