Confused by autodiff behaviour

Sourena_Yadegari: <p>I thought I had a good grasp of automatic differentiation but I cannot make sense of the following snippet I wrote:</p>
<pre><code class="lang-python">import torch

def f1(t):
    return t ** 2

def f2(t):
    return t ** 3

def get_t():
    return torch.tensor([1., 2.], requires_grad=True)

# case 1
t = get_t()
print('grad after initialization', t.grad)

y = (f1(t)).norm()
y.backward()
print('case 1 grad ', t.grad)

# case 2
t = get_t()
print('grad after initialization', t.grad)

y = (f1(t) + f2(t).detach()).norm()
y.backward()
print('case 2 grad ', t.grad)

# case 3
t = get_t()
print('grad after initialization', t.grad)

y = (f1(t) + f2(t)).norm()
y.backward()
print('case 3 grad ', t.grad)
</code></pre>
<p>With the following output:</p>
<pre><code class="lang-plaintext">grad after initialization None
case 1 grad  tensor([0.4851, 3.8806])
grad after initialization None
case 2 grad  tensor([0.3288, 3.9456])
grad after initialization None
case 3 grad  tensor([ 0.8220, 15.7823])
</code></pre>
<p>My understanding is that case 1 and case 2 should have the same gradients, since I detach the value of <code>f2</code> before adding it to <code>f1</code>. I tried different versions of this code (<code>with torch.no_grad</code> and deleting the intermediate variables) but the values didn’t change. Can someone help with this?</p>

ptrblck: <aside class="quote no-group" data-username="Sourena_Yadegari" data-post="1" data-topic="187681">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sourena_yadegari/48/14001_2.png" class="avatar"> Sourena_Yadegari:</div>
<blockquote>
<p>My understanding is that case 1 and case 2 should have the same gradients, since I detach the value of <code>f2</code> before adding it to <code>f1</code>.</p>
</blockquote>
</aside>
<p>It would depend on the next operations and you will see your expected results if you reduce the values via <code>.sum()</code> or <code>.mean()</code>.<br>
To see the influence of a constant on the <code>norm</code> you could write down the calculation and check the gradients manually:</p>
<pre data-code-wrap="python"><code class="lang-python">t = get_t()
tmp = f1(t)
#loss = (tmp**2).sum().sqrt() 
loss = ((tmp+100.)**2).sum().sqrt()
loss.backward()
print(t.grad)
</code></pre>

Sourena_Yadegari: <p>I was confused about this and I see your point <a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks! As an analogy, this is like to expect that the derivation of <code>(x + 1)^2</code> and <code>x^2</code> with respect to <code>x</code> to be the same (<code>1</code> here plays the role of the detached tensor and <code>^2</code> plays the role of the norm operation).</p>

ptrblck: <p>Not quite, since in your example the constant won’t have any effect on <code>x</code>:</p>
<pre><code class="lang-python">x = torch.randn(1, requires_grad=True)

y = (x**2 + 1)
y.backward()
print(x.grad)
# tensor([0.0637])

x.grad = None
y = (x**2)
y.backward()
print(x.grad)
# tensor([0.0637])
</code></pre>
<p>However, if you use <code>(x + a)**2</code> it will since it’s equal to <code>(x**2 + 2xa + a**2)</code>:</p>
<pre><code class="lang-python">x.grad = None
y = ((x + 1)**2)
y.backward()
print(x.grad)
# tensor([2.0637])

x.grad = None
y = (x**2 + 2*x + 1)
y.backward()
print(x.grad)
# tensor([2.0637])
</code></pre>

Sourena_Yadegari: <p>Sorry, example was correct in my head but I wrote it wrong! It meant to be <code>(x + 1)^2</code>. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>