Scale gradient without scaling loss values

username: lisyuan
cooked: <p>Hi everyone,</p>
<p>I recently watch some code style for scaling gradient values without scaling loss value, but I am confusing about it. I implemented a simple example code myself below.</p>
<pre><code class="lang-auto">torch.manual_seed(0)
x = torch.randn(2, 3, requires_grad=True)
y = 5

z = 2 * x
z_backward = z * y
z_new = z.detach() + (z_backward - z_backward.detach())
z_new2 = z + (z_backward - z_backward.detach())

print('------------------------')
print(f'x input: {x}')
print(f'x gradient: {x.grad}')
print('------------------------')
print(f'z: {z}')
print(f'z_new: {z_new}')
print('------------------------')
z.mean().backward(retain_graph=True)
print(f'x gradient from z :{x.grad}')
x.grad.zero_()
z_new.mean().backward(retain_graph=True)
print(f'x gradient from z_new :{x.grad}')
x.grad.zero_()
z_backward.mean().backward(retain_graph=True)
print(f'x gradient from z_backward :{x.grad}')
x.grad.zero_()
z_new2.mean().backward(retain_graph=True)
print(f'x gradient from z_new2 :{x.grad}')
print('------------------------')
</code></pre>
<p>The corresponding outputs are showing below</p>
<pre><code class="lang-auto">------------------------
x input: tensor([[ 1.5410, -0.2934, -2.1788],
        [ 0.5684, -1.0845, -1.3986]], requires_grad=True)
x gradient: None
------------------------
z: tensor([[ 3.0820, -0.5869, -4.3576],
        [ 1.1369, -2.1690, -2.7972]], grad_fn=&lt;MulBackward0&gt;)
z_new: tensor([[ 3.0820, -0.5869, -4.3576],
        [ 1.1369, -2.1690, -2.7972]], grad_fn=&lt;AddBackward0&gt;)
------------------------
x gradient from z :tensor([[0.3333, 0.3333, 0.3333],
        [0.3333, 0.3333, 0.3333]])
x gradient from z_new :tensor([[1.6667, 1.6667, 1.6667],
        [1.6667, 1.6667, 1.6667]])
x gradient from z_backward :tensor([[1.6667, 1.6667, 1.6667],
        [1.6667, 1.6667, 1.6667]])
x gradient from z_new2 :tensor([[2., 2., 2.],
        [2., 2., 2.]])
------------------------
</code></pre>
<p>I can understand that the gradient from <code>z</code> is 0.3333, which is 2 divided by 6. It is also easy to understand that gradient from <code>z_backward </code> is scaled by 5.</p>
<p>However, what I am confusing is about gradient from <code>z_new </code> and <code>z_new2</code>. The output of <code>z</code>, <code>z_new</code> and <code>z_new2</code> generate the same values but different gradient value corresponding to <code>x</code></p>
<p>Could anyone answer me how the values from <code>z_new</code> and <code>z_new2</code> are generated ? Also, the gradent values from <code>z_backward</code> and <code>z_new</code> are the same. What is the difference between these two?</p>
<p>Thanks for any suggestion in advance.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="lisyuan" data-post="1" data-topic="178060">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/6a8cbe/48.png" class="avatar"> lisyuan:</div>
<blockquote>
<p>Could anyone answer me how the values from <code>z_new</code> and <code>z_new2</code> are generated ?</p>
</blockquote>
</aside>
<p><code>z_new</code> uses <code>z_backward</code> to calculate the gradients as its the only attached tensor to the computation graph. The other two tensors were explicitly detached.<br>
<code>z_new2</code> uses <code>z</code> and <code>z_backward</code> to calculate the gradients as both are attached to the computation graph.</p>
<aside class="quote no-group" data-username="lisyuan" data-post="1" data-topic="178060">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/6a8cbe/48.png" class="avatar"> lisyuan:</div>
<blockquote>
<p>Also, the gradent values from <code>z_backward</code> and <code>z_new</code> are the same.</p>
</blockquote>
</aside>
<p>That’s expected since <code>z_new</code> can only use <code>z_backward</code> during its backward pass since the other two tensors are detached and thus constants.</p>
<aside class="quote no-group" data-username="lisyuan" data-post="1" data-topic="178060">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/6a8cbe/48.png" class="avatar"> lisyuan:</div>
<blockquote>
<p>What is the difference between these two?</p>
</blockquote>
</aside>
<p>Nothing when it comes to the gradient, but their forward pass differs since <code>z_new</code> returns <code>z</code> as its forward output activation.</p>

username: lisyuan
cooked: <p>Thanks for your quickly response. Now I’m understanding how the gradients from <code>z_new</code> and <code>z_new_2</code> come from. Is it better just write something like <code>z_new2 = z + z_backward</code>?</p>
<p>I am also still confusing some points</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="178060">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>z_new</code> uses <code>z_backward</code> to calculate the gradients as its the only attached tensor to the computation graph. The other two tensors were explicitly detached.</p>
</blockquote>
</aside>
<p>I still cannot understand why should do something like <code>(z_backward - z_backward.detach())</code>?<br>
Why not just do <code>z_new = z.detach() + z_backward</code> or just simply <code>z_backward.backward()</code> itself.<br>
Also, is there any advantage or consideration without coding <code>z_backward.backward()</code> itself ? What I guess is that it can scale gradient value without scale the output <code>z_new</code> (loss), maybe it’s better to adust learning rate?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="lisyuan" data-post="3" data-topic="178060">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/6a8cbe/48.png" class="avatar"> lisyuan:</div>
<blockquote>
<p>I still cannot understand why should do something like <code>(z_backward - z_backward.detach())</code>?<br>
Why not just do <code>z_new = z.detach() + z_backward</code> or just simply <code>z_backward.backward()</code> itself.</p>
</blockquote>
</aside>
<p>The advantage of</p>
<pre data-code-wrap="python"><code class="lang-python">z_new = z.detach() + (z_backward - z_backward.detach())
</code></pre>
<p>is that you will get <code>z</code> as the forward output and will use <code>z_backward</code> to calculate the gradients in the backward pass.<br>
In the foward pass <code>z_backward</code> is cancelled due to the subtraction and only <code>z.detach()</code> will define the value. However, during the backward Autograd will only use <code>z_backward</code> since it’s the only attached tensor.</p>

username: lisyuan
cooked: <p>Thanks for your clear explanation!!!  I think I understand it clearly now <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>