Deriving part of the path with a smaller magnitude - gradients question

Ilya_Kotlov: <p>Hi,<br>
So first of a simple pre-question to the main:</p>
<p><code>In train mode (with a model with dropouts, BN and e.t.c) is it true that if y = model(x), then model((x,x))=(y,y') and y=y' with all other identity matters (like in-variance to dropouts, backward path and e.t.c)</code></p>
<p><strong>So to the main question:</strong><br>
I have the following scheme of forward and the light-blue and blue forward paths which are part of one forward path of my GAIN model (see Guided-Inferrence-Attention-Network for more details, which are not necessary for my question):</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef82767dc1d70942371460904ce0cdaf68f9e9fd.png" data-download-href="https://discuss.pytorch.org/uploads/default/ef82767dc1d70942371460904ce0cdaf68f9e9fd" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/ef82767dc1d70942371460904ce0cdaf68f9e9fd_2_510x500.png" alt="image" data-base62-sha1="yaND6RBwTFKj2TWOPE8brLku5C5" width="510" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/ef82767dc1d70942371460904ce0cdaf68f9e9fd_2_510x500.png, https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef82767dc1d70942371460904ce0cdaf68f9e9fd.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef82767dc1d70942371460904ce0cdaf68f9e9fd.png 2x" data-dominant-color="F3F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">672×658 84.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Pay attention that the second input image is derivable through the first blue path.</p>
<p>Now I want that the magnitude of the gradients of the light-blue path will be 1/10 of its original and the regular blue will be as usual 1.</p>
<p>So I’ll make the following:</p>
<blockquote>
<p>(y,y’) = model.forward((x,x))   # y=y’ from the above assumption on the pre-question<br>
y,y’ = y,y’.detach()<br>
light_blue_loss = loss_fn(y’, _) # _ wildcard for something, irrelevant<br>
light_blue_loss.backward(gradient=-0.9)<br>
total_loss = loss_fn(y, _) # _ wildcard for something, irrelevant<br>
total_loss.backward() # total_loss.backward(gradient=1) default</p>
</blockquote>
<p>Now, is this true that for the light-blue path on the detached input according to the scheme on top, the gradients were accumulated with a constant multiplying them by -0.9, thus after accumulating gradients with the total_loss backward with default gradient parameter 1, it can be said the light-blue path was influenced with gradients of magnitude 0.1 (1+(-0.9)) and the whole other path with magnitude 1 of the gradients ?</p>
<p>I’ll be grateful for your answer, thank you.</p>

soulitzer: <p>A better way to accomplish this might be to use hooks:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

def grads(mod):
    return list(x.grad for x in mod.parameters())

torch.manual_seed(0)

x = torch.tensor([1., 2.])
mod = nn.Linear(2, 2)
mod2 = nn.Linear(2, 2)

y = mod2(mod(x))
y.sum().backward()

print(grads(mod))
print(grads(mod2))

print("--------------")

torch.manual_seed(0)

x = torch.tensor([1., 2.])
mod = nn.Linear(2, 2)
mod2 = nn.Linear(2, 2)

for p in mod2.parameters():
    p.register_hook(lambda x: x/10)

y = mod2(mod(x))
y.sum().backward()

print(grads(mod))
print(grads(mod2))
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">[tensor([[-0.0768, -0.1535],
        [ 0.7478,  1.4955]]), tensor([-0.0768,  0.7478])]
[tensor([[ 0.4810, -1.4331],
        [ 0.4810, -1.4331]]), tensor([1., 1.])]
--------------
[tensor([[-0.0768, -0.1535],
        [ 0.7478,  1.4955]]), tensor([-0.0768,  0.7478])]
[tensor([[ 0.0481, -0.1433],
        [ 0.0481, -0.1433]]), tensor([0.1000, 0.1000])]
</code></pre>
<blockquote>
<p><code>In train mode (with a model with dropouts, BN and e.t.c) is it  true that if y = model(x), then model((x,x))=(y,y') and y=y' with all  other identity matters (like in-variance to dropouts, backward path and  e.t.c)</code></p>
</blockquote>
<p>I guess it would be the same, but a better way of doing that would just to be to clone the output y and then detach?</p>

Ilya_Kotlov: <p><a class="mention" href="/u/soulitzer">@soulitzer</a>, that’s an interesting point, about the hooks.<br>
What would you do if mod and mod2 have to be the same network with exactly the same parameters (as it is in my case) ? Could you think about easy way to handle it in this case?<br>
As much as I can understand I need some variable to differ the point of using the hooks and not using them on the same parameters, right ?<br>
like to support a flag of some specific layer, suppose I know that the last layer name is “last_layer”(and during the backward path I should see it only twice in total), so my hook should look like:</p>
<pre><code class="lang-auto">use_hook = 0
def backward_hook(x):
   if cur_layer_name == 'last_layer':
       use_hook+=1
    if use_hook &lt; 2:
        return x/10
    else:
       return x
</code></pre>
<p>And that is possible if I can indeed to check through the backward path using a hook the current layer name (<em>can I?</em>) ?</p>
<aside class="quote no-group" data-username="soulitzer" data-post="2" data-topic="123432">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/839c29/48.png" class="avatar"> soulitzer:</div>
<blockquote>
<p>I guess it would be the same, but a better way of doing that would just to be to clone the output y and then detach?</p>
</blockquote>
</aside>
<p>I need to think about that.</p>
<p>And another thing, correct me if I wrong, if you had two nn.Linear layers, you still have to do the hook only once because of the chain rule (and the multiplications of all the gradients according to it, correct ?)?</p>
<p>Or is should be done for all the differential layers (wouldn’t it be 10^{<span class="hashtag-raw">#layers_num</span>} factor in that case) ?</p>

soulitzer: <blockquote>
<p>And another thing, correct me if I wrong, if you had two nn.Linear layers, you still have to do the hook only once because of the chain rule (and the multiplications of all the gradients according to it, correct ?)?</p>
</blockquote>
<p>Hmm I probably misunderstood your original question. In my example I used two modules in sequence to demonstrate that it is possible to use hooks to selectively alter the gradients of a subgraph of the original network, and in this particular case, so that the first module’s parameters remain the original gradient, but the second module’s gradients are a tenth of what they would’ve been.</p>
<p>We have to register a hook individually for each tensor (aka parameter) in module 2 because if we register the hook directly on the tensor, it triggers when grad is computed for that tensor, and alters only the gradient computation for that individual tensor.</p>
<p>I actually don’t quite understand what you are going for yet, but if you wanted to alter the gradients of both nn.Linear layers you’d need to individually register hooks for four tensors. Or do “for each mod.parameters()” twice.</p>
<blockquote>
<p>And that is possible if I can indeed to check through the backward path using a hook the current layer name</p>
</blockquote>
<p>Not for hooks registered directly on a tensor. But you do still have access when you loop through the parameters of the module itself.</p>

Ilya_Kotlov: <aside class="quote no-group" data-username="soulitzer" data-post="4" data-topic="123432">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/839c29/48.png" class="avatar"> soulitzer:</div>
<blockquote>
<p>I actually don’t quite understand what you are going for yet, but if you wanted to alter the gradients of both nn.Linear layers you’d need to individually register hooks for four tensors. Or do “for each mod.parameters()” twice.</p>
</blockquote>
</aside>
<p>I’ll make it a little bit clearer about that. I see what you’ve done, and it’s ok if you have two modules, but in my case, as in the image, it is the same module, thus the same parameters, so if I register a hook on them, it will be called actually twice in the backward path of the one called ‘am_loss’, and what I seek to achieve is a gradient divided by 10 only on the first time of two where the backward of the parameter is called.</p>
<p>So if I have only one module and I’ll register a backward hook:</p>
<pre><code class="lang-auto">mod = nn.Linear(2, 2)

for p in mod2.parameters():
    p.register_hook(lambda x: x/10)

y_hat = mod(x)
y = mod(y_hat)
y.sum().backward()
</code></pre>
<p>in that case unwillingly the grads will be twice computed with division by 10, and I wanted only for y gradients will be divided by 10 and going back through y_hat the gradients will remain as is.</p>
<p>So I want to do a manipulation of the same parameters gradients, but they exists twice in the backward graph (if you look with torchviz on the graph of y you will see view functions that create a another second view of the parameter, but that is the same parameter, so if I register a hook, it will be called twice, and I want only once).</p>
<p>That’s why I thought to create a variable as a counter of amount of times I visited that parameter (I know it is 2 at most, so I will increase it every time the hook called, so if it is 1, then I’ll devide the grads by 10 and if it is 2 I will not)</p>
<aside class="quote no-group" data-username="soulitzer" data-post="4" data-topic="123432">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/839c29/48.png" class="avatar"> soulitzer:</div>
<blockquote>
<p>We have to register a hook individually for each tensor (aka parameter) in module 2 because if we register the hook directly on the tensor, it triggers when grad is computed for that tensor, and alters only the gradient computation for that individual tensor.</p>
</blockquote>
</aside>
<p>Ok, got you on that. I asked this because I know the chain rule is like Dy/Dz * Dz/Dw …<br>
So I thought although it is individual for each tensor-parameter now we have Dy/(Dz*10)*Dz/(Dw*10) so it will be a mistake to do it for each tensor-parameter because of the math, but you’re saying that not the case and in case of a network of many layers, the correct logic is to register ‘devide_10_hook’ for each layer-parameters, is that correct ?<br>
And the effect in this case will be:<br>
1/10 * (Dy/Dz * Dz/Dw …)</p>

soulitzer: <blockquote>
<p>Ok, got you on that. I asked this because I know the chain rule is like Dy/Dz * Dz/Dw …</p>
</blockquote>
<p>Registering a hook to the each of the parameters of a module would not affect future gradient computation, because parameters are leaf tensors. If you registered a hook to a non-leaf tensor (e.g., the output of your module), yes it would do what you describe, and you would only need to register a single hook in that case.</p>
<p>So I guess you could just do the below actually:</p>
<pre><code class="lang-auto">mod = nn.Linear(2, 2)
y_hat = mod(x)
y = mod(y_hat)
y.register_hook(lambda x: x/10)
y.sum().backward()
</code></pre>
<p>In this case, you can avoid registering hooks directly on the parameters as well so it should be okay to use the same mod twice. But the issue is that it would also affect gradient computation of the second backward as well. So we can actually register another hook to multiply by 10 again.</p>
<pre><code class="lang-auto">mod = nn.Linear(2, 2)
y_hat = mod(x)
y_hat.register_hook(lambda x: x*10)
y = mod(y_hat)
y.register_hook(lambda x: x/10)
y.sum().backward()
</code></pre>
<p>Does this resolve your issue?</p>

Ilya_Kotlov: <aside class="quote no-group" data-username="soulitzer" data-post="6" data-topic="123432">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/839c29/48.png" class="avatar"> soulitzer:</div>
<blockquote>
<p>Does this resolve your issue?</p>
</blockquote>
</aside>
<p>That idea also came to my mind, about the multiplication by 10. I think it will resolve it.</p>
<p>As an alternative, suppose <code>mod</code> is a model with environment where I can save a counter, which role is to determine the number of the visit through the backward through this one layer.<br>
So using this counter as an indicator with an if statement what to do when the hook called:<br>
If it is the first visit (there is a deterministic definite order of the backwards because of the forwards sequence) I need to divide it by 10, if it is the second visit, I should do nothing (just zero the the counter for the next forward and backward iteration), did you get my suggestion and its logic by that way?<br>
That’s how I can avoid the multiplication on the second output, although in that solution my hook will be on the parameters of the module and not on the output (where the problem is the distinction between the two visits through backward function which I think can be resolved by a simple counter).</p>
<p>But this solution with division and then a correction by multiplication should do the job I think, that’s good also and maybe even more simple.</p>

soulitzer: <p>It is good to consider the alternatives indeed - the counter method might work also:<br>
One way to do it is actually save the counter as the functions default argument! This avoids maintaining an extra global set of counters for each parameter.</p>
<pre><code class="lang-auto">fns = []

for i in range(10):
    def fn(x = [0]):
        x[0] += 1
        print(x[0])
    fns.append(fn)

fns[0]() # 1
fns[1]() # 1
fns[2]() # 1

fns[0]() # 2
</code></pre>

Ilya_Kotlov: <p><a class="mention" href="/u/soulitzer">@soulitzer</a>, hmm … never wrote such code with a counter as a default argument, can it be zeroed ?<br>
like some code which will do:</p>
<pre><code class="lang-auto">fns = []

for i in range(10):
    def fn(x = [0]):
        x[0] += 1
        print(x[0])
    fns.append(fn)

fns[0]() # 1
fns[1]() # 1
fns[2]() # 1

fns[0]() # 2

'some_zero_code()'
fns[0]() # 0
</code></pre>

soulitzer: <p>Uhhh, I guess you could just do the counter mod 2, i.e. only scale on every other invocation haha. But this is all hypothetical anyway - I wouldn’t recommend anyone to write such code.</p>

Ilya_Kotlov: <p><a class="mention" href="/u/alband">@albanD</a>, Hi, will be glad to hear your opinion about the correctness of this solution, because I find it the easiest to experiment with.</p>
<p>Think it should work ?</p>
<aside class="quote no-group" data-username="soulitzer" data-post="6" data-topic="123432">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/839c29/48.png" class="avatar"> soulitzer:</div>
<blockquote>
<pre><code class="lang-auto">mod = nn.Linear(2, 2)
y_hat = mod(x)
y_hat.register_hook(lambda x: x*10)
y = mod(y_hat)
y.register_hook(lambda x: x/10)
y.sum().backward()
</code></pre>
<p>Does this resolve your issue?</p>
</blockquote>
</aside>
<p>In my case, I’ll register a hook on the am_loss which will divide by 10 and another hook on the input of the second path, which is the masked_image, which will multiply the grads back by 10, so I’ll get the light-blue path gradients divided by 10 and the blue as regular, will I ?</p>
<p>Another solution as we discussed is to register a hook for each parameter in the module, which is a little bit painful (and to maintain counters or as <a class="mention" href="/u/soulitzer">@soulitzer</a> suggested with internal counters trick and e.t.c which is still isn’t nice). But the idea using many hook for modules which are in nn.Sequential isn’t likable, especially to to the fact that right now the backward_hook for nn.Sequential is broken in terms of it only works on the gradients of the last module in it and not on every one, so it means I need to loop over each module handly and register a hook which will divide by 10 the gradients.</p>

albanD: <p>That solution would work yes.</p>
<p>For Sequential hooks, you can now use the <code>full_backward_hook</code> that actually work <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>

Ilya_Kotlov: <p>good news. Didn’t find the docs about it, do you mind to share a link <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> ?</p>

albanD: <p>1.9 was released 2 minutes ago: <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=full_backward#torch.nn.Module.register_full_backward_hook" class="inline-onebox">Module — PyTorch 1.9.0 documentation</a> haha</p>