How to automatically disable register_hook when training model is in eval() phase in PyTorch?

username: someAdjectiveNoun
cooked: <p>I require to update grads of an intermediate tensor variable using the  <code>register_hook</code>  method. Since the variable isn’t a leaf-variable, I require to add the  <code>retain_grad()</code>  method to it after which, I can use the  <code>register_hook</code>  method to alter the grads.</p>
<pre><code class="lang-auto">score.retain_grad()
h = score.register_hook(lambda grad: grad * torch.FloatTensor(...))
</code></pre>
<p>This works perfectly fine during the training ( <code>model.train()</code> ) phase. However, it gives an error during the evaluation phase ( <code>model.eval()</code> ).</p>
<p>The error:</p>
<pre><code class="lang-auto">File "/home/envs/darthvader/lib/python3.6/site-packages/torch/tensor.py", line 198, in register_hook
    raise RuntimeError("cannot register a hook on a tensor that "
RuntimeError: cannot register a hook on a tensor that doesn't require gradient
</code></pre>
<p>How could the model automatically disable the  <code>register_hook</code>  method when it in the <code>eval()</code>  phase?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I think the simplest thing to do here is to guard the <code>register_hook()</code> call with <code>if score.requires_grad:</code>.</p>
<p>Also side note: you don’t need to call <code>retain_grad()</code> for the hook to work. Only if you want to be able to read the value in the <code>.grad</code> field on the Tensor.</p>

username: someAdjectiveNoun
cooked: <p>Hi <a class="mention" href="/u/alband">@albanD</a>,</p>
<p>Thanks for the answer ^^. You are correct, the <code>register_hook</code> works without retain_grad(). The code works perfectly fine during the <code>eval()</code> phase now.</p>
<p>I was confused about using <code>if score.requires_grad:</code> on the <code>score</code> variable as it is not a leaf-node variable, but an intermediate-node variable.</p>

username: albanD
cooked: <aside class="quote no-group" data-username="someAdjectiveNoun" data-post="3" data-topic="93941">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/someadjectivenoun/48/17452_2.png" class="avatar"> someAdjectiveNoun:</div>
<blockquote>
<p>I was confused about using <code>if score.requires_grad:</code> on the <code>score</code> variable as it is not a leaf-node variable, but an intermediate-node variable.</p>
</blockquote>
</aside>
<p>The intermediate Tensor will have have their requires_grad=True in python. So you can use that as a hint if they are used in the computation of something that needs gradient computed.</p>

username: someAdjectiveNoun
cooked: <aside class="quote no-group" data-username="albanD" data-post="4" data-topic="93941">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>The intermediate Tensor will have have their requires_grad=True in python. So you can use that as a hint if they are used in the computation of something that needs gradient computed.</p>
</blockquote>
</aside>
<p>Yes, exactly. Somehow, it completely slipped my mind even when I checked the <code>grad_fn</code> attribute of the <code>score</code> variable in question.</p>