How to remove multiple hooks?

username: Brando_Miranda
cooked: <p>is this correct for removing multiple hooks?</p>
<pre><code class="lang-auto">def remove_hook(mdl: nn.Module, hook):
    """
    ref: https://github.com/pytorch/pytorch/issues/5037
    """
    handle = mdl.register_forward_hook(hook)
    handle.remove()


def remove_hooks(mdl: nn.Module, hooks: list):
    """
    ref: https://github.com/pytorch/pytorch/issues/5037
    """
    for hook in hooks:
        remove_hook(mdl, hook)
</code></pre>
<p>my code example:</p>
<pre><code class="lang-auto">        &gt;&gt;&gt; hooks1 = SimilarityHook.create_hooks(model1, ['name1', ...])
        &gt;&gt;&gt; hooks2 = SimilarityHook.create_hooks(model2, ['name1', ...])
        &gt;&gt;&gt; with torch.no_grad():
        &gt;&gt;&gt;    model1(input)
        &gt;&gt;&gt;    model2(input)
        &gt;&gt;&gt; [[hook1.distance(hook2) for hook2 in hooks2] for hook1 in hooks1]
</code></pre>

username: tom
cooked: <pre><code class="lang-auto">    handle = mdl.register_forward_hook(hook)
    handle.remove()
</code></pre>
<p>No, you need to keep the handles from the registration and then just call handle.remove.</p>

username: Brando_Miranda
cooked: <p>apologies, not sure what you mean. What is wrong with the code I proposed?</p>
<p>Thanks for your time in advance.</p>

username: tom
cooked: <p>The order of things is</p>
<ol>
<li>register hook, keep handle,</li>
<li>make use of the hook,</li>
<li>remove hook.</li>
</ol>
<p>You code has 1 and 3 without any 2 in between. This is wrong, either because you forgot 2 or because you think you are coding only 3 but in reality are doing 1+3 and not undoing the effect of step 1 before.</p>
<p>You could print <code>m._forward_hooks</code> at various times in your code to get a feeling of what your code does w.r.t. forward hooks. (Note that <code>m._forward_hooks</code> is an internal datastructure, so no guarantees for that staying viable, but for now it seems to be OK for educational purposes.)</p>
<p>Best regards</p>
<p>Thomas</p>

username: Brando_Miranda
cooked: <pre><code class="lang-auto">The order of things is

register hook, keep handle,
make use of the hook,
remove hook.
You code has 1 and 3 without any 2 in between. This is wrong, either because you forgot 2 or because you think you are coding only 3 but in reality are doing 1+3 and not undoing the effect of step 1 before.

You could print m._forward_hooks at various times in your code to get a feeling of what your code does w.r.t. forward hooks. (Note that m._forward_hooks is an internal datastructure, so no guarantees for that staying viable, but for now it seems to be OK for educational purposes.)

Best regards

Thomas
</code></pre>
<p>Hi Tom,</p>
<p>Thanks for your kind reply.</p>
<p>I think I am using the hook:</p>
<pre><code class="lang-auto">hooks1 = SimilarityHook.create_hooks(model1, ['name1', ...])
hooks2 = SimilarityHook.create_hooks(model2, ['name1', ...])
with torch.no_grad():
    model1(input)
    model2(input)
[[hook1.distance(hook2) for hook2 in hooks2] for hook1 in hooks1]
</code></pre>
<p>when I run the forward pass the hook registers the tensors and then later uses those to compute “distances between models”.</p>
<p>Perhaps a full example will make more sense (with your numbering):</p>
<pre><code class="lang-auto"># - register hooks
hooks1 = SimilarityHook.create_hooks(model1, ['name1', ...])
hooks2 = SimilarityHook.create_hooks(model2, ['name1', ...])

# - use hooks
with torch.no_grad():
    model1(input)
    model2(input)
dists: list[float] = [[hook1.distance(hook2) for hook2 in hooks2] for hook1 in hooks1]
print(dists)
[[hook1.clear(), hook2.clear() for hook2 in hooks2] for hook1 in hooks1]  # clear stored tensors

# - unregister  *all* hooks
remove_hooks(mdl1, hooks1)
remove_hooks(mdl2, hooks2)
</code></pre>
<p>I think that is a complete example.</p>
<p>Is this correct now? I suppose I am most interested that step 3 works.</p>

username: tom
cooked: <p>What does the printing of the hooks say?</p>

username: Tejan_Mehndiratta
cooked: <p>Hi, Tom</p>
<p>Could you please help me with this?<br>
<a href="https://discuss.pytorch.org/t/pytorch-hooks-removal/">https://discuss.pytorch.org/t/pytorch-hooks-removal/</a></p>