Extract features from layer of submodule of a model

kl_divergence: <p>My model looks like this:</p>
<pre><code class="lang-auto">from __future__ import absolute_import

import torch
from torch import nn
from torch.nn import functional as F
import torchvision
    
class hybrid_cnn(nn.Module):
    def __init__(self,**kwargs):
        super(hybrid_cnn,self).__init__()
        resnet = torchvision.models.resnet50(pretrained=True)
        self.base = nn.Sequential(*list(resnet.children())[:-2])

    def forward(self,x)
        x = self.base(x)
        y = self.conv2(self.conv1(x))
        clf_outputs = {}
        num_fcs = 2
        

        return clf_outputs,x,y
</code></pre>
<p>I want a Tensor which would have the output coming from ResNet-&gt;conv1-&gt;conv2 (classifier layer removed) from both augmented 1 and augmented 2.<br>
I have tried with <code>register_forward_hook</code>, but removable_hook object is not callable,<br>
. Is there any efficient way to extract features from submodule (output from conv2 layer from both augmented1 and augmented2) ?</p>

ptrblck: <p>The <code>forward_hook</code> should work:</p>
<pre><code class="lang-python">activation = {}
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook

model.fc0.conv2.register_forward_hook(get_activation('fc0.conv2'))
model.fc1.conv2.register_forward_hook(get_activation('fc1.conv2'))

output = model(x)
print(activation['fc0.conv2'])
print(activation['fc0.conv1'])
</code></pre>
<p>I’m not sure, what <code>y = self.conv2(self.conv1(x))</code> in <code>hybrid_cnn</code> is, so I just removed it.</p>

kl_divergence: <p>Thanks for answering. I am not sure what’s happening here, I followed your steps:</p>
<pre><code class="lang-auto">model = hybrid_convnet2.hybrid_cnn().to(device)
t = torch.randn([64,3,256,128]).cuda()
</code></pre>
<p>I am able to get the output of model(t) at this stage, but after adding forward hooks</p>
<pre><code class="lang-auto">model.fc0.conv2.register_forward_hook(get_activation('fc0.conv2'))
model.fc1.conv2.register_forward_hook(get_activation('fc1.conv2'))
</code></pre>
<p>And then calling model(t), it gives <code>NoneType object is not callable</code> error.</p>

ptrblck: <p>Do you get the same error without registering the hooks?</p>

kl_divergence: <p>No, it prints out the correct output before registering hooks</p>

ptrblck: <p>What does <code>print(model.fc0.conv2)</code> return?<br>
After registering the hooks the error is thrown in the forward pass?</p>

kl_divergence: <pre><code class="lang-auto">Sequential(
  (0): Conv2d(1024, 2048, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  (1): BatchNorm2d(2048, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True)
  (2): LeakyReLU(negative_slope=0.1, inplace)
)
</code></pre>
<p>I was doing it your way.<br>
Do I have to add hooks in forward pass of hybrid_cnn ?</p>

ptrblck: <p>The hooks should be already registered with <code>hybrid_cnn</code>'s augmented modules.<br>
Could you post the complete error message?</p>

kl_divergence: <p>Here it is:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/13ea8a7f92ddc7b484d731da48679c97f37d3436.png" data-download-href="https://discuss.pytorch.org/uploads/default/13ea8a7f92ddc7b484d731da48679c97f37d3436" title="w1"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/1/13ea8a7f92ddc7b484d731da48679c97f37d3436_2_690x329.png" alt="w1" data-base62-sha1="2QbzzqErFoODdgpOksJXKJY1uDQ" width="690" height="329" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/1/13ea8a7f92ddc7b484d731da48679c97f37d3436_2_690x329.png, https://discuss.pytorch.org/uploads/default/optimized/2X/1/13ea8a7f92ddc7b484d731da48679c97f37d3436_2_1035x493.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/1/13ea8a7f92ddc7b484d731da48679c97f37d3436.png 2x" data-dominant-color="F2F3F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">w1</span><span class="informations">1129×539 50.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/e125504eee0e11febe913258a5e1858d1cbd3c2e.png" data-download-href="https://discuss.pytorch.org/uploads/default/e125504eee0e11febe913258a5e1858d1cbd3c2e" title="w2"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/e/e125504eee0e11febe913258a5e1858d1cbd3c2e_2_690x387.png" alt="w2" data-base62-sha1="w7JnfRooCPMerITuxhOM93HVKMe" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/e/e125504eee0e11febe913258a5e1858d1cbd3c2e_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/2X/e/e125504eee0e11febe913258a5e1858d1cbd3c2e_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/e/e125504eee0e11febe913258a5e1858d1cbd3c2e.png 2x" data-dominant-color="F5F8F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">w2</span><span class="informations">1094×615 79.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/3016bc1107f9c9b61e11d93e4ce157cd99c54291.png" data-download-href="https://discuss.pytorch.org/uploads/default/3016bc1107f9c9b61e11d93e4ce157cd99c54291" title="w3"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3016bc1107f9c9b61e11d93e4ce157cd99c54291_2_690x374.png" alt="w3" data-base62-sha1="6RpCTo2PCRj94wOkBvDnHHsjWYp" width="690" height="374" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3016bc1107f9c9b61e11d93e4ce157cd99c54291_2_690x374.png, https://discuss.pytorch.org/uploads/default/optimized/2X/3/3016bc1107f9c9b61e11d93e4ce157cd99c54291_2_1035x561.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/3/3016bc1107f9c9b61e11d93e4ce157cd99c54291.png 2x" data-dominant-color="F6F9F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">w3</span><span class="informations">1135×616 80 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

kl_divergence: <p>Before registering hooks:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/0f774c1123ce9ceda0bd691d109a67cc3f8a9217.png" data-download-href="https://discuss.pytorch.org/uploads/default/0f774c1123ce9ceda0bd691d109a67cc3f8a9217" title="w4"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0f774c1123ce9ceda0bd691d109a67cc3f8a9217_2_690x345.png" alt="w4" data-base62-sha1="2cOKObXUngZqKP0osUMncj2sOZF" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0f774c1123ce9ceda0bd691d109a67cc3f8a9217_2_690x345.png, https://discuss.pytorch.org/uploads/default/optimized/2X/0/0f774c1123ce9ceda0bd691d109a67cc3f8a9217_2_1035x517.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/0/0f774c1123ce9ceda0bd691d109a67cc3f8a9217.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">w4</span><span class="informations">1140×570 55.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I’m not sure, what’s going on.<br>
Could you post your code as a gist so that I can run exactly the same code?</p>

kl_divergence: <p>Here it is:<br>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/prajjwal1/d88b8cd63b9177fe1e544b036455eac9" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/prajjwal1/d88b8cd63b9177fe1e544b036455eac9" target="_blank" rel="nofollow noopener">https://gist.github.com/prajjwal1/d88b8cd63b9177fe1e544b036455eac9</a></h4>
<h5>hook.py</h5>
<pre><code class="Python">from __future__ import absolute_import

import torch
from torch import nn
from torch.nn import functional as F
import torchvision

def conv_layer(ni,nf,kernel_size=3,stride=1):
    return nn.Sequential(
            nn.Conv2d(ni,nf,kernel_size=kernel_size,bias=False,stride=stride,padding=kernel_size//2),</code></pre>
This file has been truncated. <a href="https://gist.github.com/prajjwal1/d88b8cd63b9177fe1e544b036455eac9" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

ptrblck: <p>Thanks for the code.<br>
Your return statement is at the wrong place.<br>
<code>get_activation</code> should return <code>hook</code>, not <code>hook</code> itself:<br>
Right way:</p>
<pre><code class="lang-python">def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook
</code></pre>
<p>Your current implementation:</p>
<pre><code class="lang-python">def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
        return hook
</code></pre>

kl_divergence: <p>Oh my bad, didn’t realize it.</p>

kl_divergence: <p>Dictionary remains empty after calling get_activation()</p>

ptrblck: <p>Your code I’ve fixed works for me.</p>
<pre><code class="lang-python">output = model(t)
print(activation['fc0.conv2'].shape)
&gt; torch.Size([1, 2048, 8, 4])
print(activation['fc1.conv2'].shape)
&gt; torch.Size([1, 2048, 8, 4])
</code></pre>
<p>Could you check it again?</p>

kl_divergence: <p>I had one another doubt. I wanted to add forbenius norm of covariances with my cross entropy.<br>
i have tried this:</p>
<pre><code class="lang-auto">class f_norm(nn.Module):
    def __init__(self,epsilon,beta):
        super(f_norm,self).__init__()
        self.epsilon = epsilon
        self.beta = beta
    def forward(self, cs,ct):
        epsilon=0.1
        beta = 0.2
        x = self.epsilon*( torch.sqrt((torch.abs(cs-ct))**2)-self.beta)
        y = x.view(-1)
        y = torch.sum(y)
        return y/1000
</code></pre>
<p>This doesn’t seem to work, it doesn’t return a Tensor.<br>
I am using this function instead:</p>
<pre><code class="lang-auto">def f_norm(cs,ct):
    epsilon=0.1
    beta = 0.2
    x = epsilon*( torch.sqrt((torch.abs(cs-ct))**2)-beta)
    y = x.view(-1)
    y = torch.sum(y)
    return y/1000
</code></pre>
<pre><code class="lang-auto">loss = cross_entropy(inputs,targets)+f_norm(cs,ct)
</code></pre>
<p>If I call loss.backward(), would this work fine ? I read in forums that unless it’s autograd variable it would not be updated. So was having concerns with this implementation.</p>

ptrblck: <p>As long as you stick with PyTorch functions, you are fine.<br>
What’s wrong with the first implementation?</p>
<p>If the backward pass doesn’t work, PyTorch will throw an error.<br>
So if you don’t get an error, everything should be fine.</p>

kl_divergence: <p>Okay, got it. The first implementation (f_norm class) doesn’t return a Tensor.</p>

ptrblck: <p>I’m not sure, why your code outputs strange things, but your code is returning a <code>tensor</code>. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"><br>
I’ve just removed the unnecessary values (<code>epsilon</code> and <code>beta</code> are defined twice) and it’s working.</p>