How can I get output of intermediate hidden layers in a Neural Net to be passed as input explicitly to the hidden layer in a pretrained model to get the final layer output?

username: Nithin_Upadhya
cooked: <p>Let us assume I have a trained model saved  with 5 hidden layers (fc1,fc2,fc3,fc4,fc5,fc6).</p>
<p>Suppose I need to get output of Fc3 layer from the existing model,<br>
BY defining</p>
<p>def get_activation(name):<br>
def hook(model, input, output):<br>
activation[name] = output.detach()<br>
return hook</p>
<p>hidden_fc3_output = model.fc3.register_forward_hook(get_activation(‘fc3’)) …</p>
<p>How can I use this hidden_fc3_output to pass it explicitly from fc4 layer to get the final fc6 output ?</p>

username: ptrblck
cooked: <p><code>hidden_fc3_output</code> will be the handle to the hook and the activation will be stored in <code>activation['fc3']</code>.<br>
I’m not sure to understand the use case completely, but if you would like to pass this stored activation to <code>fc4</code> and all following layers, you could create a switch in your <code>forward</code> method and pass it to the model. This would split the original forward path into two steps.<br>
You could of course still use the forward hook if needed.<br>
Have a look at this code snippet and let me know, if something would be useful or if I misunderstood your use case:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc1 = nn.Linear(1, 1)
        self.fc2 = nn.Linear(1, 1)
        self.fc3 = nn.Linear(1, 1)
        self.fc4 = nn.Linear(1, 1)
        self.fc5 = nn.Linear(1, 1)
        self.fc6 = nn.Linear(1, 1)
        self.act = nn.ReLU()

    def forward_half1(self, x):
        x = self.act(self.fc1(x))
        x = self.act(self.fc2(x))
        x = self.act(self.fc3(x))
        return x
    
    def forward_half2(self, x):
        x = self.act(self.fc4(x))
        x = self.act(self.fc5(x))
        x = self.fc6(x)
        return x
    
    def forward(self, x, path='all'):
        if path=='all':
            x = self.forward_half1(x)
            x = self.forward_half2(x)
        elif path=='half1':
            x = self.forward_half1(x)
        elif path=='half2':
            x = self.forward_half2(x)
        else:
            raise NotImplementedError
        return x

            
model = MyModel()        
x = torch.randn(1, 1)        
out1 = model(x, path='half1')
out2 = model(out1, path='half2')
out = model(x)
print(out2 == out)

model.fc3.register_forward_hook(get_activation('fc3'))
out1 = model(x, path='half1')
print(out1 == activation['fc3'])
out2 = model(activation['fc3'], path='half2')
print(out2 == out)
</code></pre>

username: Nithin_Upadhya
cooked: <p>Thanks ptrblck for the reply <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I was trying to understand the difference between defining (midoutput) architecture in forward definition  and calling register_forward_hook function.<br>
Why is it that I get a difference in predicting output if I use register_forward_hook.<br>
It seems that  model(x,path) is more accurate.</p>
<p>Example -</p>
<p>Actual Output - [8.1009,7.4015,7.7013]<br>
Predicted Output<br>
By model(model(x,path=‘half1’),path=‘half2’) – [8.1009,7.4015,7.7013]<br>
By register_forward_hook in fc3 layer and then calling model(activation[‘fc3’],path=‘half2’) – [8.3453,7.6397,7.9421]</p>
<p>There is a huge difference in the predictions.</p>
<p>The reason is activation [’‘fc3’’] and model(x,path=‘half1’) are not the same.<br>
Any specific reason for this?</p>
<p>Regards,<br>
Nithin</p>

username: ptrblck
cooked: <p>In my example the hooked activation as well as the output using <code>path='half1'</code> yield the same result.<br>
Are you using any batchnorm layers or other layers with running stats?<br>
If so, note that the running estimates will be changes in each forward pass, so that the next one will yield different results.<br>
For the sake of debugging you could set the model to eval mode (<code>model.eval()</code>) and test it again.</p>