Proper way to deal with Intermediate activations

username: two_Two
cooked: <p>Hi,It’s pretty common to see extract and process Intermediate activations  for some extending operations.<br>
However,due to leak of knowledge of how computation graph works in Pytorch, I have couple questions to ask and separate in two cases  :</p>
<p>Case 1: define function inside of class</p>
<pre><code class="lang-auto">class MODEL(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = Conv2d(3,32,kernel_size = 3,padding = 1)
        self.bn1 = BatchNorm2d(32)
        self.pooling1 = MaxPool2d(2,2)
        self.conv2 = Conv2d(32,64,kernel_size = 3,padding = 1)
        self.bn2 = BatchNorm2d(64)
        self.pooling2 = MaxPool2d(2,2)
        self.extra_conv1 = Conv2d(32,32,kernel_size=3,padding=1)
    
    def feature_extractor(self,x):
        out = self.extra_conv1(x)
        return out
    
    def forward(self,x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = F.relu(x)
        x = self.feature_extractor(x)
        x = self.pooling1(x)
        x = self.conv2(x)
        x = self.bn2(x)
        x = F.relu(x)
        x = self.pooling2(x)
        return x
  
model = MODEL()
summary(model,(3,64,64))
</code></pre>
<p>The code above is a simple example of extracting feature,In this case I have <code>Total params: 28,832</code>.</p>
<p>Case 2 : define function outside of class</p>
<pre><code class="lang-auto">def feature_extractor(x):
    in_channels= x.size(1)
    extractor =Conv2d(in_channels,32,kernel_size = 3,padding = 1)
    out = extractor(x)
    return out

class MODEL(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = Conv2d(3,32,kernel_size = 3,padding = 1)
        self.bn1 = BatchNorm2d(32)
        self.pooling1 = MaxPool2d(2,2)
        self.conv2 = Conv2d(32,64,kernel_size = 3,padding = 1)
        self.bn2 = BatchNorm2d(64)
        self.pooling2 = MaxPool2d(2,2)
        self.extra_conv1 =feature_extractor
    
    def forward(self,x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = F.relu(x)
        x = self.extra_conv1(x)
        x = self.pooling1(x)
        x = self.conv2(x)
        x = self.bn2(x)
        x = F.relu(x)
        x = self.pooling2(x)
        return x

model = MODEL()
summary(model,(3,64,64))
</code></pre>
<p>In this I define <code>feature_extractor</code> outside of class And I only get <code>Total params: 19,584</code>, It seems that the <code>feature_extractor</code> is not part of graph…</p>
<p>I know I could’ve done it in a better way(do it in <code>forward</code> function),but just for curiosity,should I define the function inside of class,why or why not? Thanks in advance!</p>

username: ptrblck
cooked: <p>The second approach will reinitialize the <code>nn.Conv2d</code> module in each forward pass, which is probably not the desired behavior.</p>
<p>If you define <code>extractor</code> once and just use it inside <code>feature_extractor</code>, you’ll get the same behavior as in your first example, as long as you pass the parameters of <code>extractor</code> to the optimizer.</p>

username: two_Two
cooked: <blockquote>
<p>If you define  <code>extractor</code>  once and just use it inside  <code>feature_extractor</code></p>
</blockquote>
<p>I don’t quite understand this,Can you give a example show me how to do it?</p>

username: ptrblck
cooked: <p>Basically like this:</p>
<pre><code class="lang-python">def feature_extractor(x, extractor):
    out = extractor(x)
    return out

class MODEL(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3,32,kernel_size = 3,padding = 1)
        self.bn1 = nn.BatchNorm2d(32)
        self.pooling1 = nn.MaxPool2d(2,2)
        self.conv2 = nn.Conv2d(32,64,kernel_size = 3,padding = 1)
        self.bn2 = nn.BatchNorm2d(64)
        self.pooling2 = nn.MaxPool2d(2,2)
        self.extra_conv1 =feature_extractor
    
    def forward(self,x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = F.relu(x)
        x = self.extra_conv1(x)
        x = self.pooling1(x)
        x = self.conv2(x)
        x = self.bn2(x)
        x = F.relu(x)
        x = self.pooling2(x)
        return x

extractor = nn.Conv2d(32, 32, kernel_size = 3,padding = 1)
model = MODEL()
</code></pre>

username: two_Two
cooked: <p>yeah,that works with adding<br>
<code>self.extractor = extractor</code><br>
and in <code>foward</code> function<br>
<code>x = self.extra_conv1(x,self.extractor)</code> ,<br>
and the <code>summary</code> would be exactly like first case.Thanks!</p>
<p>And by the way,some people would initialize parameters inside of <code>__init__</code> like:</p>
<pre><code class="lang-auto">def __init__(self):
    super().__init__()
    self.conv1 = Conv2d(3,32,kernel_size = 3,padding = 1)
    self.bn1 = BatchNorm2d(32)
    self.pooling1 = MaxPool2d(2,2)
    self.conv2 = Conv2d(32,64,kernel_size = 3,padding = 1)
    self.bn2 = BatchNorm2d(64)
    self.pooling2 = MaxPool2d(2,2)
    self.extra_conv1 = Conv2d(32,32,kernel_size=3,padding=1)
    for module in self.modules():
        if isinstance(module,nn.Conv2d):
            init.ones_(module.weight)
</code></pre>
<p>And  I add <code>print("forward")</code> in <code>foward</code> function to check if <code>foward</code> would be called while initializing , However When I build the model and write a code test :</p>
<pre><code class="lang-auto">model = MODEL()
for module in model.modules():
    if isinstance(module,nn.Conv2d):
        print(module.weight)
</code></pre>
<p>And the weight has already been initialized and there’s no print shows.<br>
So to my understanding ,does this mean all weights  could be set before the whole model is created? Thanks!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="two_Two" data-post="5" data-topic="64267">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/two_two/48/11554_2.png" class="avatar"> two_Two:</div>
<blockquote>
<p>,does this mean all weights could be set before the whole model is created?</p>
</blockquote>
</aside>
<p>Once you create a layer, you can directly initialize its parameters, as they will be created immediately.<br>
Let me know, if I misunderstood your question.</p>

username: two_Two
cooked: <p>OK,that answers my question.Thanks!</p>