Add Residual connection

kl_divergence: <p>My models looks like this:</p>
<pre><code class="lang-auto">class hybrid_cnn(nn.Module):
    def __init__(self,**kwargs):
        super(hybrid_cnn,self).__init__()
        resnet = torchvision.models.resnet50(pretrained=True)
        self.base = nn.Sequential(*list(resnet.children())[:-2])
    def forward(self,x):
        x = self.base(x)
        clf_outputs = {}
    

</code></pre>
<p>I want to add residual connection from Resnet to conv2 in both blocks. I am not able to find implementations online plus other questions regarding ResNets have not been answered.</p>

ptrblck: <p>You could have a look at the implementation in <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L25">torchvision.models</a>.<br>
There you will find the <code>BasicBlock</code> and <code>Bottleneck</code> layers which apply the residual connection.</p>

kl_divergence: <p>I Checked it, but still I have doubts in implementation, could you please provide a small snippet?</p>

ptrblck: <p>Sure, here is a simple one:</p>
<pre><code class="lang-python">class block1(nn.Module):
    def __init__(self,ni):
        super(block1, self).__init__()
        self.conv1 = nn.Conv2d(ni, ni, 1)
        self.conv2 = nn.Conv2d(ni, ni, 3, 1, 1)
        self.classifier = nn.Linear(ni*24*24,751)

    def forward(self,x):
        residual = x
        out = F.relu(self.conv1(x))
        out = F.relu(self.conv2(out))
        
        out += residual
        
        out = out.view(out.size(0),-1)
        return self.classifier(out)

block = block1(16)
x = torch.randn(1, 16, 24, 24)
output = block(x)
</code></pre>
<p>Depending on your number of in and out channels, you might need to add a <code>downsample</code> layer as in the examples.</p>

Ditlev_Jorgensen: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="20148">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>def forward(self,x): residual = x out = F.relu(self.conv1(x)) out = F.relu(self.conv2(out)) out += residual out = out.view(out.size(0),-1) return self.classifier(out)</p>
</blockquote>
</aside>
<p>Thanks for the great snippet! I just have one question. With additional blocks is the residual added like this:</p>
<pre><code class="lang-auto">def forward(self,x):
        #First block
        residual = x
        out = F.relu(self.conv1(x))
        out = F.relu(self.conv2(out))
        
        out += residual
        
        #Second block
        out = F.relu(self.conv1(x))
        out = F.relu(self.conv2(out))
        
        out += residual

        out = out.view(out.size(0),-1)
        return self.classifier(out)
</code></pre>
<p>or like this:</p>
<pre><code class="lang-auto">def forward(self,x):
        #First block
        residual1 = x
        out = F.relu(self.conv1(x))
        out = F.relu(self.conv2(out))
        
        out += residual1
        residual2 = out

        #Second block
        out = F.relu(self.conv1(x))
        out = F.relu(self.conv2(out))
        
        out += residual2

        out = out.view(out.size(0),-1)
        return self.classifier(out)
</code></pre>
<p>Clarification is much appreciated. I dont seem to be able to de-code the pytorch vision model <img src="https://discuss.pytorch.org/images/emoji/apple/see_no_evil.png?v=12" title=":see_no_evil:" class="emoji" alt=":see_no_evil:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>I’m not sure how the second block works, as you are using the same layers with the same input.<br>
Basically</p>
<pre><code class="lang-python">out = F.relu(self.conv1(x))
out = F.relu(self.conv2(out))
</code></pre>
<p>is repeated, so that you could remove one call of these layers.</p>
<p>However, in your first example you are throwing away <code>out</code> using the second block, so only one block is used.<br>
In your second example you are calculating <code>out = out + (out + x)</code>. So again the second block is unnecessary, since you’ve already calculated the output of the block.</p>
<p>Could you explain your use case a bit?</p>

Ditlev_Jorgensen: <p>Sorry I just copy-pasted the example code.</p>
<p>What i want to know is how the blocks are connected.<br>
Lets say i make a class containing two blocks each containing 3 convolutional layers:</p>
<pre><code class="lang-auto">class Two_Blocks(nn.Module):
    def __init__(self):
        self.block1 = nn.Sequential(         
            nn.Conv1d(
                in_channels=128,              
                out_channels=128,             
                kernel_size=5,              
                stride=1,                   
                padding=2,                  
            ),
            nn.BatchNorm1d(128),                       
            nn.ReLU(),
            nn.Conv1d(
                in_channels=128,              
                out_channels=128,             
                kernel_size=5,              
                stride=1,                   
                padding=2,                  
            ),
            nn.Conv1d(
                in_channels=128,              
                out_channels=128,             
                kernel_size=5,              
                stride=1,                   
                padding=2,                  
            ),
            nn.BatchNorm1d(128),                       
            nn.ReLU(),
        )
        self.block2 = nn.Sequential(         
            nn.Conv1d(
                in_channels=128,              
                out_channels=128,             
                kernel_size=3,              
                stride=1,                   
                padding=1,                  
            ),
            nn.BatchNorm1d(128),                       
            nn.ReLU(),
            nn.Conv1d(
                in_channels=128,              
                out_channels=128,             
                kernel_size=3,              
                stride=1,                   
                padding=1,                  
            ),
            nn.Conv1d(
                in_channels=128,              
                out_channels=128,             
                kernel_size=3,              
                stride=1,                   
                padding=1,                  
            ),
            nn.BatchNorm1d(128),                       
            nn.ReLU(),
        )   

    ### END ###
        
        self.Global_Avg_Pool = nn.AvgPool1d(kernel_size = 2)
        
        self.end = nn.Sequential(
                nn.Dropout(p=0.3),
                nn.Linear(128*1000, 3),
        )
        
        
    def forward(self, x):
        
        residual1 = x    #Save input as residual
        x = self.block1(x)
        
        x += residual1 #add input to output of block1
        residual2 = x  #save output of block1 as residual
        
        x = self.block2(x)
        
        x += residual2 #add output of block1 to output of block2
        
        x = self.Global_Avg_Pool(x) #Global average pooling instead of fully connected.
        x = x.view(-1, 128*1000)
        x = F.softmax(self.end(x),dim=1)
        
        return x
</code></pre>
<p>Does this make it more clear?</p>
<p>The other way would be to do the following instead:</p>
<pre><code class="lang-auto">def forward(self, x):
        
        residual = x   #Save input as residual
        x = self.block1(x)
        
        x += residual #add input to output of block1
       
        x = self.block2(x)
        
        #The same input is added for block 2 as for block 1:
        x += residual #add input to output of block2 
        
        x = self.Global_Avg_Pool(x) #Global average pooling instead of fully connected.
        x = x.view(-1, 128*1000)
        x = F.softmax(self.end(x),dim=1)
        
        return x
</code></pre>
<p>In my head the first example is the correct one. But just wanted to be sure that the residual saved in block one is not the same used for all blocks throughout the model. Sorry if i overcomplicated my question.</p>

ptrblck: <p>The first example looks like the “common” res net architecture, i.e. you add the residual before the block to its output.<br>
I wouldn’t say it’s the right approach, as the second one also looks interesting. There you add the same residual to both block outputs. It looks a bit like <a href="https://arxiv.org/pdf/1608.06993.pdf">Densely Connected Convolutional Networks</a>.</p>
<p>Anyway, if you want to stick to a ResNet-lile architecture, the first approach looks good.</p>

Ditlev_Jorgensen: <p>Thanks alot for clarifying! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=6" title=":smiley:" class="emoji" alt=":smiley:"></p>

Sushil_Khadka: <p>residual = x<br>
Doesn’t this stmt do copy by reference? So residual variable will point to the same memory location as out. Please correct me If I’m wrong.</p>

ptrblck: <p>It won’t as seen in this simple test:</p>
<pre><code class="lang-python">class block1(nn.Module):
    def __init__(self, ni):
        super(block1, self).__init__()
        self.conv1 = nn.Conv2d(ni, ni, 1)

    def forward(self,x):
        residual = x
        out = F.relu(self.conv1(x))
        print((residual - out).abs().max())
        return out

m = block1(1)
x = torch.randn(1, 1, 224, 224)
out = m(x)
# tensor(4.7888, grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
<p>since neither <code>self.conv1</code> nor <code>F.relu</code> are manipulating the input inplace.<br>
In case you are unsure if any layer will manipulate the input inplace, you should <code>clone()</code> it:</p>
<pre><code class="lang-python"># wrong since relu will manipulate x inplace and thus also residual
class block1(nn.Module):
    def __init__(self):
        super(block1, self).__init__()

    def forward(self,x):
        residual = x
        out = F.relu(x, inplace=True)
        print((residual - out).abs().max())
        return out

m = block1()
x = torch.randn(1, 1, 224, 224)
out = m(x)
# tensor(0.)

# right
class block1(nn.Module):
    def __init__(self):
        super(block1, self).__init__()

    def forward(self,x):
        residual = x
        out = F.relu(x.clone(), inplace=True)
        print((residual - out).abs().max())
        return out

m = block1()
x = torch.randn(1, 1, 224, 224)
out = m(x)
# tensor(3.7760)
</code></pre>

Sushil_Khadka: <p>Thanks! I now understand very well.</p>