Help in ResNet BasicBlock architecture

username: Eyal
cooked: <p>Hello there.<br>
I’m currently doing a deep learning course and trying to implement a resnet, while doing so, I’ve been bumping into this implementation in github:</p>
<pre><code class="lang-auto">class BasicBlock(nn.Module):
    def __init__(self, in_planes, planes):
        super(BasicBlock, self).__init__()
        self.conv1 = nn.Conv2d(in_planes, planes, 3, padding = 1)
        self.bn1 = nn.BatchNorm2d(planes)
        
        self.conv2 = nn.Conv2d(planes, planes, 3, padding = 1)
        self.bn2 = nn.BatchNorm2d(planes)
        
        self.shortcut = nn.Sequential()
        if (in_planes != planes):
            self.shortcut = nn.Sequential( nn.Conv2d(in_planes, planes, 3, padding = 1),
                                           nn.BatchNorm2d(planes))
            
    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out += self.shortcut(x) #for the input
        out = F.relu(out)
        return out
</code></pre>
<p>I don’t understand the shortcut, the if statements make it executable only if the channels between two adjacent layers are the same, however, if they’re not it just executes the regular basic block again.</p>
<p>However, later in the code, they execute the resnet like that:</p>
<pre><code class="lang-auto">    def __init__(self, in_channel, hidden_channels, num_classes):
        super(SmallResNet, self).__init__()
        self.conv = nn.Conv2d(in_channel, hidden_channels[0], 3, padding = 1) #first conv
        self.bn = nn.BatchNorm2d(hidden_channels[0]) #then batchNorm
        #now use 3 residual blocks
        self.res1 = BasicBlock(hidden_channels[0],hidden_channels[1])
        self.res2 = BasicBlock(hidden_channels[1],hidden_channels[2])
        self.res3 = BasicBlock(hidden_channels[2],hidden_channels[3])
        #now do the maxpooling
        self.maxpool = nn.MaxPool2d(2, 2) 
        self.fc = nn.Linear(hidden_channels[3] * 16 * 16 , num_classes) #from maxpooling 
</code></pre>
<p>And calling it with hidden channels</p>
<p><code>hidden = [16,32,64,128]</code><br>
So it seems to me like missing the whole point of ResNet, because there are no ‘‘skips’’ here.<br>
What am I seeing wrong?</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>If the number of channels differ, the additional <code>conv</code> and batchnorm layers in <code>shortcut</code> will make sure that you can add the residual connection back to <code>out</code>.<br>
On the other hand, if the channels match already, <code>x</code> will be directly added to <code>out</code>, since an empty <code>nn.Sequential</code> module will act as an identity:</p>
<pre><code class="lang-python">seq = nn.Sequential()
x = torch.randn(1, 3, 24, 24)
out = seq(x)
print((out == x).all())
&gt; tensor(True)
</code></pre>
<p>The <a href="https://github.com/pytorch/vision/blob/80f41f8d32b1fcb380d5df2116063af7034ff29a/torchvision/models/resnet.py#L37-L83">torchvision implementation</a> uses a similar approach, but will skip the <code>downsample</code> layer, if it’s not needed.</p>

username: Eyal
cooked: <p>Thank you very much! So just to make sure I understood when he called the net with<br>
<code>hidden = [16,32,64,128]</code><br>
No residuals were taken place but adding the conv - batchnorm block twice. right?</p>

username: ptrblck
cooked: <p>The <code>hidden</code> list is used to initialize the layers with the right channels. No forward pass is used in this call.<br>
Let me know, if I misunderstood the question.</p>

username: Eyal
cooked: <p>What I mean is, when the hidden layers were initialized as<br>
<code>hidden = [16, 32, 64, 128]</code><br>
Not time was the Identity added because the channels differ in every two adjacent layers.<br>
So I’m wondering, in this initializment, What’s the role of the resnet? it seems to me like it’s missing the whole point.</p>

username: ptrblck
cooked: <p>Thanks for clarifying the question.<br>
I’m not familiar with the model and would suggest to ask the author.<br>
My best guess is that the author indented to write the model in a flexible way and this particular implementation would use the <code>shortcut</code> blocks.</p>