Residual blocks with 1 layer per block?

username: jayz
cooked: <p>Hey,</p>
<p>I am currently reading the resnet paper, and I noticed their residual blocks always contain two convolutions. I see the first convolution is used to map the input channels to the desired channel number of the residual block (if there is a change in channel dimensions between subsequent residual blocks), while the second convolution keeps the channel dimension fixed. When there is no change in dimension between two residual blocks, both keep the input channel dimension fixed. I was wondering, why is it actually necessary to have two convolutions in the same block? One could just have a single convolution:</p>
<pre><code class="lang-python">xconv = nn.Conv2d(in_channels, out_channels, kernel_size=(k,k))(x_in)
</code></pre>
<p>and for the convolution corresponding to the skip connection for computing the residual, one would have:</p>
<pre><code class="lang-python">xskip = nn.Conv2d(in_channels, out_channels, kernel_size=(1,1))(x_in)
</code></pre>
<p>Then, the output of the residual block:</p>
<pre><code class="lang-python">x = xconv + xskip
</code></pre>
<p>Is the reason for having blocks with two (or more) layers just a design choice, or is there a specific reason to use at least two conv layers (and not one) per residual block?</p>
<p>Skip connections are just elementwise additions and computationally cheap, therefore, I can’t imagine that computational effort would be the explanation.</p>
<p>Thanks!<br>
Best, JZ</p>

username: ptrblck
cooked: <p>I’m not sure if you are referring to <code>BasicBlock</code> and <code>Bottleneck</code> defined <a href="https://github.com/pytorch/vision/blob/93104c1697384273a75381d57be2f872265eeefb/torchvision/models/resnet.py#L57-L161">here</a>, but note that:</p>
<pre><code class="lang-python">        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)

        out = self.conv2(out)
        out = self.bn2(out)
        out = self.relu(out)
</code></pre>
<p>cannot be replaced with a single convolution.</p>

username: jayz
cooked: <p>Yes, referring to that.<br>
The layers are defined like:</p>
<pre><code class="lang-python">self.conv1 = conv3x3(inplanes, planes, stride)
self.bn1 = norm_layer(planes)
self.relu = nn.ReLU(inplace=True)
self.conv2 = conv3x3(planes, planes)
self.bn2 = norm_layer(planes)
</code></pre>
<p>If its just about getting the right shape of out, one could replace it with a single convolution:</p>
<pre><code class="lang-python">conv = conv3x3(inplanes,planes,stride)
</code></pre>
<p>no?<br>
But you mean the fact that there is a nonlinearity ReLU in between changes that?</p>
<p>Best, JZ</p>

username: ptrblck
cooked: <p>Right, if you are only concerned about getting the shape right a single layer would do it. However, the actual processing of multiple layers with non-linearities between them would not be the same, so you might lose the actual training properties of these blocks.</p>

username: jayz
cooked: <p>ok, thanks for clarifying!</p>

username: jayz
cooked: <p>Hey once more,</p>
<p>so I continued looking at the resnet code. The downsampling function is:</p>
<pre><code class="lang-python">downsample = nn.Sequential(
                conv1x1(self.inplanes, planes * block.expansion, stride),
                norm_layer(planes * block.expansion),
            )
</code></pre>
<p>This means that when stride s &gt; 1, just every s’th entry of the input x is “transferred” via the skip connection. Isn’t this a loss of valuable information? Is this what actually happens? Because then it could be better to use a pooling function like avgpooling with stride=s before the conv1x1 to take care of the downsampling and then perform the conv1x1 with stride=1?</p>
<p>Thanks!<br>
Best, JZ</p>

username: ptrblck
cooked: <p>Sure, your idea sounds valid and sounds as if some experiments might be worth to try out.<br>
Let us know if you see any improvement <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

username: jayz
cooked: <p>Yes, I will do that!</p>