How does ResNet residual layer get preserved when converted to Sequential?

Pramesh_Gautam1: <p>I am using first two blocks of resnet50 in the following way</p>
<pre><code class="lang-auto">from timm.models.resnet import resnet50
r50 = resnet50(pretrained=True)
2_layers = torch.nn.Sequential(*list(r50.children())[:-4])
</code></pre>
<p>My thought was that since the blocks are now in Sequential, the residual part won’t be used and the output of the first 2 blocks of <code>r50</code> and the final output of <code>2_layers</code> should be different. However, the output is the same. Can someone explain how is the residual part retained when the layers are converted to <code>Sequential</code>? Link to code would be useful. Thanks in advance!</p>

Matias_Vasquez: <p>The skip connections are defined inside of self contained <code>Modules</code> (Bottleneck &amp; BasicBlock). Since they are done in these modules, they are kept.</p>
<p>If the skip connections were done in the <code>forward</code> pass of the actual ResNet class, then they would not be kept. <a href="https://github.com/rwightman/pytorch-image-models/blob/c9f208f7f8d5917f9a900ef2f96d9aa2ffb9264e/timm/models/resnet.py#L710" rel="noopener nofollow ugc">Here</a> is the documentation for resnet from timm, directed to the <code>forward</code> method of the resnet class.</p>
<p>Here is a little dumm example to show what I mean.</p>
<p>The basic block adds 2 to the input in an extra function.<br>
The ResN block does exactly the same, adding 5 instead.</p>
<p>When I run a tensor with ones through the whole model, I get 8 in every value as expected (1+2+5).<br>
If I now do a feature extraction and put them in a <code>nn.Sequential</code> as you suggested, the result is 3.</p>
<p>It is not the same as the whole module.<br>
The <strong>+5</strong> from ResN is gone.<br>
However, the <strong>+2</strong> from the block is kept.</p>
<p>Hope this helps.</p>
<pre><code class="lang-python">import torch


class BasicBlock(torch.nn.Module):
    def __init__(self):
        super().__init__()

    def _functional_API(self, x):
        return x + 2

    def forward(self, x):
        x = self._functional_API(x)
        return x


class ResN(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.blk = BasicBlock()

    def _functional_API(self, x):
        return x + 5

    def forward(self, x):
        x = self._functional_API(x)
        return self.blk(x)


model = ResN()
feature_extraction = torch.nn.Sequential(*list(model.children()))

tensor = torch.ones(5, 5)

print(model(tensor))
print(feature_extraction(tensor))
</code></pre>
<pre><code class="lang-python"># Output:
tensor([[8., 8., 8., 8., 8.],
        [8., 8., 8., 8., 8.],
        [8., 8., 8., 8., 8.],
        [8., 8., 8., 8., 8.],
        [8., 8., 8., 8., 8.]])
tensor([[3., 3., 3., 3., 3.],
        [3., 3., 3., 3., 3.],
        [3., 3., 3., 3., 3.],
        [3., 3., 3., 3., 3.],
        [3., 3., 3., 3., 3.]])
</code></pre>

saluei: <p>The layer of ResNet is Converted to Sequential but there is a custom froward function for the ResNet block that keeps the residual part</p>
<pre><code class="lang-auto">def forward(self, x: Tensor) -&gt; Tensor:
        identity = x

         ......

        out += identity # add residual part
        out = self.relu(out)

        return out
</code></pre>
<p>full code is available at<br>
<a href="https://pytorch.org/vision/stable/_modules/torchvision/models/resnet.html#resnet50" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/stable/_modules/torchvision/models/resnet.html#resnet50</a></p>