Building ResNet from scratcg-something does not work

Ioannis_D: <p>Hello, I try to build ResNet18 from scratch (similar to the archtecture of the library), though for some reason it does not behave as I want to.<br>
In the first residual block (ResidualBlock), I dont want the downsampling layers to exist however, when I print the model, they are shown in the architecture.<br>
Am I missing something?</p>
<p>Here is the residual block:</p>
<blockquote>
<p>class ResidualBlock(nn.Module):<br>
def <strong>init</strong>(self, in_channels, out_channels, stride = 1, downsample = False):<br>
super(ResidualBlock, self).<strong>init</strong>()</p>
<pre><code>    self.conv = nn.Sequential(
                    nn.Conv2d(in_channels, out_channels, kernel_size = 3, stride = stride, padding = 1),
                    nn.BatchNorm2d(out_channels),
                    nn.ReLU(),
                    nn.Conv2d(in_channels, out_channels, kernel_size = 3, stride = stride, padding = 1),
                    nn.BatchNorm2d(out_channels))
    
    self.downsample = downsample
    print('--------------', self.downsample)
    
    self.downsample_layers = nn.Sequential(
                    nn.Conv2d(in_channels, out_channels, kernel_size = 1, stride = 2),
                    nn.BatchNorm2d(out_channels))
    
    self.relu = nn.ReLU()
    self.out_channels = out_channels

def forward(self, x):
    residual = x
    out = self.conv(x)
    if self.downsample:
        residual = self.downsample_layers(x)
    out += residual
    out = self.relu(out)
    return out
</code></pre>
</blockquote>
<p>Here is the ResNet:</p>
<blockquote>
<p>class ResNet(nn.Module):<br>
def <strong>init</strong>(self, block, layers, num_classes = 10):<br>
super(ResNet, self).<strong>init</strong>()</p>
<pre><code>    self.inplanes = 64
    
    self.conv1 = nn.Sequential(
                    nn.Conv2d(3, 64, kernel_size = 7, stride = 2, padding = 3),
                    nn.BatchNorm2d(64),
                    nn.ReLU(),
                    nn.MaxPool2d(kernel_size = 3, stride = 2, padding = 1))
    
    self.layer0 = self._make_layer(block, 64, layers[0], stride = 1, downsample = False)
    self.layer1 = self._make_layer(block, 128, layers[1], stride = 2, downsample = True)
    self.layer2 = self._make_layer(block, 256, layers[2], stride = 2, downsample = True)
    self.layer3 = self._make_layer(block, 512, layers[3], stride = 2, downsample = True)
    
    self.avgpool = nn.AvgPool2d(7, stride=1)
    self.fc = nn.Linear(512, num_classes)
    
def _make_layer(self, block, planes, blocks, stride=1, downsample = False):
    
    layers = []

    if downsample:
        layers.append(block(self.inplanes, planes, stride, downsample=True))
    
    else:
        layers.append(block(self.inplanes, planes, stride, downsample=False))
    
    self.inplanes = planes
    

    return nn.Sequential(*layers)


def forward(self, x):
    x = self.conv1(x)
    x = self.maxpool(x)
    x = self.layer0(x)
    x = self.layer1(x)
    x = self.layer2(x)
    x = self.layer3(x)

    x = self.avgpool(x)
    x = x.view(x.size(0), -1)
    x = self.fc(x)

    return x
</code></pre>
</blockquote>
<p>I have based my code on this article: <a href="https://blog.paperspace.com/writing-resnet-from-scratch-in-pytorch/" class="inline-onebox" rel="noopener nofollow ugc">Writing ResNet from Scratch in PyTorch</a> which works but I wanted to changed it a bit.<br>
Thank you</p>

ptrblck: <aside class="quote no-group" data-username="Ioannis_D" data-post="1" data-topic="168665">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ioannis_d/48/52562_2.png" class="avatar"> Ioannis_D:</div>
<blockquote>
<p>In the first residual block (ResidualBlock), I dont want the downsampling layers to exist however, when I print the model, they are shown in the architecture.<br>
Am I missing something?</p>
</blockquote>
</aside>
<p>Im not sure if I understand your use case correctly, but you are explicitly initializing <code>self.downsample_layers</code> which is why they are showing up when you are printing the model.</p>

Ioannis_D: <p>What I was aiming is, when self.downsample==True, then residual= self.downsample_layers. Can you please explain me where it is initiated?</p>

ptrblck: <p>It’s initialized in the <code>__init__</code>:</p>
<pre><code class="lang-python">    self.downsample_layers = nn.Sequential(
                    nn.Conv2d(in_channels, out_channels, kernel_size = 1, stride = 2),
                    nn.BatchNorm2d(out_channels))
</code></pre>
<p>and you could guard it with an if-condition checking for <code>self.downsample</code>.</p>

Ioannis_D: <p>Yes, but how is this added to the structure?<br>
I mean, the network architecture is defined in the <code>def forward(self, x):</code>, no?<br>
What I mean is that, why the if statement is ‘ignored’?</p>

ptrblck: <p>The forward pass is defined in the <code>forward</code> while the <code>__init__</code> is responsible to initialize all submodules, parameters, and buffers.<br>
Once you’ve created a layer in the <code>__init__</code> method, it’ll be registered and will show up when you print the model. In the <code>forward</code> you can then use these layers (or could of course skip their usage).</p>