Forward and backward pass in ResNet50

Anita: <p>Hi, I had rather a simple question. I have two encoders as below, the first one uses ResNet50 to extract feature maps for each example and the output shape is (batch_size, encoding_vector) e.g. [(32, 1000)].</p>
<p>The second encoder removes the last two layers (pooling and fc) and extracts feature maps keeping the spatial size and outputs [(batch_size, 2048, 19, 19)].</p>
<p>After the feature encoder, I then have the classifier layers.</p>
<p>My question is how does the forward pass and backward pass happens here (in the first encoder and in the second). If it’s the same in both, then are the examples from each batch passed separately to the forward pass or they are passed all of them on the same time?</p>
<p>class My_Network_1(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(My_Network_1, self).<strong>init</strong>()</p>
<pre><code>    # Initialize ResNet-50 feature extractor
    self.feature_extractor_1 = torch.hub.load('pytorch/vision:v0.10.0', 'resnext50_32x4d', pretrained=True)

def forward(self, x):

# Input [(batch_size, channel_nr, width, height)]

    h_1 = self.feature_extractor_1(x)
    
    # Output [(batch_size, 1000)]


# classifier Y = ....
   
    return Y, probs
</code></pre>
<p>class My_Network_2(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(My_Network_2, self).<strong>init</strong>()</p>
<pre><code># Remove the last two layers from ResNet-50
self.feature_extractor_2 = nn.Sequential(*list(self.feature_extractor.children())[:-2])	

def forward(self, x):

# Input [(batch_size, channel_nr, width, height)]
        
h_2 = self.feature_extractor_2(x)

# Output size [(batch_size, 2048, 19, 19)]


# classifier Y = ....
   
    return Y, probs
</code></pre>

ptrblck: <aside class="quote no-group" data-username="Anita" data-post="1" data-topic="171377">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anita/48/56469_2.png" class="avatar"> Anita:</div>
<blockquote>
<p>My question is how does the forward pass and backward pass happens here (in the first encoder and in the second).</p>
</blockquote>
</aside>
<p>I assume the classifier is a (block of) linear layers (with activation functions) and tries to map the input features to the output logits?<br>
If so, then note that the output shape would be different in both approaches, since the second one (using the 4-dimensional feature tensor) would “iterate” the additional dimensions.<br>
Here is a small example:</p>
<pre data-code-wrap="python"><code class="lang-python">batch_size = 16

# 2D
features = torch.randn(batch_size, 1000)
classifier = nn.Linear(1000, 10)

out = classifier(features)
print(out.shape)
# torch.Size([16, 10])

# 4D
features = torch.randn(batch_size, 2048, 19, 19)
classifier = nn.Linear(19, 10) # the last dimension of features defines the in_features!
out = classifier(features)
print(out.shape)
# torch.Size([16, 2048, 19, 10])
</code></pre>
<p>The linear layer using <code>in_features=19</code> will be applied to all samples from <code>dim0</code>, <code>dim1</code>, and <code>dim2</code>.<br>
You can verify it by flattening the input:</p>
<pre data-code-wrap="python"><code class="lang-python"># test
ref = classifier(features.view(-1, 19))
print((out.view(-1, 10) - ref).abs().max())
# tensor(0., grad_fn=&lt;MaxBackward1&gt;)
</code></pre>