Why does Alexnet in torch vision use Average Pooling

kamathhrishi: <p>The original AlexNet paper does not seem to be using any average pooling layers. <a href="https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf" rel="nofollow noopener">ImageNet Classification with Deep Convolutional Neural Networks</a>, I was just curious if there was any rational behind this from which I could learn. This is what I get when I printed the model (I changed number of output classes)</p>
<pre><code class="lang-auto">AlexNet(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))
    (1): ReLU(inplace=True)
    (2): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (3): Conv2d(64, 192, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (4): ReLU(inplace=True)
    (5): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (6): Conv2d(192, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (7): ReLU(inplace=True)
    (8): Conv2d(384, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (9): ReLU(inplace=True)
    (10): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace=True)
    (12): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(6, 6))
  (classifier): Sequential(
    (0): Dropout(p=0.5, inplace=False)
    (1): Linear(in_features=9216, out_features=4096, bias=True)
    (2): ReLU(inplace=True)
    (3): Dropout(p=0.5, inplace=False)
    (4): Linear(in_features=4096, out_features=4096, bias=True)
    (5): ReLU(inplace=True)
    (6): Linear(in_features=4096, out_features=67, bias=True)
  )
</code></pre>

ptrblck: <p>The average pooling layer were added to a lot of <code>torchvision</code> models to relax the input shape limitation and allow more flexible input shapes.<br>
If you pass the expected <code>3x224x224</code> image tensors, then you’ll get the same result as with the original pooling operation.</p>

kamathhrishi: <p>Thank you ptrblck <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

minimafinder: <p>This makes sense, thank you. But I am curious: <strong>would it even be possible to have a functioning AlexNet model without the average pooling layer?</strong></p>
<p>Wouldn’t there be an output mismatch between the fifth convolution and sixth dense layer? Unless I am missing something, I think the author’s didn’t describe how they remedied this mismatch.</p>

ptrblck: <p>Sorry, I wasn’t clear enough. The <em>adaptive</em> pooling layer was added to avoid shape mismatches as it returns a defined output shape.<br>
I would assume the original model used <em>non-adaptive</em> pooling layers, but don’t know for sure.</p>

minimafinder: <p>Got it — In the paper, they explicitly state that they only used (max) pooling layers with a <code>kernel_size=3</code> and <code>stride=2</code>, but I am unsure how the author’s made it work.</p>
<p>I have been trying to <a href="https://discuss.pytorch.org/t/alexnet-output-shape-mismatch-between-conv2d-and-dense-layer/206809">recreate the model as described in the paper</a>, and the transition from the convolution to the dense layers is a roadblock I haven’t been able to resolve yet. I am probably missing something basic and would appreciate your input!</p>

ptrblck: <aside class="quote no-group" data-username="minimafinder" data-post="6" data-topic="84313">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/dfb087/48.png" class="avatar"> minimafinder:</div>
<blockquote>
<p>In the paper, they explicitly state that they only used (max) pooling layers with a <code>kernel_size=3</code> and <code>stride=2</code>, but I am unsure how the author’s made it work.</p>
</blockquote>
</aside>
<p>After looking again at the model definition this <code>MaxPool2d</code> layer is already used <a href="https://github.com/pytorch/vision/blob/c2619ba5316a50b6ecb01a9c51d78d2782cfb2dd/torchvision/models/alexnet.py#L34">here</a>.</p>
<p>Removing the adaptive pooling layer works for an input of <code>[batch_size, 3, 224, 224]</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">class AlexNet(nn.Module):
    def __init__(self, num_classes: int = 1000, dropout: float = 0.5) -&gt; None:
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(64, 192, kernel_size=5, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(192, 384, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(384, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
        )
        #self.avgpool = nn.AdaptiveAvgPool2d((6, 6))
        self.classifier = nn.Sequential(
            nn.Dropout(p=dropout),
            nn.Linear(256 * 6 * 6, 4096),
            nn.ReLU(inplace=True),
            nn.Dropout(p=dropout),
            nn.Linear(4096, 4096),
            nn.ReLU(inplace=True),
            nn.Linear(4096, num_classes),
        )

    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        x = self.features(x)
        #x = self.avgpool(x)
        x = torch.flatten(x, 1)
        x = self.classifier(x)
        return x

model = AlexNet()
x = torch.randn(16, 3, 224, 224)
out = model(x)
</code></pre>
<p>so the adaptive pooling layer seems to be an addition.</p>