Global Average Pooling in Pytorch

Bixqu: <p>I am trying to use global average pooling, however I have no idea on how to implement this in pytorch. So global average pooling <a href="https://www.quora.com/What-is-global-average-pooling" rel="noopener nofollow ugc">is described</a> briefly as:</p>
<blockquote>
<p>It means that if you have a 3D 8,8,128 tensor at the end of your last convolution, in the traditional method, you flatten it into a 1D vector of size 8x8x128. And you then add one or several fully connected layers and then at the end, a softmax layer that reduces the size to 10 classification categories and applies the softmax operator.</p>
</blockquote>
<blockquote>
<p>The global average pooling means that you have a 3D 8,8,10 tensor and compute the average over the 8,8 slices, you end up with a 3D tensor of shape 1,1,10 that you reshape into a 1D vector of shape 10. And then you add a softmax operator without any operation in between. The tensor before the average pooling is supposed to have as many channels as your model has classification categories.</p>
</blockquote>
<p>So, to me it sounds like in one case you do</p>
<pre><code>tensor = tensor.view(8*8*10, 10)
tensor = self.Linear(tensor)   # size 10
tensor = self.Softmax(tensor)
</code></pre>
<p>and, in the other, you do</p>
<pre><code>tensor = self.Conv2d(output_size = 10, kernel_size=1)  #to get [10x8x8] size
tensor = self.GovalAvgPooling(tensor)  #whatever this is , to get [10, 1, 1]
tensor = self.Squeeze_Dims(tensor)  # to just get a vector [10]
tensor = self.Softmax(tensor)
</code></pre>
<p>Here are the questions:</p>
<ol>
<li>Are the above examples correct, keeping in mind the description of global average pooling?</li>
<li>How can I do the global average pooling? Should I use the functional module?</li>
<li>The paper I am trying to reproduce (residual nets) says that:</li>
</ol>
<blockquote>
<p>The network ends with a global average pooling, a 10-way fully-connected layer, and softmax.</p>
</blockquote>
<p>But this does not make sense ?? Why do they need the 10-way fc layer?</p>

allenye0119: <p>Is <a href="http://pytorch.org/docs/master/nn.html#torch.nn.AvgPool2d" rel="nofollow noopener">nn.AvgPool2d()</a> what you’re looking for?</p>

Bixqu: <p>That is the pooling layer, yes, but it seems to be more complicated than just using pooling. Or am I wrong?</p>

vabh: <p>Global average pooling means that you average <em>each</em> feature map separately. In your case if the feature map is of dimension <code>8 x 8</code>, you average each and obtain a single value. The important part here is that you do the average operation per-channel. You can think of each of the feature maps as the final feature representation per category over which you want to do classification.</p>
<p>To do this you can apply either <code>nn.AvgPool2d</code> or <code>F.avg_pool2d</code> with <code>kernel_size</code> equal to the dimensions of the feature maps (in this case, 8).</p>
<p>The 10-way fc is because there are 10 categories. It’s like you extract features from all the preceeding conv layers and feed them into a linear classifier.</p>

smth: <p>why not use <code>torch.mean</code> to achieve this?</p>

ywu36: <p>torch.mean works on one dimension instead of all three dimensions.</p>

ywu36: <p>Try this:</p>
<blockquote>
<p>F.avg_pool3d(tensor, kernel_size=input.size()[2:]).view(input.size()[0],-1)</p>
</blockquote>

irustandi: <p>Just a note, the SqueezeNet architecture (available in PyTorch model zoo) uses global average pooling. Here’s global average pooling as implemented there:</p>
<pre><code>final_conv = nn.Conv2d(512, self.num_classes, kernel_size=1)
self.classifier = nn.Sequential(
    nn.Dropout(p=0.5),
    final_conv,
    nn.ReLU(inplace=True),
    nn.AvgPool2d(13)
)
</code></pre>
<p>512 is the number of channels in the feature maps feeding in to this layer, and 13 is the number of rows and columns in the feature maps going in to this layer. You’ll need to change these depending on your network structure.</p>

wangchust: <p>x = nn.avg_pool2d(x, x.size()[2:])  works fine when x.shape=N * C * H * W</p>

jdhao: <p>Another way to do global average pooling for each feature map is to use <code>torch.mean</code> as suggested by <a class="mention" href="/u/soumith_chintala">@Soumith_Chintala</a>, but we need to flatten each feature map into to vector. The following snippet illustrates the idea,</p>
<pre><code class="lang-auto"># suppose x is your feature map with size N*C*H*W
x = torch.mean(x.view(x.size(0), x.size(1), -1), dim=2)
# now x is of size N*C
</code></pre>
<p>Also you can use <a href="http://pytorch.org/docs/0.3.0/nn.html#torch.nn.functional.adaptive_avg_pool2d"><code>adaptive_avg_pool2d</code></a> to achieve global average pooling, just set the output size to (1, 1),</p>
<pre><code class="lang-auto">import torch.nn.functional as F
x = F.adaptive_avg_pool2d(x, (1, 1))
</code></pre>

Lin_Eric: <p>use nn.AdaptiveMaxPool2d, <a href="http://pytorch.org/docs/master/nn.html#adaptivemaxpool2d" rel="nofollow noopener">http://pytorch.org/docs/master/nn.html#adaptivemaxpool2d</a></p>

Sharpy: <p>Did anyone make any benchmarks? I’m guessing <code>mean</code>  was probably the fastest?</p>

tset0401: <p>I’ve been searching an detail explanation for this type of pooling layer but it seems like a magic function, do you have any idea about it?</p>

bluesky314: <p>Think of it like this: suppose I am classifying dogs and all the images of dog are on the right. The first fc layer will form very strong weights with that region and not the rest as weights keep fluctuting there. If i avg pool instead before the fc, I make the fc invariant of position, so it will not recognise dogs in any position and not only the ones it was trained on.</p>

Pt_S: <pre><code class="lang-python">x = torch.randn(1, 256, 100, 100)

x = torch.nn.AvgPool2d(kernel_size = 100, stride = 0, padding = 0, ceil_mode=False, count_include_pad=True)(x)

x = x.squeeze()

print(x.shape) # torch.Size([256])
</code></pre>

bluesky314: <p>edit : it will recognise*</p>

huangbiubiu: <p><code>nn.AdaptiveAvgPool2d(1)</code> is another solution</p>

huangbiubiu: <p>A simple benchmark (run on Google colab):</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

x = torch.randn((256, 96, 128, 128)).cuda()

%timeit F.avg_pool2d(x, x.size()[2:])

%timeit F.adaptive_avg_pool2d(x, (1, 1))

%timeit torch.mean(x.view(x.size(0), x.size(1), -1), dim=2)

</code></pre>
<p>gives the output:</p>
<pre><code class="lang-auto">1000 loops, best of 3: 104 ms per loop
10 loops, best of 3: 104 ms per loop
1 loop, best of 3: 208 ms per loop
</code></pre>
<p>Looks <code>mean</code> is actually faster.</p>

xssChauhan: <p>Global Average Pooling was introduced in the paper <a href="https://arxiv.org/pdf/1312.4400.pdf" rel="nofollow noopener">Network in Network</a>.</p>
<p>Up until this paper most networks used the convolutional backbones as feature extractors, and then these features were fed into fully connected layers, followed by an output layer. The problem with fully connected layers are manifold:</p>
<ul>
<li>They are prone to overfitting, and rely on regularizers like Dropout.</li>
<li>They sit like a black box between the categorical outputs, and the spatial features extracted by convolutional backbone. In this way the correspondence between the features extracted, and the output are not clear. This is specially true while back propagating.</li>
</ul>
<p>The idea is to remove an overfitting prone black box, and to replace it with a layer that uses the spatial features extracted by convolutional layers for outputs. Now, how do you do that?</p>
<p>First, you get in something that does not learn, or has no parameters.<br>
Second, you want to directly translate the spatial features learned as direct indicators of categorical signals.<br>
What is something that can fit well with these two points? Pooling. Not only does it not learn, but it also outputs something that directly corresponds to the spatial features learned by convolutional backbone.</p>
<p>And hence the idea of global average pooling, where you just average each channel to give an output.</p>
<p>Another direct benefit is during backpropagation: the categorical level errors are directly informed to the places of their origin in the convolutional backbone, and the learning is much better.</p>

Valerio_Biscione: <p>I keep seeing no reason to not use torch.mean… maybe anyone can explain clearly why it shouldn’t work?</p>