Adding a Global Average Pooling layer in VGG. Am I doing this correctly?

Valerio_Biscione: <p>Hello. I am trying to do a bit of model surgery to add a GAP layer in a VGG16 net, just before the classifier, after the conv layers. I am replacing the <code>AdaptiveAvgPool2d((7, 7))</code> normally saved in network.avgpool.<br>
Basically I am doing this:</p>
<pre><code class="lang-auto">network = torchvision.models.vgg16(pretrained=False, progress=True)
network.avgpool = torch.nn.AdaptiveAvgPool2d((1, 1))
network.classifier[0] = torch.nn.Linear(512, 4096)
</code></pre>
<p>the problem is that this network takes ages to learn, and the testing accuracy is not even that great. I am comparing it with supposedly the same implementation in keras, training it on the same dataset. In keras it trains withtout problem and reaches an accuracy of ~100%, in PyTorch it takes ages, and the accuracy is ~70%, so something is obviously <em>not</em> the same. After having spent long time checking that everything is exactly the same, I am now wondering if my way of adding a GAP layer is wrong. Maybe doing it in that way is breaking some connection in the autograd?<br>
Any help is deeply appreciated, I am totally stuck with this.</p>

ptrblck: <p>Could you explain, what a GAP layer is?</p>
<p>I ran a quick profile on the original VGG16 model and using your modifications and it seems your modifications yield a small speedup (which is expected):</p>
<pre><code class="lang-python">import torch
import torchvision.models as models
import time


model = models.vgg16().cuda()
x = torch.randn(32, 3, 224, 224, device='cuda')

nb_iters = 100

# warmup
for _ in range(10):
    out = model(x)


torch.cuda.synchronize()
t0 = time.time()
for _ in range(nb_iters):
    out = model(x)
    out.backward(torch.ones_like(out))
    model.zero_grad()
torch.cuda.synchronize()
t1 = time.time()
print((t1 - t0)/nb_iters)
&gt; 0.1372007703781128

model.avgpool = torch.nn.AdaptiveAvgPool2d((1, 1)).cuda()
model.classifier[0] = torch.nn.Linear(512, 4096).cuda()

torch.cuda.synchronize()
t0 = time.time()
for _ in range(nb_iters):
    out = model(x)
    out.backward(torch.ones_like(out))
    model.zero_grad()
torch.cuda.synchronize()
t1 = time.time()
print((t1 - t0)/nb_iters)
&gt; 0.1326659917831421
</code></pre>
<p>You can set <code>torch.backends.cudnn.benchmak = True</code> to profile the kernels and select the fastest one, but I didn’t get a major speedup for this use case.</p>

Valerio_Biscione: <p>Sorry, GAP = <a href="https://alexisbcook.github.io/2017/global-average-pooling-layers-for-object-localization/" rel="noopener nofollow ugc">Global Average Pooling</a>. More than timing, I was worried of correctedness of assigning to <code>network.avgpool</code> the new operation. Is this the correct way of doing  so? I would assume that, since I would probably get an error if it wasn’t. I am just scraping the bottom of the barrel in order to understand why I get such weird results. For example, the loss goes down (to an accuracy of around 100%) before spiking up to a extremely high value for few iterations:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/1/31eae967bcf36fbc2d22224d5609b952527bc502.png" data-download-href="https://discuss.pytorch.org/uploads/default/31eae967bcf36fbc2d22224d5609b952527bc502" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/31eae967bcf36fbc2d22224d5609b952527bc502_2_690x362.png" alt="image" data-base62-sha1="77AGUyqTqKVH70aLk1eexoTzeSu" width="690" height="362" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/31eae967bcf36fbc2d22224d5609b952527bc502_2_690x362.png, https://discuss.pytorch.org/uploads/default/original/3X/3/1/31eae967bcf36fbc2d22224d5609b952527bc502.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/1/31eae967bcf36fbc2d22224d5609b952527bc502.png 2x" data-dominant-color="FBFCFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">848×446 24.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <aside class="quote no-group" data-username="Valerio_Biscione" data-post="3" data-topic="87330">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/valerio_biscione/48/22759_2.png" class="avatar"> Valerio_Biscione:</div>
<blockquote>
<p>Is this the correct way of doing so? I would assume that, since I would probably get an error if it wasn’t.</p>
</blockquote>
</aside>
<p>Yes, it should be the right way. Since you are also changing the first linear layer, you would have a shape mismatch, if you are assigning the new pooling layer to a new attribute (e.g. due to a typo).</p>
<p>Are you using an adaptive optimizer? If not, could you try to lower the learning rate before the spike?<br>
Sometimes when the learning rate is too high, your model might hit these high loss peaks.</p>

Valerio_Biscione: <p>I am using a standard SGD with no adaptive optimizer. I will try to lower the learning rate, thanks!</p>

Valerio_Biscione: <p>I just wanted to report about the spikes and the slow learning. I have tried with a learning rate of 0.0001 (compared to the 0.001 I had before) and it didn’t help at all. It took REALLY LONG to learn (compared to SGD 0.0001), it got rid of the spikes, but accuracy at test time was really slow.<br>
I then changed it to Adam optimizer: accuracy at test time is still high, and it takes 1/10th of the time to train!</p>