How do I create an L2 pooling 2d layer?

username: ProGamerGov
cooked: <p>In this article here: <a href="https://arxiv.org/abs/1511.06394" class="inline-onebox" rel="noopener nofollow ugc">[1511.06394] Geodesics of learned representations</a>, they describe using L2 pooling layers instead of max pooling or average pooling. The details of their implementation can be found under under 3.1:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a23628dbfaa4ac66fba4f176daa9fe7b089c5760.png" data-download-href="https://discuss.pytorch.org/uploads/default/a23628dbfaa4ac66fba4f176daa9fe7b089c5760" title="equation"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a23628dbfaa4ac66fba4f176daa9fe7b089c5760.png" alt="equation" data-base62-sha1="n8ZnYdTaOFn9KeafhGrdTZD8vS0" width="690" height="254" data-dominant-color="E2E2E2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">equation</span><span class="informations">866×319 56.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m having trouble trying to figure out how to translate their equations to PyTorch, and I’m unsure as to how I would create a custom 2d pooling layer as well.</p>
<p>How do I implement this pooling layer in PyTorch?</p>
<p>I have the MaxPooling2d class rewritten like this:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.nn.modules.utils import _pair, _quadruple

class MaxPool2d(torch.nn.Module):

    def __init__(self, kernel_size=3, stride=1, padding=0):
        super(MaxPool2d, self).__init__()
        self.k = _pair(kernel_size)
        self.stride = _pair(stride)
        self.padding = _quadruple(padding)
    
    def forward(self, x):
        x = F.pad(x, self.padding, mode='reflect')
        x = x.unfold(2, self.k[0], self.stride[0]).unfold(3, self.k[1], self.stride[1])
        x = x.contiguous().view(x.size()[:4] + (-1,))
        pool, indices = torch.max(x, dim=-1)
        return pool
</code></pre>
<p>And based on that I create the L2 layer like this:</p>
<pre><code class="lang-auto">class L2Pool2d(torch.nn.Module):

    def __init__(self, kernel_size=3, stride=1, padding=0):
        super(L2Pool2d, self).__init__()
        self.k = _pair(kernel_size)
        self.stride = _pair(stride)
        self.padding = _quadruple(padding)
        
    def l2(self, x, constant=0, epsilon = 1e-6):
        pool = torch.sum(x - constant, dim=-1)
        return torch.sqrt(epsilon + pool)

    def forward(self, x):
        x = F.pad(x, self.padding, mode='reflect')
        x = x.unfold(2, self.k[0], self.stride[0]).unfold(3, self.k[1], self.stride[1])
        x = x.contiguous().view(x.size()[:4] + (-1,))
        pool = self.l2(x)
        return pool
</code></pre>

username: Rdijk
cooked: <p>I’m also looking for the implementation of this layer in PyTorch. Have you managed to succesfully implement it?</p>

username: Eta_C
cooked: <p>Since the following answer implement L2 Pooling in TF</p><aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/36620995/how-to-use-l2-pooling-in-tensorflow" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/3251207/yw-p-kwon" target="_blank" rel="noopener nofollow ugc">
    <img alt="YW P Kwon" src="https://i.stack.imgur.com/fFPz4.png?s=128&amp;g=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/36620995/how-to-use-l2-pooling-in-tensorflow" target="_blank" rel="noopener nofollow ugc">How to use L2 pooling in Tensorflow?</a>
</h4>

<div class="tags">
  <strong>python, tensorflow, conv-neural-network</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/3251207/yw-p-kwon" target="_blank" rel="noopener nofollow ugc">
    YW P Kwon
  </a>
  on <a href="https://stackoverflow.com/questions/36620995/how-to-use-l2-pooling-in-tensorflow" target="_blank" rel="noopener nofollow ugc">10:55AM - 14 Apr 16 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<p>
May be it would be</p>
<pre><code class="lang-python">class L2Pool(nn.Module):
    def __init__(self, *args, **kwargs):
        super().__init__()
        self.pool = nn.AvgPool2d(*args, **kwargs)
    def forward(self, x):
        return torch.sqrt(self.pool(x ** 2))
</code></pre>
<p><strong>I AM NOT SURE IT IS RIGHT…</strong></p>

username: Rdijk
cooked: <p>Looking at different implementations I found for tensorflow, like this one: <a href="https://stackoverflow.com/questions/56782868/how-to-implement-a-l2-pooling-layer-in-keras" class="inline-onebox" rel="noopener nofollow ugc">tensorflow - How to implement a L2 pooling layer in Keras? - Stack Overflow</a> , I think you are right. I see most people implementing it like this. However, I’m not quite sure it’s the same as the L2 norm as this is described as:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/0/20876c6eaa40eeb13468ef1a7094947d3a46a91d.png" alt="image" data-base62-sha1="4DLqcRyehIpwGAxE6hQDm6SvPzL" width="99" height="58"></p>
<p>The only difference in your (and other implementations I found) is the reduction by division of the number of elements (so taking the mean instead of the sum). Perhaps this reduction does not matter too much though…</p>
<p>Edit:<br>
I figured out that you can use divisor_override = 1 in avgpool() so that you won’t have this issue.</p>

username: Eta_C
cooked: <p>If reduction is not needed…<br>
Option 1:</p>
<pre><code class="lang-python">class L2Pool(nn.Module):
    def __init__(self, *args, **kwargs):
        super().__init__()
        ## DO NOT USE nn.Parameter HERE
        self.weight = torch.zeros(kernel_h, kernel_w).float()
    def forward(self, x):
        return torch.sqrt(nn.functional.conv2d(x ** 2, self.weight), stride...and so on)
</code></pre>
<p>Option 2</p>
<pre><code class="lang-python">class L2Pool(nn.Module):
    def __init__(self, *args, **kwargs):
        super().__init__()
        self.pool = nn.AvgPool2d(*args, **kwargs)
        self.n = kernel_h * kernel_w
    def forward(self, x):
        return torch.sqrt(self.pool(x ** 2) * self.n)
</code></pre>
<p>Option 3</p>
<pre><code class="lang-python">class L2Pool(nn.Module):
    def __init__(self, *args, **kwargs):
        super().__init__()
        kwargs["divisor_override"] = 1  # div 1 instead of kernel size
        self.pool = nn.AvgPool2d(*args, **kwargs)
    def forward(self, x):
        return torch.sqrt(self.pool(x ** 2))
</code></pre>

username: Rdijk
cooked: <p>Thanks for the quick replies! Looks good to me now <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Now let’s hope it actually works well in the model!</p>