Maxpool from scratch

username: Gilles_Jack
cooked: <p>Hi, I am trying to implement maxpool fonction from scatch (for fun) and use backward() on it.<br>
In my implementation below, the output Y of maxpool is correct (I verified it).<br>
But the gradient of the input at a zero tensor, which is wrong.</p>
<p>code :</p>
<pre><code class="lang-auto">import torch
def maxpool_fp(X):
    pool_dim = 2
    pool_stripe = 2
    bs, cx, dx, _ = list(X.size())  # batch size ; nb of channel of X ;  dimension of X
    dy = int(X.shape[2] / pool_stripe)  # dimension of Y

    Y = X[:, :, :dy, :dy] * 1  # *1 to avoid: RuntimeError: leaf variable has been moved into the graph interior
    for yn in range(bs):
        for yc, x in enumerate(X[yn]):
            for yh, h in enumerate(range(0, dx, pool_stripe)):
                for yw, w in enumerate(range(0, dx, pool_stripe)):
                    Y[yn, yc, yh, yw] = torch.max(x[h:h + pool_dim, w:w + pool_dim]).item()
    return Y


X = torch.randn(2, 2, 8, 8, requires_grad=True)

Y = maxpool_fp(X)
S = torch.sum(Y)
S.backward()

print("S =", S) # ==&gt; Correct
print("X.grad\n", X.grad) # ==&gt;  zero tensor !!!!!!!!!!
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>The <code>.item()</code> that you use converts the result of your max into a python number. And we can’t track gradient for it. So it is ignored.<br>
You will need to remove this <code>.item()</code> to get proper gradients.</p>

username: Gilles_Jack
cooked: <p>Great. Working now. Thanks a lot</p>

username: Gilles_Jack
cooked: <p>But in order to keep track of the gradient I did this :</p>
<pre><code class="lang-auto">dy = int(X.shape[2] / pool_stripe)  # dimension of Y
Y = X[:, :, :dy, :dy] * 1
</code></pre>
<p>Can you see a more elegant way to do so? Or does it look fine for you ?</p>

username: albanD
cooked: <p>Why do you need the <code>*1</code> here?<br>
Can you share the full code sample maybe that will be clearer <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Gilles_Jack
cooked: <p>Before removing item() the *1 was there avoid the Error:</p>
<blockquote>
<p>RuntimeError: leaf variable has been moved into the graph interior</p>
</blockquote>
<p>When executing the first :</p>
<pre><code class="lang-auto">
Y[yn, yc, yh, yw] = torch.max(x[h:h + pool_dim, w:w + pool_dim]).item()

</code></pre>
<p>X is_leaf attribute changed from True to False. I put the *1 to set the is_leaf attribute to False in the first place in order to avoid the change.</p>
<p>After removing item(), the *1 is still needed to avoid another Error:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [2, 2]], which is output 0 of SliceBackward, is at version 64; expected version 63 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
</blockquote>
<p>which I can’t really explain</p>
<p>You can copy/past the code below and remove the *1 if you want to reproduce the RuntimeError.  But please note that my problem was fixed by removing the .item(). I put this maxpool() function along with a hardcoded conv2d in a  CNN model, and it works on MNIST.</p>
<pre><code class="lang-auto">
import torch

def maxpool_fp(X):
    pool_dim = 2
    pool_stripe = 2
    bs, cx, dx, _ = list(X.size())  # batch size ; nb of channel of X ;  dimension of X
    dy = int(X.shape[2] / pool_stripe)  # dimension of Y

    Y = X[:, :, :dy, :dy] * 1  # &lt;=============
    for yn in range(bs):
        for yc, x in enumerate(X[yn]):
            for yh, h in enumerate(range(0, dx, pool_stripe)):
                for yw, w in enumerate(range(0, dx, pool_stripe)):
                    Y[yn, yc, yh, yw] = torch.max(x[h:h + pool_dim, w:w + pool_dim])
    return Y


X = torch.randn(2, 2, 8, 8, requires_grad=True)

Y = maxpool_fp(X)
S = torch.sum(Y)
S.backward()

print("S =", S)
print("X.grad\n", X.grad)
</code></pre>

username: albanD
cooked: <p>Ho right!<br>
Yes, you can’t modify <code>X</code> inplace because it’s a leaf. And when you do <code>Y = X[:, :, :dy, :dy]</code>, <code>Y</code> is actually a view into <code>X</code>. And so when you modify <code>Y</code> inplace by doing <code>Y[yn, yc, yh, yw] = foo</code>, you also modify X leading to the error.<br>
The right way to fix that is to use <code>.clone()</code> to ensure you get new memory that is not a view of X: <code>Y = X[:, :, :dy, :dy].clone()</code> (should be slightly faster than the *1 you use).</p>