Runtime Error: Inplace operation error with custom Pytorch layer

Shiny_Sheepy: <p>Hi,<br>
I am writing a custom binarization layer which takes an input and outputs its binarization result. As this process does not change the expectation of the input variable, the back-prop gradient should stay unchanged.</p>
<pre><code class="lang-python">import torch
from torch.autograd import Function
import torch.nn as nn

class _DefineBinarization(Function):
    '''As this binarization process does not change expectation,
    we pass the gradient unchanged.
    '''
    @staticmethod
    def forward(ctx, input):
        p = torch.bernoulli((1+input)/2).to(torch.bool)
        np = torch.bitwise_not(p)
        input[p] = 1
        input[np] = -1
        return input

    @staticmethod
    def backward(ctx, grad):
        return grad


class Binarization(nn.Module):
    def __init__(self):
        super(Binarization, self).__init__()
        return
    
    def forward(self, input):
        return _DefineBinarization.apply(input)

class BinarizeLayer(nn.Module):
    def __init__(self, num):
        super(BinarizeLayer, self).__init__()
        self.binarize = Binarization()
        self.tanh = nn.Tanh()

    def forward(self, input):
        input = self.tanh(input)
        input = (self.binarize(input)+1)/2
        return input

binarize_layer = BinarizeLayer(10)
x = torch.rand(1,10).requires_grad_()
y = binarize_layer(x)
h = torch.mean(y)
h.backward()
</code></pre>
<p>This gives me the error “RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 10]], which is output 0 of TanhBackward, is at version 2; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).”</p>
<p>Shouldn’t Pytorch simply follow my backward function without checking whether this forward operation is dirty or not?</p>
<p>Thanks!</p>

Scott_Hoang: <p>can you try<br>
input = self.tanh(input.clone()) ?</p>

Shiny_Sheepy: <p>Thanks for your reply. However, using <code>self.tanh(input.clone())</code> gives me the same error.<br>
Nonetheless, we did manage to find a similar temporal workaround</p>
<pre><code class="lang-python">class _DefineBinarization(Function):
    '''As this binarization process does not change expectation,
    we pass the gradient unchanged.
    '''
    @staticmethod
    def forward(ctx, input):
        p = torch.bernoulli((1+input)/2).to(torch.bool)
        np = torch.bitwise_not(p)
        new_input = input.clone()
        new_input[p] = 1
        new_input[np] = -1
        return new_input

    @staticmethod
    def backward(ctx, grad):
        # input = ctx.saved_tensors
        return grad
</code></pre>
<p>Still confused why the error was thrown and why this temporary fix worked.</p>

Scott_Hoang: <p>new_input[p]=1<br>
new_input[np] = -1<br>
are in place operation i think.</p>

Shiny_Sheepy: <p>Yes these two ops are indeed inplace. What I am confused at is that since it’s in my custom layer, pytorch is not supposed to track inplace modifications and should simply run my backward pass, and thus this error should not be thrown.</p>

Scott_Hoang: <p>did you forgot to turn off the grad flag ?</p>

Shiny_Sheepy: <p>According to <a href="https://pytorch.org/docs/stable/notes/extending.html" rel="noopener nofollow ugc">this official doc</a>,</p>
<p>" <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function.forward" rel="noopener nofollow ugc"><code>forward()</code></a> - the code that performs the operation. It can take as many arguments as you want, with some of them being optional, if you specify the default values. All kinds of Python objects are accepted here. <code>Tensor</code> arguments that track history (i.e., with <code>requires_grad=True</code> ) will be converted to ones that don’t track history before the call, and their use will be registered in the graph."</p>
<p>In the forward call, the gradient flag doesn’t need to be explicitly turned off. So I don’t think the grad flag is the problem.</p>