Encounter the RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

YongyiTang92: <p>I am going to define my layer. How ever, I encounter the RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation while running backward().</p>
<p>I found that if i commented the second for loop ‘for j in range(self.number_person):’ or make ‘u_i[:,j,:] = (1 - self.lumbda)*u_i[:,j,:]’, the backward() was fine.</p>
<p>I wonder where is inplace operation and why it does not work? ‘p_rnn_feature’ and ‘u_sum’ has been compute before.</p>
<p>BTW, this code is run on pytorch 0.19.7ad948f</p>
<pre><code>def myNet():
    def __init__():
          #do som init
    def forward():
           #compute  p_rnn_feature,u_sum
           p_rnn_feature = Variable(torch.ones(p_rnn_feature.size())).cuda()
            u_sum = Variable(torch.ones(u_sum.size())).cuda()
            for i in range(self.embedding_length):
                u_s = u_s.clone()
                u_i = u_i.clone()

                for j in range(self.number_person):
                    
                    alpha_i = Variable(torch.zeros(batch_size, self.number_person, 1)).cuda()
                    comp_mask = Variable(j*torch.ones(valid_person_num.size())).cuda()
                    comp_mask = torch.lt(comp_mask, valid_person_num) # (batch_size, 1)
                    comp_mask_ui = comp_mask.repeat(1, self.hyper_d)
                    tmp_x = torch.cat((p_rnn_feature[:,j,:], u_sum[:,j,:], u_s), 1) # size: (batch_size, 2*rnn_cell_size+hyper_d)


                    u_i[:,j,:] = (1 - self.lumbda)*u_i[:,j,:] + self.lumbda*F.relu(self.u_i_linear(tmp_x))
                    u_i[:,j,:] = u_i[:,j,:]*comp_mask_ui.float()

                    alpha_i[:,j,:] = F.tanh(self.alpha_i_linear(torch.cat((u_i[:,j,:], u_s),1)))
                    alpha_i[:,j,:] = alpha_i[:,j,:]*comp_mask.float()

                alpha_sum = torch.sum(alpha_i,1)
                alpha_sum = alpha_sum.repeat(1,self.number_person,1)

                gate = alpha_i / Variable(torch.max(alpha_sum.data, torch.ones(alpha_sum.size()).cuda())).cuda()
                gate = gate.repeat(1,1,self.hyper_d)

                gated_ui_sum = gate*u_i
                gated_ui_sum = torch.sum(gated_ui_sum,1)
                gated_ui_sum = torch.squeeze(gated_ui_sum, dim=1)
                
                tmp_s = torch.cat((u_s, p_feature_sum, gated_ui_sum), 1)   # size: (batch_size, hyper_d+rnn_cell_size+hyper_d)
                u_s = (1 - self.lumbda) * u_s + self.lumbda * F.relu(self.u_s_linear(tmp_s))

            pred_tmp = torch.cat((torch.squeeze(torch.sum(u_i, 1), dim=1), u_s), 1)
            pred = self.pred_dropout(self.pred_linear(pred_tmp))
            pred = self.pred_linear_second(pred)</code></pre>

apaszke: <p>Assignments to Variables are in-place operations and you’re doing a lot of them (<code>u_i[:,j,:]</code>). You’re using that variable in lots of other contexts and some of the functions require it to not change. This might help (I added some calls to <code>clone</code>):</p>
<pre><code class="lang-python">u_i[:,j,:] = (1 - self.lumbda)*u_i[:,j,:].clone() + self.lumbda*F.relu(self.u_i_linear(tmp_x))
u_i[:,j,:] = u_i[:,j,:].clone()*comp_mask_ui.float()

alpha_i[:,j,:] = F.tanh(self.alpha_i_linear(torch.cat((u_i[:,j,:], u_s),1)))
alpha_i[:,j,:] = alpha_i[:,j,:].clone()*comp_mask.float()

</code></pre>

YongyiTang92: <p>Thanks, it’s working. But what do you mean by ‘Assignments to Variables are in-place operations’? So something like x=x+1 is in-place operation? Or just because I am using indexing in a matrix?</p>

apaszke: <p><code>x = x + 1</code> is not in-place, because it takes the objects pointed to by <code>x</code>, creates a new Variable, adds 1 to <code>x</code> putting the result in the new Variable, and overwrites the object referenced by <code>x</code> to point to the new var. There are no in-place modifications, you only change Python references (you can check that <code>id(x)</code> is different before and after that line).</p>
<p>On the other hand, doing <code>x += 1</code> or <code>x[0] = 1</code> will modify the data of the Variable in-place, so that no copy is done. However some functions (in your case <code>*</code>) require the inputs to never change after they compute the output, or they wouldn’t be able to compute the gradient. That’s why an error is raised.</p>

YongyiTang92: <p>Thanks! That’s a great explanation!</p>

AaronYALai: <p>Nice explanation!!<br>
I encountered the same problem and solved it by the explanation!<br>
Thanks</p>

wasiahmad: <p><a class="mention" href="/u/apaszke">@apaszke</a> I am facing a similar problem in my code, is there any way I can find which Variable is causing the problem because of in-place operations in my code? I am kind of stuck at this point, any help would be appreciated.</p>

YongyiTang92: <p>I think you can just try to clone the variable before you use it.</p>

bhattad2: <p>I’m getting same error for a different scenario. I am running a LSTM code and my code works completely fine when I use criterion as MSELoss and it gives me this error when I change my criterion to CrossEntropyLoss (of course I am feeding in desired type of inputs to my criterion). I get this error when I call loss.backward(). Strangely, the code runs perfectly fine when I call loss.backward() at every time step in the time loop instead of calling after the entire sequence has been completed.</p>
<p>Is it possible to have some pointing to a variable at least which is causing this trouble or any other way to reason out the possible error?</p>
<p>Thank you in advance for your help.</p>

tom: <p>Hi,</p>
<p>One thing you can do is check the line numbers in the traceback. If you look up the class the line belongs to in torch/autograd/_functions/*.py, you can tell which operation it bails out on to narrow it down.<br>
I wonder whether it might be worth adding a “debug” modus that records the stack of the op in the forward pass and spits it out on error in the backward. That way, it would point to the right line of code directly.</p>
<p>Best regards</p>
<p>Thomas</p>

Asif_Hossain: <p>x11[:,:,0:int(w_f/2),0:int(h_f/2)]= x11[:,:,0:int(w_f/2),0:int(h_f/2)]*xx1[0]<br>
suppose x11 is a auto grad variable. Now my issue is, when i am running the code it say:</p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation<br>
but if i do x11=x11*xx1[0], then no error things go correctly<br>
i tried:<br>
x11[:,:,0:int(w_f/2),0:int(h_f/2)]= x11[:,:,0:int(w_f/2),0:int(h_f/2)].clone()*xx1[0]<br>
still its not working <a class="mention" href="/u/apaszke">@apaszke</a></p>

chaoyan1073: <p>For pytoch 0.4.0, <code>.clone()</code> does not work. Do you have any idea?</p>
<pre><code class="lang-auto">theta[:, 0] = 1 - theta[:, 0].clone()
</code></pre>

kevin-ssy: <p>Same problem. Have you found the solution?</p>

RyanCV: <p>Hi <a class="mention" href="/u/yongyitang92">@YongyiTang92</a> <a class="mention" href="/u/apaszke">@apaszke</a>, I tried clone in my following code, but still give me the same error as <a class="mention" href="/u/yongyitang92">@YongyiTang92</a> 's. Thanks for your help.</p>
<pre><code class="lang-auto">        phi = x_out[:, :, 0] # [B, C]
        phi.clone()
        Batch = phi.shape[0]
        
        for i in range(Batch):        	
        	phi[i] = phi_constant * phi[i].clone() + phi_offset 
</code></pre>

RyanCV: <p>actual, using a list to append the values in the for loop solved this problem.</p>

cerisara: <p>Thank you for the explanation. It’s just quite weird that x+=1 is in-place while x=x+1 is not.<br>
That makes both operations have a different semantic, while I’m used to consider them as equivalent in python.</p>

gwg: <p>For anyone else brought here by Google, you can use <code>set_detect_anomaly</code> to find the location of the inplace operation. See <a href="https://github.com/pytorch/pytorch/issues/15803" rel="nofollow noopener">this issue</a> I had created on the topic.</p>

zeakey: <p>Hi apaszke,</p>
<p>I encountered this error in a cuda extension.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.autograd import Function
import sys, os
sys.path.insert(0, os.path.abspath("build/lib.linux-x86_64-3.6/"))
import reprelu_cuda, reprelu_cpp

class RepReLUFunction(Function):
    @staticmethod
    def forward(ctx, input, pos_weight, neg_weight):

        pos_weight = pos_weight.contiguous()
        neg_weight = neg_weight.contiguous()

        output = reprelu_cuda.forward(input, pos_weight, neg_weight)[0]
        
        ctx.save_for_backward(input, pos_weight, neg_weight)

        return output

    @staticmethod
    def backward(ctx, grad_output):

        outputs = reprelu_cuda.backward(grad_output.contiguous(), *ctx.saved_variables)

        d_input, d_pos_weight, d_neg_weight = outputs

        return d_input, d_pos_weight, d_neg_weight

class RepReLU(nn.Module):

    def __init__(self, planes, neg_slope=0.25):
        
        super(RepReLU, self).__init__()


    
    def forward(self, x):

        w_pos = torch.nn.several_computations(x)
        w_neg = torch.nn.other_computations(x)

        return RepReLUFunction.apply(x, w_pos, w_neg)
</code></pre>
<p>Where <code>reprelu</code> is a cuda extension strictly follow <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">this tutorial</a>.</p>
<p>When running following test code, the error occurs:</p>
<pre><code class="lang-auto">data = torch.zeros(1, 2, 2, 2).cuda()
reprelu = RepReLU(2).cuda()

loss = reprelu(data).sum()
loss.backward()
</code></pre>
<p>The error messege told me that<br>
<strong>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</strong> in</p>
<pre><code class="lang-auto">outputs = reprelu_cuda.backward(grad_output.contiguous(), *ctx.saved_variables)
</code></pre>
<p>However, <code>reprelu_cuda.backward</code> is implemented in cuda and wrapped in c++, I didn’t make any change to inputs in c++ functions and cuda kernel functions.</p>

dkwasny: <p>Hello, I am experiencing a similar issue, however I do not fully understand the behavior, so I would be grateful for some insight.<br>
When I run the code with these 2 lines of code, it causes the mentioned In-place operation error</p>
<pre><code class="lang-auto">attention = nn.functional.softmax(torch.matmul(key.transpose(1, 2), query) / math.sqrt(num_of_channels), dim=1)
attention[:, 0, 0] = torch.ones(attention.shape[0], requires_grad = True)
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation
</code></pre>
<p>However, if I run this code like this:</p>
<pre><code class="lang-auto">attention = nn.functional.softmax(torch.matmul(key.transpose(1, 2), query), dim=1) / math.sqrt(num_of_channels)
attention[:, 0, 0] = torch.ones(attention.shape[0], requires_grad = True)
</code></pre>
<p>it works perfectly fine.</p>
<p>What is the reason for such behavior? The resulting tensor has the attribute requires_grad set to True in both cases, only the backward function differs. Actually, what is fun is that I can just use a dummy division by 1.0 to avoid getting an error, but it does not seem elegant at all:</p>
<pre><code class="lang-auto"> attention = nn.functional.softmax(torch.matmul(key.transpose(1, 2), query) / math.sqrt(num_of_channels), dim=1)/1.0
attention[:, 0, 0] = torch.ones(attention.shape[0], requires_grad = True)
</code></pre>

Shih-han_Wang: <p>Super useful. It works for me. Thank you so much.</p>