RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 6]]

Raphael_Emeka: <p>Hi<br>
I encountered this problem and haven’t been able to sort it out for a long while now. Everything works fine until the backward propagation.<br>
it gives me the error message:<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 6]], which is output 0 of SliceBackward, is at version 30; expected version 27 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
<p>Heres the code:</p>
<pre><code class="lang-auto">#running [1,6] vector inputs to 3 NN models
torch.autograd.set_detect_anomaly(True)
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)
#training
k=0
TrainData0=Mynodein[0,0:1,:].clone().detach().requires_grad_(True)
TrainData1=Mynodein[0,1:2,:].clone().detach().requires_grad_(True)
TrainData2=Mynodein[0,2:3,:].clone().detach().requires_grad_(True)

for epoch in range(50):
    
    epoch_lossN0 = 0
    epoch_lossN1 = 0
    epoch_lossN2 = 0
    lgcount=0
    #for i in range(k*Tbatch,k*Tbatch+Tbatch):
    for i in range(1):
        N0_weight = model.T0_forward(TrainData0)
        N1_weight = model.T1_forward(TrainData1)
        N2_weight = model.T2_forward(TrainData2)
        
        diff0=t[:,i]+TimeOfFlight_Matrix[0,:]-t[0,i]
        #diff0=t[:,time_idx].clone()-t[0,time_idx].clone()  #without propagation delay
        diff0_cat=torch.cat([diff0[0:0], diff0[0+1:]])                                                                                                                            
        prod0=torch.sum(diff0_cat* N0_weight)
        t0=t[0,i]+Tnode[0] + prod0  #evaluates the next timestamp for node 0, i.e., t[0,i+1]
        t[0,i+1]=t0
    
        diff1=t[:,i]+TimeOfFlight_Matrix[1,:]-t[1,i]
        #diff1=t[:,time_idx].clone()-t[1,time_idx].clone()  #without propagation delay
        diff1_cat=torch.cat([diff1[0:1], diff1[1+1:]])
        prod1=torch.sum(diff1_cat* N1_weight)
        t1=t[1,i]+Tnode[1] + prod1  #evaluates the next timestamp for node 1, i.e., t[1,i+1]
        t[1,i+1]=t1
    
        diff2=t[:,i]+TimeOfFlight_Matrix[2,:]-t[2,i]
        #diff2=t[:,time_idx].clone()-t[e,time_idx].clone()  #without propagation delay
        diff2_cat=torch.cat([diff2[0:2], diff2[2+1:]])
        prod2=torch.sum(diff2_cat* N2_weight)
        t2=t[2,i]+Tnode[2] + prod2  #evaluates the next timestamp for node 2, i.e., t[2,i+1]
        t[2,i+1]=t2
        
        timevec=[t0,t1,t2]
           
        lgtensor=torch.tensor(2+lgcount)
        
        local_lossN0= torch.log(lgtensor)*lossfn0(timevec)
        local_lossN1= torch.log(lgtensor)*lossfn1(timevec)
        local_lossN2= torch.log(lgtensor)*lossfn2(timevec)
        
     
    optimizer.zero_grad()
    local_lossN0.backward(retain_graph=True)
    local_lossN1.backward(retain_graph=True)
    local_lossN2.backward(retain_graph=True)
        
    optimizer.step()
</code></pre>
<p>I don’t know what I am doing wrong</p>

ptrblck: <p>I don’t know where tensor <code>t</code> is coming from but based on your code I would guess that these inplace operations are causing the issue:</p>
<pre><code class="lang-python">t[0,i+1]=t0
t[1,i+1]=t1
t[2,i+1]=t2
</code></pre>
<p>Replace these calls with:</p>
<pre><code class="lang-python">t = torch.stack((t0, t1, t2))
</code></pre>
<p>(or <code>torch.cat</code> depending on the shape of the tensors) and it might work.</p>
<p>Also, recreating a new tensor will detach it from the computation graph, so replace:</p>
<pre><code class="lang-python">lgtensor=torch.tensor(2+lgcount)
</code></pre>
<p>with:</p>
<pre><code class="lang-python">lgtensor = 2+lgcount
</code></pre>
<p>assuming <code>lgcount</code> is a tensor attached to a computation graph.</p>

Raphael_Emeka: <p>Thanks for your help. I was able to fix the issue. you were right it was with the leaf tensor t.<br>
Recreating the tensor at each epoch detached it from the computation graph. I have been able to get it working now.</p>
<p>On another note, I am obtaining the loss curve and it appears to be declining as expected, however, curve does not seem to converge, here is a figure of the learning curve.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea5d1ff496b6bf51979d34e50f8dec8558460176.png" data-download-href="https://discuss.pytorch.org/uploads/default/ea5d1ff496b6bf51979d34e50f8dec8558460176" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea5d1ff496b6bf51979d34e50f8dec8558460176_2_690x469.png" alt="image" data-base62-sha1="xrhfjRSTENLilGNssTIWAjmEiuG" width="690" height="469" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea5d1ff496b6bf51979d34e50f8dec8558460176_2_690x469.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea5d1ff496b6bf51979d34e50f8dec8558460176_2_1035x703.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea5d1ff496b6bf51979d34e50f8dec8558460176_2_1380x938.png 2x" data-dominant-color="FBFBFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">3067×2086 115 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Do you have an suggestions?</p>

ptrblck: <p>I don’t know which loss function you are using and what the expected ranges are. Is a start loss of <code>~4e-5</code> expected, which seems to be quite low?</p>

Raphael_Emeka: <p>hi I am using an MSE loss, and I was expecting the start loss to be around e-4 to e-6</p>

Raphael_Emeka: <p>Hi, thanks, i think it’s sorted now, I just needed to adjust the learning rate and I added momentum to aid it to converge faster.</p>
<p>Here is the new result<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/5/95c8494191af21b8f36fde424e78c89f8baf60cf.png" data-download-href="https://discuss.pytorch.org/uploads/default/95c8494191af21b8f36fde424e78c89f8baf60cf" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/95c8494191af21b8f36fde424e78c89f8baf60cf_2_690x474.png" alt="image" data-base62-sha1="ln2fLb3pxKOA3qCBf0IVPflR3bh" width="690" height="474" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/95c8494191af21b8f36fde424e78c89f8baf60cf_2_690x474.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/95c8494191af21b8f36fde424e78c89f8baf60cf_2_1035x711.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/95c8494191af21b8f36fde424e78c89f8baf60cf_2_1380x948.png 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">3031×2086 95.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>