Problem: one of the variables needed for gradient computation has been modified by an inplace operation

ProjectUser2021: <p>Hello! I hope it’s ok I’m asking, but I’m having troubles with my nn code. It seems as if the loss dosen’t decrease and when running in debug mode I encounter this problem:</p>
<p>Exception has occurred: RuntimeError<br>
one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [10, 1]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!<br>
File “…/FinalProject/trip.py”, line 167, in <br>
loss.backward(retain_graph=True)</p>
<p>I’ve been searching and reading online answers and tried many things but can’t figure this out. I’d love to get your help.</p>
<p>The whole traceback:<br>
[W python_anomaly_mode.cpp:104] Warning: Error detected in AddmmBackward. Traceback of forward call that caused the error:<br>
File “…/anaconda3/lib/python3.8/runpy.py”, line 194, in _run_module_as_main<br>
return _run_code(code, main_globals, None,<br>
File “…/anaconda3/lib/python3.8/runpy.py”, line 87, in _run_code<br>
exec(code, run_globals)<br>
File “…/.vscode-server/extensions/ms-python.python-2021.9.1191016588/pythonFiles/lib/python/debugpy/<strong>main</strong>.py”, line 45, in <br>
cli.main()<br>
File “/…/.vscode-server/extensions/ms-python.python-2021.9.1191016588/pythonFiles/lib/python/debugpy/…/debugpy/server/cli.py”, line 444, in main<br>
run()<br>
File “…/.vscode-server/extensions/ms-python.python-2021.9.1191016588/pythonFiles/lib/python/debugpy/…/debugpy/server/cli.py”, line 285, in run_file<br>
runpy.run_path(target_as_str, run_name=compat.force_str(“<strong>main</strong>”))<br>
File “/…/anaconda3/lib/python3.8/runpy.py”, line 265, in run_path<br>
return _run_module_code(code, init_globals, run_name,<br>
File “/…/anaconda3/lib/python3.8/runpy.py”, line 97, in _run_module_code<br>
_run_code(code, mod_globals, init_globals,<br>
File “/…/anaconda3/lib/python3.8/runpy.py”, line 87, in _run_code<br>
exec(code, run_globals)<br>
File “…/PycharmProjects/FinalProject/trip.py”, line 156, in <br>
out_q,out_p,out_n = model(query,pos,neg) # triplets and anchors into the nn<br>
File “/…/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 889, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/…/PycharmProjects/FinalProject/trip.py”, line 94, in forward<br>
out_n = <a href="http://self.net" rel="noopener nofollow ugc">self.net</a>(neg)<br>
File “/…/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 889, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/…/anaconda3/lib/python3.8/site-packages/torch/nn/modules/container.py”, line 119, in forward<br>
input = module(input)<br>
File “/…/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 889, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/…/anaconda3/lib/python3.8/site-packages/torch/nn/modules/linear.py”, line 94, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
File “/…/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py”, line 1753, in linear<br>
return torch._C._nn.linear(input, weight, bias)<br>
(function _print_stack)</p>
<p>Code attached here: It’s supposed to train on triplets I’ve generated and calculate triplet loss.</p>
<h1>
<a name="heading-1" class="anchor" href="#heading-1"></a>==================================</h1>
<h1>
<a name="neural-network-2" class="anchor" href="#neural-network-2"></a>Neural Network</h1>
<h1>
<a name="heading-3" class="anchor" href="#heading-3"></a>==================================</h1>
<h1>
<a name="define-the-network-class-4" class="anchor" href="#define-the-network-class-4"></a>Define the Network Class</h1>
<h1>
<a name="heading-5" class="anchor" href="#heading-5"></a>=========================</h1>
<p>class MyNetwork(nn.Module):</p>
<pre><code>def __init__(self):

    # call constructor from superclass

    super().__init__()

    # define network layers

    self.net = nn.Sequential(

        # Hidden Layer 1

        nn.Linear(num_of_features, 100),

        nn.ReLU(),

        # Hidden Layer 2

        nn.Linear(100, 10),

        nn.ReLU(),

        # Output Layer

        nn.Linear(10, 1)

    )

def forward(self, query, pos, neg):

    out_q = self.net(query)

    out_p = self.net(pos)

    out_n = self.net(neg)

    out_q = torch.clone(out_q)

    out_p = torch.clone(out_p)

    out_n = torch.clone(out_n)

    return out_q,out_p,out_n
</code></pre>
<h1>
<a name="instantiate-the-model-and-send-to-cuda-device-6" class="anchor" href="#instantiate-the-model-and-send-to-cuda-device-6"></a>Instantiate the model and send to cuda device:</h1>
<h1>
<a name="heading-7" class="anchor" href="#heading-7"></a>---------------------------------------------</h1>
<p>model = MyNetwork()</p>
<p>model.to(device)</p>
<h1>
<a name="loss-criterion-8" class="anchor" href="#loss-criterion-8"></a>Loss Criterion</h1>
<h1>
<a name="heading-9" class="anchor" href="#heading-9"></a>================</h1>
<p>triplet_loss_fn = nn.TripletMarginLoss(margin=1.0, p=2)</p>
<h1>
<a name="optimization-10" class="anchor" href="#optimization-10"></a>Optimization</h1>
<h1>
<a name="heading-11" class="anchor" href="#heading-11"></a>=============</h1>
<h1>
<a name="optimizer-type-12" class="anchor" href="#optimizer-type-12"></a>optimizer type:</h1>
<h1>
<a name="heading-13" class="anchor" href="#heading-13"></a>---------------</h1>
<p>optimizer = optim.SGD(model.parameters(), lr=lr, momentum=momentum, weight_decay=weight_decay)#,nesterov=True)</p>
<h1>
<a name="heading-14" class="anchor" href="#heading-14"></a>=========================</h1>
<h1>
<a name="train-phase-15" class="anchor" href="#train-phase-15"></a>Train phase</h1>
<h1>
<a name="heading-16" class="anchor" href="#heading-16"></a>=========================</h1>
<p>params = list(model.parameters())</p>
<p>print(len(params))</p>
<p>train_loader = DataLoader(train_data, batch_size=batch_size,shuffle=True)</p>
<p>val_loader = DataLoader(val_data, batch_size=batch_size,shuffle=True)</p>
<p>train_losses, train_accuracy, val_losses, val_accuracy = ([] for i in range(4))  # create four empty lists</p>
<p>model.train()</p>
<p>triplet_train_losses=[]</p>
<p>epochs=20</p>
<p>print(“Starting Train Loop”)</p>
<p>for epoch in range(epochs):  # loop over the dataset multiple times</p>
<pre><code>loss = 0.0

batches_acc = 0

triplets_num = 0

loss_tri = 0

loss_phy = 0

# iterate over the data

for batch_idx, data in enumerate(train_loader):

   # data = data.to(device)   # move data to the GPU (when using a GPU) 

    #features = (data[:,:,:-1].float()).to(device)

    features = torch.clone(data[:,:,:-1]).float().to(device)

    labels = torch.clone(data[:,0,-1]).to(device) 
    query = torch.clone(features[:,0,:]).to(device)
    pos = torch.clone(features[:,1,:]).to(device)
    neg = torch.clone(features[:,2,:]).to(device)

    

    # Forward pass:

    out_q,out_p,out_n = model(query,pos,neg) # triplets and anchors into the nn

    out_q = torch.clone(out_q).to(device)

    out_p = torch.clone(out_p).to(device)

    out_n = torch.clone(out_n).to(device)

    # compute loss

    loss = torch.clone(loss + triplet_loss_fn(out_q,out_p,out_n))

    # zero the parameter gradients

    optimizer.zero_grad()

    loss.backward(retain_graph=True)

    optimizer.step()

    params = list(model.parameters())   

    grads0=(list(model.parameters())[0].grad) 

    grads1=(list(model.parameters())[1].grad) 

    grads2=(list(model.parameters())[2].grad) 

    grads3=(list(model.parameters())[3].grad) 

    grads4=(list(model.parameters())[4].grad) 

    grads5=(list(model.parameters())[5].grad) 

    

# Normalizing the loss by the total number of train batches

num_batches = len(train_loader)

train_losses.append(loss/num_batches)


print("Epoch: {0} |loss: {1}% |".format(epoch+1,  train_losses[-1]))
</code></pre>
<p>Thank you very much!!!</p>

ptrblck: <p>Are you only seeing this error in debug mode or also if you just execute your script?</p>

ProjectUser2021: <p>Hi, first of all thank you for replying!<br>
Actually in this code I get the error even when executing it (no debug). It’s a part of a bigger code with combined loss of 2 loss function, one of them is the triplet loss. In the other code I get this kind of error only in debug after one batch…</p>

ptrblck: <p>In the posted code snippet you are using <code>loss.backward(retain_graph=True)</code>. Could you explain why this is necessary for your use case, as it’s often applied as a workaround for other issues and could yield the disallowed inplace modification instead.</p>

ProjectUser2021: <p>Well, I used it as if it could help solving the problem, I really didn’t know what to do since it’s the first time I’ve encounterd this error, and tried many things I saw in other posts with this kind of error. But even if I remove it, the error still exits…</p>

hatala91: <p>Probably a bit late but hope this still helps someone <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>In the forward function instead of first passing the query, pos and neg into the model and then cloning the tensors I would try and pass clones of the tensors in:</p>
<pre><code class="lang-auto">out_q = self.net(query.clone())
</code></pre>