RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation (Meta Learning)

hduc-le: <p>Hi everybody!<br>
I’m trying to implement an algorithm for MAML model (meta learning). The idea of my code is trying to train a LM model like BERT with different sets of data in a few training steps and optimize the parameters over those sets of data. Finally, the model parameters will be updated again over the sum of losses, computed in the last training steps. However, my implementation has a bug as below. I did google this bug. I think it might has something wrong while performing backward pass for the sum of losses, but I have not solved it so far. Can you guys help me solve it. Thank you in advance!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7cca900f59325f4c537bf62ea1a11b3a0bdab50.png" data-download-href="https://discuss.pytorch.org/uploads/default/b7cca900f59325f4c537bf62ea1a11b3a0bdab50" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7cca900f59325f4c537bf62ea1a11b3a0bdab50_2_690x236.png" alt="image" data-base62-sha1="qdXRhRAzownzEA5CfA5gD4j4iUE" width="690" height="236" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7cca900f59325f4c537bf62ea1a11b3a0bdab50_2_690x236.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7cca900f59325f4c537bf62ea1a11b3a0bdab50_2_1035x354.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7cca900f59325f4c537bf62ea1a11b3a0bdab50_2_1380x472.png 2x" data-dominant-color="F7F6F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1458×499 78.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto"># Model definition
class BertClassifier(nn.Module):
    def __init__(self, pretrain_path, num_labels): 
        nn.Module.__init__(self)
        self.bert = AutoModel.from_pretrained(pretrain_path)
        self.linear_out = nn.Linear(self.bert.config.hidden_size, num_labels)

    def forward(self, input_ids, attention_mask):
        output = self.bert(input_ids=input_ids, attention_mask=attention_mask)
        output = self.linear_out(output.last_hidden_state[:,0,:])
        return output


# Training configuration
num_epochs = 2
MAX_ITER = 5  
optim1 = torch.optim.AdamW(model.parameters(), lr=1e-3)
optim2 = torch.optim.AdamW(model.parameters(), lr=1e-2)
loss_fn = nn.CrossEntropyLoss()


# Training loop
for epoch in range(num_epochs):

    loss_list = []
    num_iter = 0
    model.train()
    optim2.zero_grad()
    
    while num_iter &lt; MAX_ITER:
        optim1.zero_grad()
        batch_supp, _, _ = next(train_data_loader)
        supp_labels = batch_supp['label'].to(device)
        supp_ids = batch_supp['ids'].to(device)
        supp_mask = batch_supp['mask'].to(device)

        #forward pass
        logits = model(input_ids=supp_ids, attention_mask=supp_mask)

        #compute loss
        loss = loss_fn(logits, supp_labels)
        loss.backward(retain_graph=True)
        loss_list.append(loss)
        optim1.step()
        
        num_iter += 1
        print("&gt;&gt; Loss at iter {}: {}".format(num_iter, loss.item()))

    
    total_loss = sum(loss_list)
    total_loss.backward()
    optim2.step()
    print("&gt;&gt; Overall loss: {}".format(total_loss.item()))
</code></pre>

srishti-git1110: <aside class="quote no-group" data-username="hduc-le" data-post="1" data-topic="161783">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hduc-le/48/53370_2.png" class="avatar"> hduc-le:</div>
<blockquote>
<p><code>loss.backward(retain_graph=True)</code></p>
</blockquote>
</aside>
<p>Hi, why have you used <code>retain_graph=True</code> in this backward call? Any specific reason to do that explicitly?</p>
<p>If not, please remove this and re-run the code and let me know if you still face the error.</p>

hduc-le: <p>Hi <a class="mention" href="/u/srishti-git1110">@srishti-git1110</a> , if we do <code>loss.backward()</code> at each training step and then perform <code>total_loss.backward()</code> without setting <code>retain_graph=True</code> in previous, the error message will pop up. I think because the <code>total_loss</code> is computed as the sum over all <code>loss</code> in <code>list_lost</code>. You can find more details at this link:</p><aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/46774641/what-does-the-parameter-retain-graph-mean-in-the-variables-backward-method">
  <header class="source">

      <a href="https://stackoverflow.com/questions/46774641/what-does-the-parameter-retain-graph-mean-in-the-variables-backward-method" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/1646823/jvans" target="_blank" rel="noopener nofollow ugc">
    <img alt="jvans" src="https://www.gravatar.com/avatar/8471ef6adbd84694d3c37e445768a75f?s=256&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/46774641/what-does-the-parameter-retain-graph-mean-in-the-variables-backward-method" target="_blank" rel="noopener nofollow ugc">What does the parameter retain_graph mean in the Variable's backward() method?</a>
</h4>

<div class="tags">
  <strong>neural-network, conv-neural-network, backpropagation, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/1646823/jvans" target="_blank" rel="noopener nofollow ugc">
    jvans
  </a>
  on <a href="https://stackoverflow.com/questions/46774641/what-does-the-parameter-retain-graph-mean-in-the-variables-backward-method" target="_blank" rel="noopener nofollow ugc">04:11PM - 16 Oct 17 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
By the way, I’ll give you the error message if I do not set the parameter <code>retain_graph</code> to <code>True</code> in <code>loss.backward()</code><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0fb611b42effd8147afa9b7ebf8c132d2de7f9f6.png" data-download-href="https://discuss.pytorch.org/uploads/default/0fb611b42effd8147afa9b7ebf8c132d2de7f9f6" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0fb611b42effd8147afa9b7ebf8c132d2de7f9f6_2_690x151.png" alt="image" data-base62-sha1="2eZf5IFyJY0cYMITPJsZGy8SaSG" width="690" height="151" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0fb611b42effd8147afa9b7ebf8c132d2de7f9f6_2_690x151.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0fb611b42effd8147afa9b7ebf8c132d2de7f9f6_2_1035x226.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0fb611b42effd8147afa9b7ebf8c132d2de7f9f6_2_1380x302.png 2x" data-dominant-color="F7F6F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1648×363 56.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

srishti-git1110: <p><a class="mention" href="/u/hduc-le">@hduc-le</a> You are right, I’m sorry. I just saw <code>retain_graph=True</code> and was tempted to point it out as it generally becomes a source of error.</p>
<p>While for your code, I’m not able to think exactly where the inplace modification is taking place.</p>
<p>But, could you please try another approach by initialising <code>total_loss=0.0</code> to be a float variable instead of a list.</p>
<p>And instead of the appending step, use:</p>
<pre><code class="lang-auto">total_loss = total_loss + loss
</code></pre>
<p>I’ve simulated this with a simple example on my end and its working fine. Let me know if you still face the error, please.</p>
<p>My simulation:</p>
<pre><code class="lang-auto">import torch
z = torch.tensor([5.0], requires_grad=True) # simulates the model parameters 
optim1 = torch.optim.AdamW([z], lr=1e-3)
optim2 = torch.optim.AdamW([z], lr=1e-2)
num_epochs = 2
MAX_ITER = 5 
for epoch in range(num_epochs):
    total_loss = 0.0
    num_iter = 0
    optim2.zero_grad()
    while num_iter &lt; MAX_ITER:
        optim1.zero_grad()
        loss = z*0.1
        total_loss = total_loss + loss
        loss.backward(retain_graph=True)
        optim1.step()
        num_iter += 1
        print("&gt;&gt; Loss at iter {}: {}".format(num_iter, loss.item()))
    total_loss.backward()
    optim2.step()
    print("&gt;&gt; Overall loss: {}".format(total_loss.item()))
</code></pre>
<p>out -</p>
<pre><code class="lang-auto">&gt;&gt; Loss at iter 1: 0.5
&gt;&gt; Loss at iter 2: 0.49989500641822815
&gt;&gt; Loss at iter 3: 0.4997900128364563
&gt;&gt; Loss at iter 4: 0.49968501925468445
&gt;&gt; Loss at iter 5: 0.4995799958705902
&gt;&gt; Overall loss: 2.4989500045776367
&gt;&gt; Loss at iter 1: 0.4984250068664551
&gt;&gt; Loss at iter 2: 0.4983200132846832
&gt;&gt; Loss at iter 3: 0.4982150197029114
&gt;&gt; Loss at iter 4: 0.4981100261211395
&gt;&gt; Loss at iter 5: 0.4980050027370453
&gt;&gt; Overall loss: 2.491075038909912
</code></pre>
<p>I also tried this same simulation with the list approach, and the simulation works fine for me with that, too. In your screenshot in the first post, it shows <code>sum(losses)</code> while it should be <code>sum(loss_list)</code> according to your code.</p>

srishti-git1110: <p>Hi, I have just edited my previous reply.<br>
Please see that.</p>

hduc-le: <p>Firstly, I very much appreciate your help. I’m sorry for my mistake, in the first post, I took the screenshot before changing the variable name from <code>losses</code> to <code>loss_list</code>. So it’s fine with my code for <code>sum(loss_list)</code>. But the list approach does not solve my problem.</p>
<p>I’ve tried your suggestion that uses <code>total_loss = total_loss + loss</code> and unfortunately, it still gets the same error. Do you mind if I give you the link to the notebook that I’m working on for more details? Thanks</p>
<p>Sorry, my reply has been annotated as spam, so I put the link to my notebook here, thanks: <a href="https://colab.research.google.com/drive/12_O6L1QGD9rRJm_axsEs_0nf4U_Uexkr?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>

srishti-git1110: <p>Sure, please link your notebook.</p>
<p>I see. If you are still getting the error, the simulation didn’t exactly replicate the situation. There might be more nuances to it; I’ll see in your notebook.</p>

hduc-le: <p>Again, I really appreciate your consideration. Here is the link: <a href="https://colab.research.google.com/drive/12_O6L1QGD9rRJm_axsEs_0nf4U_Uexkr?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a> that I’m working. If you figure out any mistake in my implementation, please let me know!</p>

hduc-le: <p>Hi <a class="mention" href="/u/alband">@albanD</a>, I was stuck with this problem for a day. I did a google search and saw your comment at <a href="https://discuss.pytorch.org/t/solved-pytorch1-5-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation/90256/2" class="inline-onebox">[Solved][Pytorch1.5] RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation - #2 by albanD</a>. But in my scenario, it has a little bit different, i.e:<br>
I need to optimize the model parameters over a few iterations by cross-entropy loss and optimizer <code>optim1</code>, and finally, take the sum of those losses computed over the iterations to update the model parameters for the last time with optimizer <code>optim2</code>. Unfortunately, my code has an error as in my description above. Hope you help me solve it and I appreciate that. Thank you in advance!</p>

albanD: <p>have you looked at the improved stack trace after enabling anomaly_mode as suggested in the error?<br>
That should point you to the part that does the inplace you want to remove.<br>
Note that if you can’t remove the inplace, you can also clone() the Tensor so that the one modified inplace is not the same one as the one you care about.</p>
<p>cc <a class="mention" href="/u/soulitzer">@soulitzer</a> maybe your new mode could help here?</p>

soulitzer: <p>Hi <a class="mention" href="/u/hduc-le">@hduc-le</a>, there is a new context manager <a href="https://gist.github.com/soulitzer/91fd975b60b90209aab9c7be634312dc" class="inline-onebox" rel="noopener nofollow ugc">allow_mutation_on_saved_mode.py · GitHub</a> is designed to get rid of these “variables needed for gradient computation has been modified” errors. I have tried it on your colab and it no longer runs into the error. Let me know how it works for you.</p>
<p>The usage is</p>
<pre><code class="lang-auto">with allow_mutation_on_saved_tensors():
  # all the code in your training loop
  ...
</code></pre>

ajwitty: <p>Have you been able to figure it out? I am having a similar issue as well.</p>

ajwitty: <p>Hello,</p>
<p>I am attempting to implement your code, but do not seem to have the TorchDispatchMode function available on Torch 1.11, is there a certain version of torch I need to run this program?</p>

hduc-le: <p>Hi <a class="mention" href="/u/soulitzer">@soulitzer</a>, I’ve tried your suggestion, and fortunately it works for me. I appreciate your help, many thanks sir</p>

hduc-le: <p>Hi <a class="mention" href="/u/ajwitty">@ajwitty</a>, you can try <a class="mention" href="/u/soulitzer">@soulitzer</a> 's solution above, it worked for me</p>

ajwitty: <p>Which version of torch are you running? I do not seem to have one of the necessary functions.</p>

hduc-le: <p>My code was done in GoogleColab with torch version: 1.12.1+cu113</p>

soulitzer: <p>Sorry for the late reply. Yes, 1.12 is necessary to use the context manager unfortunately. If you end up upgrading, let me know if it works for you or if you run into any issues I can help with.</p>

ajwitty: <p><a class="mention" href="/u/soulitzer">@soulitzer</a> I upgraded to 1.13 but was running into an issue where my gpu memory was increasing with each iteration. I’m not sure if it has anything to do with your code, but it is happening within the loop under which the code is wrapped.</p>

soulitzer: <p><a class="mention" href="/u/ajwitty">@ajwitty</a> thanks for the feedback. Do you have a snippet where this happens that you can post here?</p>
<p>(I would not know what is happening in your code without taking a look, but it may be expected for memory to increase if the backward graph is kept alive over multiple iterations as to backprop through it again for the outer step)</p>