RuntimeError: one of the variables needed for gradient computation has been modified by an inplace

Jake_Eum: <p>try to use sam</p><aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/davda54/sam" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:400/400;"><img src="https://avatars2.githubusercontent.com/u/937184?s=400&amp;amp;v=4" class="thumbnail" width="400" height="400"></div>

<h3><a href="https://github.com/davda54/sam" target="_blank" rel="noopener nofollow ugc">davda54/sam</a></h3>


  <p><span class="label1">SAM: Sharpness-Aware Minimization (PyTorch). Contribute to davda54/sam development by creating an account on GitHub.</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>torchversion: 1.7.0+cu110<br>
ubuntu : 18.04</p>
<p><code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1280, 5]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</code></p>
<p>following issue with code</p>
<pre><code class="lang-auto">def train_model_sam(model, epoch,train_loader,val_loader,optimizer,criterion,scheduler):
    model.train() 
    
    losses = AverageMeter()
    accs = AverageMeter()
    
    tk = tqdm(train_loader, total=len(train_loader), position=0, leave=True)
    for idx, (imgs, labels) in enumerate(tk):
        imgs_train, labels_train = imgs.cuda(), labels.cuda().long()
        output_train = model(imgs_train)

        #loss = criterion(output_train, labels_train)

        # first forward-backward pass
        optimizer.zero_grad() 
        loss = criterion(output_train, labels_train)  # use this loss for any training statistics
        loss.backward(retain_graph = True)
        optimizer.first_step(zero_grad=True)
        
        # second forward-backward pass
        optimizer.zero_grad() 
        criterion(output_train, labels_train).backward()
        optimizer.second_step(zero_grad=True)

        
        #optimizer.zero_grad() 
        #loss.backward()
        #optimizer.step() 
        
        accs.update((output_train.argmax(1)==labels_train).sum().item()/imgs_train.size(0),imgs_train.size(0))
        losses.update(loss.item(), imgs_train.size(0))

        tk.set_postfix(loss=losses.avg,acc=accs.avg)
        
    return losses.avg
</code></pre>

albanD: <p>Hi,</p>
<p>What is most likely happening is that your first optimizer step is actually changing some of the weights inplace. And so the second backward cannot run because it needs the original value of these weights.<br>
You should either delay the first step after all the backward have been done. Or re-do the forward for the weights that have been changed before doing the second backward.</p>

Jake_Eum: <p>THank you albabD<br>
but did you visit the github i provide?<br>
I try to use the sam optimizer from what they provide<br>
so can you suggest some code modification?</p>
<p>I changed it like this but I guess this is not what they are asking</p>
<blockquote>
<p>def train_model_sam(model, epoch,train_loader,val_loader,optimizer,criterion,scheduler):<br>
model.train()</p>
<pre><code>losses = AverageMeter()
accs = AverageMeter()

tk = tqdm(train_loader, total=len(train_loader), position=0, leave=True)
for idx, (imgs, labels) in enumerate(tk):
    imgs_train, labels_train = imgs.cuda(), labels.cuda().long()
    output_train = model(imgs_train)

    # first forward-backward pass
    #optimizer.zero_grad() 
    loss = criterion(output_train, labels_train)  # use this loss for any training statistics
    loss.backward(retain_graph = True)
    optimizer.first_step(zero_grad=True)
    

    # second forward-backward pass
    #optimizer.zero_grad() 
    output_train = model(imgs_train)
    loss=criterion(output_train, labels_train)
    loss.backward()
    optimizer.second_step(zero_grad=True)

    #loss = criterion(output_train, labels_train)
    #optimizer.zero_grad() 
    #loss.backward()
    #optimizer.step() 
    
    accs.update((output_train.argmax(1)==labels_train).sum().item()/imgs_train.size(0),imgs_train.size(0))
    losses.update(loss.item(), imgs_train.size(0))

    tk.set_postfix(loss=losses.avg,acc=accs.avg)
    
return losses.avg
</code></pre>
</blockquote>

albanD: <aside class="quote no-group" data-username="Jake_Eum" data-post="3" data-topic="108806">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jake_eum/48/30524_2.png" class="avatar"> Jake_Eum:</div>
<blockquote>
<p>but did you visit the github i provide?</p>
</blockquote>
</aside>
<p>I did not, but the error is in the code sample you shared <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<blockquote>
<p>I try to use the sam optimizer from what they provide</p>
</blockquote>
<p>Note that these checks were buggy in old versions of PyTorch and fixed recently. So if this repo has not been updated in a while, that might be the reason why they share this code but it doesn’t work.</p>
<p>Your updated code should run fine no?</p>

Jake_Eum: <aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/davda54/sam" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/937184?s=400&amp;amp;v=4" class="thumbnail onebox-full-image" width="60" height="60">

<h3><a href="https://github.com/davda54/sam" target="_blank" rel="noopener nofollow ugc">davda54/sam</a></h3>


  <p><span class="label1">SAM: Sharpness-Aware Minimization (PyTorch). Contribute to davda54/sam development by creating an account on GitHub.</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>