Trying to debug RuntimeError: Trying to backward through the graph a second time

username: Aladdine_Ben_Zekri
cooked: <p>Hello, so I am trying to implement an architecture from the Cam2BEV paper (<a href="https://arxiv.org/pdf/2005.04078.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2005.04078.pdf</a>) and I’ve been having this error for days and I haven’t been able to figure out its origin:</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed).
</code></pre>
<p>As a brief description, the input is a list of segmented images all of the same size and the target is also a segmented image of the same size. The issue seems to be with the way I wrote different parts of my model and not with the training loop, however here is a snippet of the training code:</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    model.train()
    
    running_loss = 0.0
    running_acc = 0.0
    for images, labels in train_loader:

        labels = labels.max(dim=1)[1]
        
        optimizer.zero_grad()
        
        outputs = model(images)
        loss = criterion(outputs, labels)
        
        loss.backward()
        optimizer.step()
        
        running_loss += loss.item()
        batch_acc = metric(outputs.detach(), labels.detach()).item()
        running_acc += batch_acc
        
    train_loss = running_loss/len(train_loader)
    train_acc = running_acc/len(train_loader)
    
    print(f"Epoch {epoch+1}/{num_epochs} - Training Loss: {train_loss:.4f} - Training Accuracy: {train_accuracy:.4f}")
</code></pre>
<p>The models can be found in this link: <a href="https://github.com/AlaaBenZekri/Cam2EBV/blob/main/models.py" class="inline-onebox" rel="noopener nofollow ugc">Cam2EBV/models.py at main · AlaaBenZekri/Cam2EBV · GitHub</a></p>
<p>Here you can find a more detailed message error:</p>
<pre><code class="lang-auto">C:\Users\alaae\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\autograd\__init__.py:200: UserWarning: Error detected in CudnnBatchNormBackward0. Traceback of forward call that caused the error:
  File "C:\Users\alaae\Desktop\PFE\Code\train.py", line 58, in &lt;module&gt;
    outputs = model(images)
  File "C:\Users\alaae\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\alaae\Desktop\PFE\Code\models.py", line 211, in forward
    joiner_output = self.joiner(encoder_outputs)
  File "C:\Users\alaae\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\alaae\Desktop\PFE\Code\models.py", line 148, in forward
    t = self.joiner_layers[d](t)
  File "C:\Users\alaae\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\alaae\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\container.py", line 217, in forward
    input = module(input)
  File "C:\Users\alaae\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\alaae\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\batchnorm.py", line 171, in forward
    return F.batch_norm(
  File "C:\Users\alaae\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\functional.py", line 2450, in batch_norm
    return torch.batch_norm(
 (Triggered internally at ..\torch\csrc\autograd\python_anomaly_mode.cpp:119.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
Traceback (most recent call last):
  File "C:\Users\alaae\Desktop\PFE\Code\train.py", line 61, in &lt;module&gt;
    loss.backward()
  File "C:\Users\alaae\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\_tensor.py", line 487, in backward
    torch.autograd.backward(
  File "C:\Users\alaae\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\autograd\__init__.py", line 200, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.
</code></pre>
<p>I’ve been stuck at this issue for a week and I would really appreciate any kind of help.</p>

username: eqy
cooked: <p>I’m not familiar with the model architecture, but my guess is that you would want to clear<br>
<code>self.joiner_outputs</code> before the loop here:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/AlaaBenZekri/Cam2EBV/blob/eead9e9297eac9d7af614228ac4797a386b6079a/models.py#L140">
  <header class="source">

      <a href="https://github.com/AlaaBenZekri/Cam2EBV/blob/eead9e9297eac9d7af614228ac4797a386b6079a/models.py#L140" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/AlaaBenZekri/Cam2EBV/blob/eead9e9297eac9d7af614228ac4797a386b6079a/models.py#L140" target="_blank" rel="noopener nofollow ugc">AlaaBenZekri/Cam2EBV/blob/eead9e9297eac9d7af614228ac4797a386b6079a/models.py#L140</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="130" style="counter-reset: li-counter 129 ;">
          <li>        layer.append(nn.Conv2d(filters*n_inputs, filters, kernel_size=3, padding = 1))
</li>
          <li>        layer.append(nn.ReLU(True))
</li>
          <li>        layer.append(nn.BatchNorm2d(filters))
</li>
          <li>        layer.append(nn.Conv2d(filters, filters, kernel_size=3, padding = 1))
</li>
          <li>        layer.append(nn.ReLU(True))
</li>
          <li>        layer.append(nn.BatchNorm2d(filters))
</li>
          <li>        self.joiner_layers.append(nn.Sequential(*layer))
</li>
          <li>
</li>
          <li>def forward(self, encoder_outputs):
</li>
          <li>    udepth = len(encoder_outputs[0])
</li>
          <li class="selected">
</li>
          <li>    for d in range(udepth):
</li>
          <li>        filters = (2**d)*self.filters1
</li>
          <li>        warped_maps = []
</li>
          <li>        for i in range(self.n_inputs):
</li>
          <li>            t = self.depth_stn_layers[d][i](encoder_outputs[i][d])
</li>
          <li>            warped_maps.append(t)
</li>
          <li>        t = torch.cat(warped_maps, dim=1) if self.n_inputs &gt; 1 else warped_maps[0]
</li>
          <li>        t = self.joiner_layers[d](t)
</li>
          <li>        self.joiner_outputs.append(t)
</li>
          <li>    return self.joiner_outputs
</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>As currently written, it looks like the <code>Joiner</code> module keeps its output alive by appending it to a list for each iteration. But after the first backward call, the graph for this output will be free’d even though it is reused in the next iteration as an input to <code>decoder</code>. This looks strange, and if this is your actual intent you would either want to append a clone of the output instead (likely with <code>.detach()</code> as well) or call backward with <code>retain_graph=True</code>. The second case would apply if you want gradients to flow through according to the previously computed outputs for each iteration, but it would increase memory usage.</p>
<p>I suspect the issue is that <code>joiner_outputs</code> should be recomputed from scratch every time and shouldn’t be a member variable that is kept alive across every iteration.</p>

username: Aladdine_Ben_Zekri
cooked: <p>Actually yes my mistake was that I didn’t clear the output of the joiner model at each iteration, kinda stupid on my part <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>