RuntimeError: Trying to backward through the graph a second time. Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True

Karnik_Kanojia: <p>So I have gone through solutions already available on the forum. I’m using <code>torchxrayvision</code> and <code>torchcam</code> library. I needed Densenet121 pretrained weigths and torchcam to generate GradCAM for the model.</p>
<p>Minimal reproducible example I can give you is this</p>
<pre><code class="lang-auto">preds = model(rescaled_output.unsqueeze(0))
cam_extractors = [
    cam(class_idx=i, scores=preds) for i, _ in enumerate(model.pathologies)
]
</code></pre>
<p><code>rescaled_output</code> just represents an image in Tensor of shape (1, 244, 244).<br>
I’m anyways encountering the RuntimeError as following:</p>
<pre><code class="lang-auto">{
	"name": "RuntimeError",
	"message": "Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.",
	"stack": "---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [24], in &lt;cell line: 2&gt;()
      1 preds = model(rescaled_output.unsqueeze(0))
----&gt; 2 cam_extractors = [
      3     cam(class_idx=i, scores=preds) for i, _ in enumerate(model.pathologies)
      4 ]

Input In [24], in &lt;listcomp&gt;(.0)
      1 preds = model(rescaled_output.unsqueeze(0))
      2 cam_extractors = [
----&gt; 3     cam(class_idx=i, scores=preds) for i, _ in enumerate(model.pathologies)
      4 ]

File /opt/homebrew/Caskroom/miniconda/base/envs/xray/lib/python3.10/site-packages/torchcam/methods/core.py:169, in _CAM.__call__(self, class_idx, scores, normalized, **kwargs)
    166 self._precheck(class_idx, scores)
    168 # Compute CAM
--&gt; 169 return self.compute_cams(class_idx, scores, normalized, **kwargs)

File /opt/homebrew/Caskroom/miniconda/base/envs/xray/lib/python3.10/site-packages/torchcam/methods/core.py:193, in _CAM.compute_cams(self, class_idx, scores, normalized, **kwargs)
    178 \"\"\"Compute the CAM for a specific output class.
    179 
    180 Args:
   (...)
    190         the k-th element of the input batch for class index equal to the k-th element of `class_idx`.
    191 \"\"\"
    192 # Get map weight &amp; unsqueeze it
--&gt; 193 weights = self._get_weights(class_idx, scores, **kwargs)
    195 cams: List[Tensor] = []
    197 with torch.no_grad():

File /opt/homebrew/Caskroom/miniconda/base/envs/xray/lib/python3.10/site-packages/torchcam/methods/gradient.py:100, in GradCAM._get_weights(self, class_idx, scores, **kwargs)
     98 \"\"\"Computes the weight coefficients of the hooked activation maps.\"\"\"
     99 # Backpropagate
--&gt; 100 self._backprop(scores, class_idx, **kwargs)
    102 self.hook_g: List[Tensor]  # type: ignore[assignment]
    103 # Global average pool the gradients over spatial dimensions

File /opt/homebrew/Caskroom/miniconda/base/envs/xray/lib/python3.10/site-packages/torchcam/methods/gradient.py:59, in _GradCAM._backprop(self, scores, class_idx, retain_graph)
     57     loss = scores.gather(1, torch.tensor(class_idx, device=scores.device).view(-1, 1)).sum()
     58 self.model.zero_grad()
---&gt; 59 loss.backward(retain_graph=retain_graph)

File /opt/homebrew/Caskroom/miniconda/base/envs/xray/lib/python3.10/site-packages/torch/_tensor.py:492, in Tensor.backward(self, gradient, retain_graph, create_graph, inputs)
    482 if has_torch_function_unary(self):
    483     return handle_torch_function(
    484         Tensor.backward,
    485         (self,),
   (...)
    490         inputs=inputs,
    491     )
--&gt; 492 torch.autograd.backward(
    493     self, gradient, retain_graph, create_graph, inputs=inputs
    494 )

File /opt/homebrew/Caskroom/miniconda/base/envs/xray/lib/python3.10/site-packages/torch/autograd/__init__.py:251, in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    246     retain_graph = create_graph
    248 # The reason we repeat the same comment below is that
    249 # some Python versions print out the first line of a multi-line function
    250 # calls in the traceback and some print out the last line
--&gt; 251 Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    252     tensors,
    253     grad_tensors_,
    254     retain_graph,
    255     create_graph,
    256     inputs,
    257     allow_unreachable=True,
    258     accumulate_grad=True,
    259 )

RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward."
}
</code></pre>
<p>Thank you in advance for your help.</p>

ptrblck: <p>Are you using <code>retain_graph=True</code>? If so, could you explain why this is needed?</p>

Karnik_Kanojia: <p>I’m not using <code>retain_graph=True</code>. So I’m using GradCAM to get activation heatmaps of the neurons. The GradCAM uses backpropagtion values to generate them as far as I know hence that’s why the graph is being accessed second time. Since I’m using <strong>torchxrayvision</strong> directly and not creating a custom model or training script hence I don’t know where I can change the attribute of <em>retain_graph</em>.</p>
<pre><code class="lang-auto">import torchxrayvision

model = xrv.models.DenseNet(weights="densenet121-res224-all")
model.eval()
</code></pre>
<p>That’s all I’m doing to get the model.</p>
<p><strong>EDIT:</strong> Checked the source code. while calling <code>self._backprop</code> it remains False as default value.</p>