Getting this warning; Output 0 of BackwardHookFunctionBackward is a view and is being modified inplace

username: Akshay_gupta
cooked: <p>I have checked the post here , trying to fix the warning but I am still getting the warning.<br>
<a href="https://discuss.pytorch.org/t/custom-autograd-function-backward-pass-not-called/71065">Custom Autograd Function Backward pass not Called</a></p>
<p><code>/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:1204: UserWarning: Output 0 of BackwardHookFunctionBackward is a view and is being modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is deprecated and will be forbidden starting version 1.6. You can remove this warning by cloning the output of the custom Function. (Triggered internally at  /pytorch/torch/csrc/autograd/variable.cpp:547.)   result = torch.relu_(input)</code></p>
<p><strong>Here is my code</strong></p>
<pre><code class="lang-auto">class _Grad():
  def __init__(self,model,target_layer=None,input_size = [3,224,224]):
    
    if not isinstance(model,torch.nn.Module):
      raise ValueError("Provide a valid model")
    self.model = model
    self.model_dic = dict(model.named_modules())
    
    if target_layer is None or target_layer not in self.model_dic.keys():
      raise ValueError("Provide a valid layer")
    self.target_layer = self.model_dic[target_layer]
    
    self.activations = None
    self.grads = None
    self.hooks = []
    back_hook = 'register_full_backward_hook' if torch.__version__ &gt;= '1.8.0' else 'register_backward_hook'
    self.hooks.append(self.target_layer.register_forward_hook(self._extract_activations))
    self.hooks.append(getattr(self.target_layer,back_hook)(self._extract_grads))

  def _extract_activations(self,module,input,output):
      self.activations = output.data

  def _extract_grads(self,module,input,output):
      self.grads = output[0].data

  def _backpropagate(self,class_indx,scores):
    if self.activations is None:
      raise TypeError("Input needs to be passed before Backpropagation")
    loss = scores[:,class_indx].sum()
    self.model.zero_grad()
    loss.backward(retain_graph=True)

  def _get_weights(self,class_indx,scores):
    self._backpropagate(class_indx,scores)
    b,c,h,w = self.grads.size()
    weights = self.grads.view(b, c, -1).mean(2)
    weights = weights.view(b,c,1,1)
    return weights.clone()

  def get_cam_map(self,class_indx,scores,normalized):
    weights = self._get_weights(class_indx,scores)
    
    cams = torch.nansum((weights*self.activations).squeeze(0),0)
    cams = F.relu(cams)
    if normalized :
      cam_map_min, cam_map_max = cams.min(), cams.max()
      cams = (cams - cam_map_min).div(cam_map_max)
    return cams

  def __call__(self,class_indx,scores,normalized=True):
    return self.get_cam_map(class_indx,scores,normalized)
    
from torchvision.models import densenet121
model = densenet121(pretrained=True).eval()
img = read_image("image.png")
input_tensor = normalize(resize(img, (224, 224)) / 255., [0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
out = model(input_tensor.unsqueeze(0))
</code></pre>

username: Akshay_gupta
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , I don’t want to tag you personally but I am not able to resolve it. Can you please help.</p>

username: eduardo4jesus
cooked: <aside class="quote no-group" data-username="Akshay_gupta" data-post="1" data-topic="122766">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/akshay_gupta/48/30602_2.png" class="avatar"> Akshay_gupta:</div>
<blockquote>
<p><code>read_image</code></p>
</blockquote>
</aside>
<p>can you provide <code>read_image</code>? I am trying to reproduce it. Is that returning <code>numpy</code> or <code>Torch.Tensor</code>? what shape?</p>

username: Akshay_gupta
cooked: <p>Their is no constraint on image size while reading. I am using torchvision for reading so output will be a tensor</p>
<pre><code class="lang-auto">from torchvision.io.image import read_image 
</code></pre>

username: eduardo4jesus
cooked: <p>As you can see <a href="https://colab.research.google.com/gist/eduardo4jesus/64658b61a7d7e8231cfec09b4dbc267e/akshay_gupta.ipynb" rel="noopener nofollow ugc">here</a>, I was not able to reproduce the exact same error your are describing.</p>
<p>How are you exactly using <code>_Grad</code>?</p>
<p>Also, what version of <code>torch</code> you are using?</p>

username: eduardo4jesus
cooked: <p>Also, It is not clear what you are trying to do. The more details the better on what you are trying to do.</p>

username: Akshay_gupta
cooked: <p>I missed the line causing the bug. Check it <a href="https://colab.research.google.com/drive/1F-RGw9pSK1VFa_qXV4UQuzDSspbtunnh?usp=sharing" rel="noopener nofollow ugc">Link</a></p>
<pre><code class="lang-auto">from torchvision.models import densenet121
model = densenet121(pretrained=True).eval()
img = read_image("image.png")
input_tensor = normalize(resize(img, (224, 224)) / 255., [0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
extract = _Grad(model=model,target_layer="features")
out = model(input_tensor.unsqueeze(0))
</code></pre>
<p>I am implementing the Grad-cam <a href="https://arxiv.org/abs/1610.02391" rel="noopener nofollow ugc">Link</a>  for visual explanations of classification models. Given the target-layer, I want extract the activations during forward pass and gradients during backward pass for it.<br>
<code>torch</code> version: 1.8.1+cu101</p>

username: kevingoh
cooked: <p>problem is with the inplace in the activation function of your model. You need to disable it.</p>

username: AlphaBetaGamma96
cooked: <aside class="quote no-group" data-username="Akshay_gupta" data-post="1" data-topic="122766">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/akshay_gupta/48/30602_2.png" class="avatar"> Akshay_gupta:</div>
<blockquote>
<pre><code class="lang-auto">    back_hook = 'register_full_backward_hook' if torch.__version__ &gt;= '1.8.0' else 'register_backward_hook'
</code></pre>
</blockquote>
</aside>
<p>Also, you don’t want to use <code>register_backward_hook</code> as its behaviour is broken (see this issue <a href="https://github.com/pytorch/pytorch/issues/598" rel="noopener nofollow ugc">here</a>)</p>

username: Akshay_gupta
cooked: <p>Thanks <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a>  <a class="mention" href="/u/kevingoh">@kevingoh</a>. Resolved.</p>

username: Ruy_Diaz
cooked: <p>Hello Kevin</p>
<p>I tried the following:</p>
<pre><code class="lang-auto">&gt; for i, (name, layer) in enumerate(model.named_modules()):
&gt;     if isinstance(layer, nn.ReLU):
&gt;         layer = nn.ReLU(inplace=False)
</code></pre>
<p>but that couldn’t disable those “in place” RELU features.</p>
<p>Would you please give some advice? Is it necessary to redefine model and retrain?</p>
<p>Thanks</p>

username: AlphaBetaGamma96
cooked: <p>Can you share your <code>model</code> code too?</p>

username: Ruy_Diaz
cooked: <p>It’s the standard resnet34 with every RELU “in place”.</p>

username: AlphaBetaGamma96
cooked: <p>So, you share a minimal reproducible example of it so I can debug your problem?</p>

username: Mark-Dou
cooked: <p>Have you resolved the problem?  I have been trapped in the same error. Could you give me some advice? Thanks a lot.</p>

username: Ruy_Diaz
cooked: <p>I just modified the code that builds the resnet model, replacing every True with False after in_place for every Relu activation. BTW, I am using Captum’s guided grad-cam.</p>

username: f_g
cooked: <p>Hi,</p>
<p>I’m facing the same issue but on the Linear module, the stack trace is as follows, this is happening when I’m using FSDP. I have tried using <code>transformer_auto_wrap_policy</code> but that doesn’t seem to help. Any ideas?</p>
<pre><code class="lang-auto">File ".../pnp_dist.runfiles/pip_torch/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: Output 0 of ViewBackward0 is a view and its base or another view of its base has been modified inplace. This view is the output of a function that returns multiple views. Such functions do not allow the output views to be modified inplace. You should replace the inplace operation by an out-of-place one.
</code></pre>