Change the gradient inplace problem in loss.backward()

username: S_M
cooked: <p>I am using loss.backward() in my code with an issue about change gradient by an inplace operation.<br>
The loss is calculated using following class. Any help to handle this issue would be appreciated.</p>
<p>import torch<br>
from torch import nn<br>
from torchvision.models.vgg import vgg16<br>
import numpy as np</p>
<p>class GeneratorLoss(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(GeneratorLoss, self).<strong>init</strong>()<br>
vgg = vgg16(pretrained=True)<br>
loss_network = nn.Sequential(*list(vgg.features)[:31]).eval()<br>
for param in loss_network.parameters():<br>
param.requires_grad = False<br>
self.loss_network = loss_network<br>
self.mse_loss = nn.MSELoss()<br>
self.tv_loss = TVLoss()</p>
<pre><code>def forward(self, out_images, target_images, class_det_loss, lambda_class):
    # Perception Loss
    perception_loss = self.mse_loss(self.loss_network(out_images), self.loss_network(target_images))
    # Image Loss
    image_loss = self.mse_loss(out_images, target_images)
    # TV Loss
    tv_loss = self.tv_loss(out_images)
    result = torch.tensor(0.0, requires_grad=True)
    result = image_loss + 0.006 * perception_loss + 2e-8 * tv_loss + lambda_class*class_det_loss

    return result
</code></pre>
<p>class TVLoss(nn.Module):<br>
def <strong>init</strong>(self, tv_loss_weight=1):<br>
super(TVLoss, self).<strong>init</strong>()<br>
self.tv_loss_weight = tv_loss_weight</p>
<pre><code>def forward(self, x):
    batch_size = x.size()[0]
    h_x = x.size()[2]
    w_x = x.size()[3]
    count_h = self.tensor_size(x[:, :, 1:, :])
    count_w = self.tensor_size(x[:, :, :, 1:])
    h_tv = torch.pow((x[:, :, 1:, :] - x[:, :, :h_x - 1, :]), 2).sum()
    w_tv = torch.pow((x[:, :, :, 1:] - x[:, :, :, :w_x - 1]), 2).sum()
    return self.tv_loss_weight * 2 * (h_tv / count_h + w_tv / count_w) / batch_size

@staticmethod
def tensor_size(t):
    return t.size()[1] * t.size()[2] * t.size()[3]
</code></pre>
<p>error:<br>
one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [512, 10]], which is output 0 of TBackward, is at version 2; expected version 1 instead.</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/s_m">@S_M</a>,</p>
<p>Given the shape of your Tensor causing the in-place error is <code>[512,10]</code>, I’d assume it’s the output layer that classifies the 10 classes?</p>
<ol>
<li>Can you check that the <code>vgg16</code> doesn’t have an in-place <code>torch.nn.ReLU()</code> calls?</li>
<li>remove the <code>    result = torch.tensor(0.0, requires_grad=True)</code>, which <em>may</em> cause the in-place error and is immediately overwritten.</li>
</ol>
<p>Also, make sure to correctly define your constructor in both <code>GeneratorLoss</code> and <code>TVLoss</code> classes and initialize the parent class correctly. So change <code>def init(*args)</code> to <code>def __init__(*args)</code> and <code>super(GeneratorLoss, self).init()</code> to <code>super(GeneratorLoss, self).__init__()</code>. An example can be found <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module" rel="noopener nofollow ugc">here</a>.</p>

username: S_M
cooked: <p>Hi <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a>,<br>
Thank you so much for your reply.<br>
The size of final layer is 512x10 for 10 classes classification problem.</p>
<ol>
<li>The vgg16 comes from torchvision.models.vgg import vgg16. How can I replace all ReLU(inplace=True) with ReLU(inplace=False) for a pretrained model?</li>
<li>The result = torch.tensor(0.0, requires_grad=True) removed.<br>
The constructor definiction correctly chenged but the issue didn’t solve.</li>
</ol>

username: Ascrypto
cooked: <p>The only thing that helped me with this issue was switching to an older PyTorch version  (1.4). Please be aware, you may run into some compatibility issues with some other packages.</p>

username: S_M
cooked: <aside class="quote no-group" data-username="AlphaBetaGamma96" data-post="2" data-topic="163039">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3da27b/48.png" class="avatar"> AlphaBetaGamma96:</div>
<blockquote>
<p>nn.ReLU()</p>
</blockquote>
</aside>
<p>The nn.ReLU(inplace=True) is changed with following code, unfortunately the error didn’t solve.<br>
vgg.features[1] = nn.ReLU(inplace=False)<br>
vgg.features[3] = nn.ReLU(inplace=False)<br>
vgg.features[6] = nn.ReLU(inplace=False)<br>
vgg.features[8] = nn.ReLU(inplace=False)<br>
vgg.features[11] = nn.ReLU(inplace=False)<br>
vgg.features[13] = nn.ReLU(inplace=False)<br>
vgg.features[15] = nn.ReLU(inplace=False)<br>
vgg.features[18] = nn.ReLU(inplace=False)<br>
vgg.features[20] = nn.ReLU(inplace=False)<br>
vgg.features[22] = nn.ReLU(inplace=False)<br>
vgg.features[25] = nn.ReLU(inplace=False)<br>
vgg.features[27] = nn.ReLU(inplace=False)<br>
vgg.features[29] = nn.ReLU(inplace=False)<br>
vgg.classifier[1]=nn.ReLU(inplace=False)<br>
vgg.classifier[4]=nn.ReLU(inplace=False)</p>

username: AlphaBetaGamma96
cooked: <p>Reverting pytorch to an older version isn’t an appropriate solution. The problem is that the final output layer of the classifier is being performed in-place for some reason, or perhaps the following layer is being performed in-place because you’re performing reverse-mode AD so the gradient are multiplied in a reverse ordering.</p>

username: AlphaBetaGamma96
cooked: <p>Can you run your code within a <code>torch.autograd.set_detect_anomaly</code> context manager? Because that should highlight what’s causing the error. You can find more out <code>torch.autograd.set_detect_anomaly</code> <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.set_detect_anomaly" rel="noopener nofollow ugc">here</a></p>
<p>An example is to do,</p>
<pre><code class="lang-auto">with torch.autograd.set_detect_anomaly():
  #place your code here... (make sure it's all indented correctly)
</code></pre>
<p>and make sure to paste in the stacktrace into your reply too as that’ll help me debug your problem.</p>
<p>Also, why are you doing this?</p>
<pre><code class="lang-auto">for param in loss_network.parameters():
  param.requires_grad = False
</code></pre>
<p>If you want to evaluate your model without update its parameters, you can just run your code within a <code>torch.no_grad()</code> context manager (more detail about that <a href="https://pytorch.org/docs/stable/generated/torch.no_grad.html" rel="noopener nofollow ugc">here</a>)</p>

username: ptrblck
cooked: <p>Besides what <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a> explained you might also be running into <a href="https://discuss.pytorch.org/t/struggling-with-a-runtime-error-related-to-in-place-operations/102222/6">this issue</a> where stale forward activations are used to calculate gradients with the already (inplace) updated parameters. Could you check if your use case is similar or the same?</p>

username: Ascrypto
cooked: <p>This might explain the useless results. Unfortunately, I’ve tried almost all the remaining solutions (for weeks) to fix it without success.</p>

username: S_M
cooked: <p>The problem was solved by add .detach() to the each part of losses carries gradient and returned back to the main</p>