Dual Loss Calculation Failed. RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

zaber666: <p>So, What i am basically trying to do can be summarized below.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9a5bf0bb98742603b64f9f209230635551e7f66f.png" data-download-href="https://discuss.pytorch.org/uploads/default/9a5bf0bb98742603b64f9f209230635551e7f66f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9a5bf0bb98742603b64f9f209230635551e7f66f.png" alt="image" data-base62-sha1="m1wvJetAmgPS7FQIDAflTn1tG9h" width="517" height="251" data-dominant-color="F1F2F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">941×457 7.65 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I want to reconstruct the parameter values of the weight of a convolutional layer and want to optimize the network on both this reconstruction loss and categorial cross-entropy loss. It works fine only while optimizing on cross-entropy loss but fails when I add the reconstruction loss. Basically the gradient is not passing through the NN(Neural Net) to update it’s parameter. I want it’s weight to get updated.</p>
<p>The network that I build was like below,</p>
<pre><code class="lang-auto">class MODEL(nn.Module):
    def __init__(self):
        super(MODEL, self).__init__()
        
        self.conv1 = nn.Conv2d(in_channels=3, out_channels=64, kernel_size=(3,3), padding=1)
        self.max_pool1 = nn.MaxPool2d(kernel_size=(3,3))
        self.reBuilder = nn.Conv2d(in_channels=3, out_channels=3, kernel_size=(5,5), padding=2)
        self.lossfn = torch.nn.L1Loss(reduction="mean")
        
        self.conv2 = nn.Conv2d(in_channels=64, out_channels=256, kernel_size=(3,3), padding=1)
        self.max_pool2 = nn.MaxPool2d(kernel_size=(3,3))
        self.conv3 = nn.Conv2d(in_channels=256, out_channels=512, kernel_size=(3,3), padding=1)
        self.max_pool3 = nn.MaxPool2d(kernel_size=(3,3))
        
        self.dense = nn.Linear(in_features=512, out_features=10)
        
        
    def forward(self, x):
        ckpt = self.state_dict()
        
        weight = ckpt["conv1.weight"].detach().clone().permute((3,2,1,0)) #output_shape (3,3,3,64)
        weightNew = self.reBuilder(weight)
        loss = self.lossfn(weightNew, weight)

        weightNew = weightNew.permute((3, 2, 1, 0))
        weightFinal = weightNew.detach().clone()

        ckpt["conv1.weight"] = weightFinal
        self.load_state_dict(ckpt)
        
        x = self.conv1(x)
        x = self.max_pool1(x)
        x = self.conv2(x)
        x = self.max_pool2(x)
        x = self.conv3(x)
        x = self.max_pool3(x)
        x = torch.flatten(x, start_dim=1)
        x = self.dense(x)
        
        return x, loss
</code></pre>
<p>The forward and backward pass code is like,</p>
<pre><code class="lang-auto">model = MODEL()
criterion = nn.BCEWithLogitsLoss().to("cuda")

outputs, recoLoss = model(images)
ceLoss = criterion(outputs, targets)
loss = ceLoss + recoLoss
loss.backward()
optimizer.step()
</code></pre>
<p>If <code>loss</code> contains only <code>ceLoss</code> it works. But when I add <code>recoLoss</code> term it gives the following error.</p>
<pre><code class="lang-auto">"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [3, 3, 5, 5]] is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True)."
</code></pre>
<p>basically it is saying that, the parameters of the NN(in picture) or <code>self.reBuilder</code> in code have been modified already.<br>
Can anyone help me in this regard?</p>

AlphaBetaGamma96: <aside class="quote no-group" data-username="zaber666" data-post="1" data-topic="156024">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/41988e/48.png" class="avatar"> zaber666:</div>
<blockquote>
<pre><code class="lang-auto">        weight = ckpt["conv1.weight"].detach().clone().permute((3,2,1,0)) #output_shape (3,3,3,64)
        weightNew = self.reBuilder(weight)
        loss = self.lossfn(weightNew, weight)

        weightNew = weightNew.permute((3, 2, 1, 0))
        weightFinal = weightNew.detach().clone()
</code></pre>
</blockquote>
</aside>
<p>Can you replace the <code>.detach().clone()</code> with just <code>.clone()</code>.</p>
<p>Also, if you load a <code>state_dict</code> during the forward pass surely that would overwrite all your weights? (And probably cause an in-place error as I <em>think</em> <code>load_state_dict</code> works in-place)</p>