VAE-GAN: INPLACE operation error

username: pankratozzi
cooked: <p>I’m new in PyTorch. I’m implementing simple VAE-GAN model, based on this great notebook: <a href="https://www.kaggle.com/carloalbertobarbano/faceswap-trump-in-a-cage/notebook" rel="noopener nofollow ugc">https://www.kaggle.com/carloalbertobarbano/faceswap-trump-in-a-cage/notebook</a>. During training Autoencoder (generator) model I always get the same error:</p>
<blockquote>
<p><strong>“RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [2048, 1024]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).”</strong></p>
</blockquote>
<p>Here is the part of my code that raises this error (one step training functions):</p>
<pre><code class="lang-auto">def train_discriminator(D, criterion, optimizer, real, fake):
    optimizer.zero_grad()
    
    with torch.set_grad_enabled(True):
        pred_real = D(real)
        pred_fake = D(fake)
        
        loss_real = criterion(pred_real, torch.ones(real.size(0), 1).to(device))
        loss_fake = criterion(pred_fake, torch.zeros(fake.size(0), 1).to(device))
        
        loss = loss_real + loss_fake
        loss.backward(retain_graph=True)

        optimizer.step()
    
    return loss.item()

def train_generator(D, criterion_G, criterion_D, optimizer, x, fake, mu, logvar):
    optimizer.zero_grad()

    with torch.set_grad_enabled(True):
        prediction = D(fake)
        #before = list(D.parameters())[0].clone()

        target = torch.ones(x.size(0), 1).to(device)
        d_loss = criterion_D(prediction, target)
        d_loss.backward(retain_graph=True)  # EXCEPTION raises here
        d_loss = d_loss.item()
        
        if criterion_G is not None:
            g_loss = criterion_G(fake, x, mu, logvar)
            grads = torch.ones_like(g_loss)
            g_loss.backward(grads, retain_graph=True)
            d_loss += g_loss.mean().item()
    
        optimizer.step()
    #after = list(D.parameters())[0].clone()
    #print(torch.equal(before.data, after.data))

    return d_loss
</code></pre>
<p>Training loop:</p>
<pre><code class="lang-auto">model.train()
D_A.train()
D_B.train()

loss_hist = {'D': list(), 'G': list()}

for epoch in range(epochs):
    print(f'Epoch {epoch+1}/{epochs}')

    g_loss = 0
    d_loss = 0

    for i, batch in enumerate(tqdm(dataloader, leave=False), 1):
        x_A, x_B = batch
        
        fake_A, mu_A, logvar_A = model(x_A)
        fake_B, mu_B, logvar_B = model(x_B, select='B')
        
        d_loss += train_discriminator(D_A, criterion_D, optimizerD_A, x_A, fake_A)
        d_loss += train_discriminator(D_B, criterion_D, optimizerD_B, x_B, fake_B)

        g_loss += train_generator(D_A, criterion, criterion_D, optimizerA, x_A, fake_A, mu_A, logvar_A)   
        g_loss += train_generator(D_B, criterion, criterion_D, optimizerB, x_B, fake_B, mu_B, logvar_B)  # EXCEPTION raises here, inplace op error

        fake_A2, mu_A2, logvar_A2 = model(x_A, select='B')
        fake_B2, mu_B2, logvar_B2 = model(x_B, select='A')

        g_loss += train_generator(D_B, None, criterion_D, optimizerB, x_A, fake_A2, mu_A2, logvar_A2)
        g_loss += train_generator(D_A, None, criterion_D, optimizerA, x_B, fake_B2, mu_B2, logvar_B2)

    d_loss /= i*2
    g_loss /= i*4

    loss_hist['D'].append(d_loss)
    loss_hist['G'].append(g_loss)

    print(f'Epoch g_loss: {g_loss:.4f}, d_loss: {d_loss:.4f}')
    print(50*'-')

    early(d_loss+g_loss, epoch=epoch, model=model, D_A=D_A, D_B=D_B, optimizerA=optimizerA, optimizerB=optimizerB, 
          optimizerD_A=optimizerD_A, optimizerD_B=optimizerD_B, )
    if early.early_stop:
        print(f'Train loss did not improve for {early.patience} epochs. Training stopped.')
        model, D_A, D_B, optimizerA, optimizerB, optimizerD_A, optimizerD_B, _, early = load_model(PATH)
        break
</code></pre>
<p>I suppose, that the problem might be in using the computation graph multiple times. I’ve tried almost everything (setting retain_graph=False, using .clone() with different tensors, detaching different tensors, etc.), but I still can’t figure out where this inplace operation took place and how to avoid it.</p>
<p>For interested readers here is the full code (if you have issues to open it, I can upload it anywhere you prefer): <a href="https://colab.research.google.com/drive/1VAHPg0fhoJjKWW_m7ghIHODng4rtvncO#scrollTo=wszuY_A-5JGt" rel="noopener nofollow ugc">Colab notebook</a></p>
<p>I’m stuck and confused. I would appreciate any suggestions, this is my Everest for now, help me, please, to conquer it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: AlphaBetaGamma96
cooked: <p>I don’t have any experience in using VAE-GAN’s but after having a brief look through your code there I can see numerous uses of inplace operators that are probably causing the issue.</p>
<p>Within PyTorch, using inplace operator break the computational graph and basically results in Autograd failing in getting your gradients. Inplace operators within PyTorch are denoted with an <code>_</code>, for example <code>mul</code> does elementwise multiplciation where <code>mul_</code> does elementwise multiplication inplace. So avoid those commands.</p>
<p>Other inplace operators are <code>+=</code> and <code>/=</code> you’ve used those commands a few times throughout your training loop, discriminator, and generator. So, you need to replace those with out-of-place operations instead, for example,</p>
<pre><code class="lang-auto">d_loss += train_discriminator(D_A, criterion_D, optimizerD_A, x_A, fake_A)
</code></pre>
<p>should be replaced with</p>
<pre><code class="lang-auto">d_loss_minibatch = train_discriminator(D_A, criterion_D, optimizerD_A, x_A, fake_A)
d_loss = d_loss + d_loss_minibatch
</code></pre>
<p>Try replacing all in-place operators with the out-of-place equivalent and see if your error goes away! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: pankratozzi
cooked: <p>Thank you so much for your quick reply!<br>
I’ve already thought and tried to replace all such " <code>+=</code>, <code>/=</code>" operations, also add_, mul_ and so on. Still getting error.<br>
In a training loop and everywhere else I summarize items from tensor (g_loss.item() …)<br>
But I’ve tried making it in every possible place either - got same error.<br>
By now I realize that the problem is in mu and logvar tensors, but I can’t figure out where exactly. I’ve changed loss function (that uses mu and logvar)</p>
<pre><code class="lang-auto">class KLDLoss(nn.Module):
    def forward(self, mu, logvar):
        return -0.5 * torch.sum(1 + logvar.clone() - torch.pow(mu, 2) - torch.exp(logvar))
</code></pre>
<p>so that there might not be any inplace operations - and finally a few training iterations ended without error! BUT. after I restarted the GPU environment in Colab for some reasons I again meet the same error <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:">  So the error is unstable!<br>
The error gone completely when I modified training loop something like that (recalculating needed tensors via updated model…):</p>
<pre><code class="lang-auto">...

    for i, batch in enumerate(tqdm(dataloader, leave=False), 1):
        x_A, x_B = batch
        
        fake_A, mu_A, logvar_A = model(x_A)
        fake_B, mu_B, logvar_B = model(x_B, select='B')
        
        d_loss += train_discriminator(D_A, criterion_D, optimizerD_A, x_A, fake_A)
        d_loss += train_discriminator(D_B, criterion_D, optimizerD_B, x_B, fake_B)

        g_loss += train_generator(D_A, criterion, criterion_D, optimizerA, x_A, fake_A, mu_A, logvar_A)
        fake_B, mu_B, logvar_B = model(x_B, select='B') ## mu and logvar raises inplace error otherwise
        g_loss += train_generator(D_B, criterion, criterion_D, optimizerB, x_B, fake_B, mu_B, logvar_B)

        fake_A2, mu_A2, logvar_A2 = model(x_A, select='B')
        fake_B2, mu_B2, logvar_B2 = model(x_B, select='A')

        g_loss += train_generator(D_B, None, criterion_D, optimizerB, x_A, fake_A2, mu_A2, logvar_A2)
        fake_B2, mu_B2, logvar_B2 = model(x_B, select='A') ##
        g_loss += train_generator(D_A, None, criterion_D, optimizerA, x_B, fake_B2, mu_B2, logvar_B2)
 ...
</code></pre>
<p>But it feels like it is not actually correct approach. And it is better to find some other solvation <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: AlphaBetaGamma96
cooked: <aside class="quote no-group" data-username="pankratozzi" data-post="3" data-topic="143895">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pankratozzi/48/46564_2.png" class="avatar"> pankratozzi:</div>
<blockquote>
<pre><code class="lang-auto">class KLDLoss(nn.Module):
    def forward(self, mu, logvar):
        return -0.5 * torch.sum(1 + logvar.clone() - torch.pow(mu, 2) - torch.exp(logvar))
</code></pre>
</blockquote>
</aside>
<p>three questions for your <code>KLDLoss</code> class,</p>
<ol>
<li>why are you calling <code>.clone()</code> within the forward? It seems unnecessary.</li>
<li>You need to define the <code>__init__</code> method as well when constructing your class</li>
<li>When you’ve constructed the <code>KLDLoss</code> class you need to call an instance of it, i.e.</li>
</ol>
<pre><code class="lang-auto">kld_loss = KLDLoss() #create instance 
result = kld_loss(mu, logvar) #use the instance as your function
</code></pre>
<p>Also, still go through your code and remove all <code>+=</code> and <code>/=</code> operations (as well all other in-place operations too) as they will be a problem eventually.</p>

username: pankratozzi
cooked: <p>Again, thanks for reply!</p>
<blockquote>
<ol>
<li>why are you calling <code>.clone()</code> within the forward? It seems unnecessary.</li>
</ol>
</blockquote>
<p>You are right, it is absolutely unnecessary. I just gone crazy trying to prevent any even impossible inplace operations <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<blockquote>
<ol start="2">
<li>You need to define the <code>__init__</code> method as well when constructing your class</li>
</ol>
</blockquote>
<p>I thought that  KLDLoss class is using default nn.Module <strong>init</strong>() method, but done!</p>
<blockquote>
<ol start="3">
<li>When you’ve constructed the <code>KLDLoss</code> class you need to call an instance of it, …</li>
</ol>
</blockquote>
<p>I’m using it in this way.</p>
<pre><code class="lang-auto">reconstruction_loss = torch.nn.functional.binary_cross_entropy
kld_loss = KLDLoss()
criterion = lambda y, x, mu, logvar: reconstruction_loss(y, x, reduction='sum') + kld_loss(mu, logvar)
</code></pre>
<p>All +=, /=, etc. ops have been removed.</p>
<p>Maybe there’s a problem in this part in the model itself.</p>
<pre><code class="lang-auto">    def reparameterize(self, mu, logvar):
        std = torch.exp(0.5*logvar)
        eps = torch.randn_like(std)
        sample = mu + (eps * std) 
        return sample

    def forward(self, x, select='A'):
        x = self.encoder(x)
  
        mu, logvar = self.mean(x), self.std(x)
        z = self.reparameterize(mu, logvar)

        if select == 'A':
            y = self.decoder_A(z)
        else:
            y = self.decoder_B(z)
        
        return y, mu, logvar
</code></pre>
<p>where self.mean and self.std are:</p>
<pre><code class="lang-auto">self.mean = nn.Linear(2048, 1024)
self.std = nn.Linear(2048, 1024)
</code></pre>
<p>respectively.</p>

username: AlphaBetaGamma96
cooked: <aside class="quote no-group" data-username="pankratozzi" data-post="5" data-topic="143895">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pankratozzi/48/46564_2.png" class="avatar"> pankratozzi:</div>
<blockquote>
<p>I thought that KLDLoss class is using default nn.Module <strong>init</strong>() method, but done!</p>
</blockquote>
</aside>
<p>When creating an <code>nn.Module</code> you need to initialise the parent class in order for it to work! That’s what <code>__init__</code> does.</p>
<aside class="quote no-group quote-modified" data-username="pankratozzi" data-post="1" data-topic="143895">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pankratozzi/48/46564_2.png" class="avatar"> pankratozzi:</div>
<blockquote>
<blockquote>
<p><strong>“RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [2048, 1024]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).”</strong></p>
</blockquote>
</blockquote>
</aside>
<p>Ok, so given the error above it seems that your error is probably coming from <code>self.mean</code> and <code>self.std</code>., as they’re the same shape as <code>[torch.cuda.FloatTensor [2048, 1024]]</code>.</p>
<aside class="quote no-group" data-username="pankratozzi" data-post="5" data-topic="143895">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pankratozzi/48/46564_2.png" class="avatar"> pankratozzi:</div>
<blockquote>
<pre><code class="lang-auto">self.mean = nn.Linear(2048, 1024)
self.std = nn.Linear(2048, 1024)
</code></pre>
</blockquote>
</aside>
<p>Did you run your code with <code>torch.autograd.set_detect_anomaly(True)</code>? That will tell you where it’s crashing</p>

username: pankratozzi
cooked: <p>Yes, I ran the cell with anomaly detection, but it just sends me back to the train_generator function:</p>
<blockquote>
<p>—&gt; 27         d_loss.backward(retain_graph=True)</p>
</blockquote>
<p>The error than:</p>
<blockquote>
<p><strong>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [2048, 1024]], which is output 0 of AsStridedBackward0, is at version 4; expected version 3 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</strong></p>
</blockquote>
<p>the last sentence looks like they knew it would be tough situation <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>

username: pankratozzi
cooked: <p>This issue looks like similar to mine:<br>
<a href="https://discuss.pytorch.org/t/inplace-operation-error-with-gan/132100">inplace error in GAN</a><br>
But I don’t have enough understanding to extrapolate this approach over my situation…</p>

username: pankratozzi
cooked: <p>I’m really sorry for disturbing you all the time, but I thought it might be interesting for you:<br>
I’m not completely sure this is the correct approach in common but the following changes in code eliminates the given inplace ops error:</p>
<pre><code class="lang-auto">def switch_params(model, on=True):
    for param in model.parameters():
        param.requires_grad = on

...
def train_generator(D, criterion_G, criterion_D, optimizer, x, fake, mu, logvar):
    optimizer.zero_grad()

    with torch.set_grad_enabled(True):
        prediction = D(fake.detach())

        target = torch.ones(x.size(0), 1).to(device)
        d_loss = criterion_D(prediction, target)
        if criterion_G is not None:
            g_loss = criterion_G(fake, x, mu, logvar)
            total_loss = d_loss + g_loss.mean()
        else:
            total_loss = d_loss
        total_loss.backward(retain_graph=True)
    
        optimizer.step()

    return total_loss.item()

# the training loop

model.train()
D_A.train()
D_B.train()

loss_hist = {'D': list(), 'G': list()}

for epoch in range(epochs):
    print(f'Epoch {epoch+1}/{epochs}')

    g_loss = 0
    d_loss = 0

    for i, batch in enumerate(tqdm(dataloader, leave=False), 1):
        x_A, x_B = batch
        switch_params(model, False)
        fake_A, mu_A, logvar_A = model(x_A)
        fake_B, mu_B, logvar_B = model(x_B, select='B')

        switch_params(model, True)
        d_loss += train_discriminator(D_A, criterion_D, optimizerD_A, x_A, fake_A)
        d_loss += train_discriminator(D_B, criterion_D, optimizerD_B, x_B, fake_B)
        
        g_loss += train_generator(D_A, criterion, criterion_D, optimizerA, x_A, fake_A, mu_A, logvar_A)
        g_loss += train_generator(D_B, criterion, criterion_D, optimizerB, x_B, fake_B, mu_B, logvar_B)

        switch_params(model, False)
        fake_A2, mu_A2, logvar_A2 = model(x_A, select='B')
        fake_B2, mu_B2, logvar_B2 = model(x_B, select='A')

        switch_params(model, True)
        g_loss += train_generator(D_B, None, criterion_D, optimizerB, x_A, fake_A2, mu_A2, logvar_A2)
        g_loss += train_generator(D_A, None, criterion_D, optimizerA, x_B, fake_B2, mu_B2, logvar_B2)

    d_loss /= i*2
    g_loss /= i*4

    loss_hist['D'].append(d_loss)
    loss_hist['G'].append(g_loss)

    print(f'Epoch g_loss: {g_loss:.4f}, d_loss: {d_loss:.4f}')
    print(50*'-')
...
</code></pre>
<p>there is a new problem with almost no loss reduction while training, but it is in fact some different issue <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: AlphaBetaGamma96
cooked: <p>If I had to guess it’s the <code>switch_params</code> command if you want to do a section of code without calcualting a gradients run it within the <code>torch.no_grad</code> context managed. So, for example change</p>
<pre><code class="lang-auto">        switch_params(model, False)
        fake_A2, mu_A2, logvar_A2 = model(x_A, select='B')
        fake_B2, mu_B2, logvar_B2 = model(x_B, select='A')
</code></pre>
<p>to</p>
<pre><code class="lang-auto">        with torch.no_grad():
          fake_A2, mu_A2, logvar_A2 = model(x_A, select='B')
          fake_B2, mu_B2, logvar_B2 = model(x_B, select='A')
</code></pre>
<p>and also remove the <code>+=</code> inplace operators!</p>