Multiple Loss Functions in a Model

MUnique: <p>Hello everyone,</p>
<p>I am trying to train a model constructed of three different modules. An encoder, a decoder, and a discriminator. These three are connected as follows. Each of these three should minimize its own loss function which is different from the others.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7ab438d07cb1d8ecd28f10f5b4bfd438392acd1b.png" data-download-href="https://discuss.pytorch.org/uploads/default/7ab438d07cb1d8ecd28f10f5b4bfd438392acd1b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7ab438d07cb1d8ecd28f10f5b4bfd438392acd1b_2_690x345.png" alt="image" data-base62-sha1="hvunfTf98vQkk9TUsTzCeqJg1Dl" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7ab438d07cb1d8ecd28f10f5b4bfd438392acd1b_2_690x345.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7ab438d07cb1d8ecd28f10f5b4bfd438392acd1b_2_1035x517.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/a/7ab438d07cb1d8ecd28f10f5b4bfd438392acd1b.png 2x" data-dominant-color="DCD8D1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1287×645 15.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here is my code snippet:</p>
<p>with torch.autograd.set_detect_anomaly(True):<br>
for epoch in range(num_epochs):<br>
for i, (data, labels) in enumerate(train_loader):</p>
<pre><code>        data = Variable(data.cuda())
        labels = Variable(labels.cuda())

        Encoder_hiddens = Variable((torch.zeros(2 * num_encoder_layers, batch_size, hidden_size)).cuda())
        Disc_hiddens = Variable((torch.zeros(2 * num_disc_layers, batch_size, hidden_size)).cuda())

        masked_data = data
        masked_data[:, 1:, mask_part] = 0

        # forward step
        optimizer_encoder.zero_grad()

        outputs_end, hidden_enc = model_encoder(data, Encoder_hiddens)
        outputs_dec = model_decoder(masked_data, hidden_enc)

        loss_ele = pixelwise_criterion(outputs_dec, data)
        # backward step
        loss_ele.backward()
        # optimization step
        optimizer_encoder.step()

        optimizer_decoder.zero_grad()
        disc_real = model_disc(data, Disc_hiddens).view(-1)
        lossD_real = adversarial_criterion(disc_real, torch.ones_like(disc_real))
        disc_fake = model_disc(outputs_dec, Disc_hiddens).view(-1)
        lossD_fake = adversarial_criterion(disc_fake, torch.zeros_like(disc_fake))
        lossD = (lossD_real + lossD_fake) / 2
        loss_decoder = adv_ratio * lossD + (1 - adv_ratio) * loss_ele
        loss_decoder.backward()
        optimizer_decoder.step()

        optimizer_disc.zero_grad()
        lossD.backward()
        optimizer_disc.step()
</code></pre>
<p>I am getting the following error:<br>
RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.</p>
<p>When I use retain_graph=True (loss_ele.backward(retain_graph=True)), I get the following error:</p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [450, 45]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later.</p>
<p>Does anybody know how it can be fixed?</p>
<p>Thanks,</p>

Dwight_Foster: <p>This is because the loss function releases the data after the backward pass. Because you are passing the outputs_dec into the discriminator after the loss has already been computed for the encoder the graphs combine. To stop this you can do</p>
<pre><code class="lang-auto">model_disc(outputs_dec.detach()
</code></pre>

MUnique: <p>Thanks for the reply. I tried disc_fake = model_disc(outputs_dec.detach(), Disc_hiddens).view(-1) but I am still getting the same error:</p>
<p>File “…/Temp.py”, line 167, in …<br>
loss_decoder.backward(retain_graph=True)<br>
File “…/lib/python3.6/site-packages/torch/tensor.py”, line 221, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “…/lib/python3.6/site-packages/torch/autograd/<strong>init</strong>.py”, line 132, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [450, 45]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later.</p>

InnovArul: <aside class="quote no-group" data-username="MUnique" data-post="1" data-topic="111464">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/d78d45/48.png" class="avatar"> MUnique:</div>
<blockquote>
<pre><code class="lang-auto">        # backward step
        loss_ele.backward()
</code></pre>
</blockquote>
</aside>
<p>You may have to tell pytorch to retain the computation graph for future use. i.e., for gradient calculation of decoder.</p>
<pre><code class="lang-auto">        loss_ele.backward(retain_graph=True)
</code></pre>
<p>You may need <code>retain_graph=True</code> for <code>loss_decoder.backward()</code> as well.<br>
But not for <code>lossD.backward()</code>, so that memory is released at the end.</p>
<p>Can you try with this?</p>

MUnique: <p>Thanks for the reply. Yeah, I am using the retain_graph=True with the loss_decoder as well. Otherwise, I am getting the first error (Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time). With using retain_graph for these two I am getting the second one (one of the variables needed for gradient computation has been modified by an inplace operation)</p>

Dwight_Foster: <p>With that you don’t want to use retain_graph=True anymore</p>

MUnique: <p>Yeah, I am not using it when I use detach but I am getting the same error message.</p>

Dwight_Foster: <p>Huh. Are you sure because in the error message you sent it says</p>
<pre><code class="lang-auto">File “…/Temp.py”, line 167, in …
loss_decoder.backward(retain_graph=True)
</code></pre>

MUnique: <p>Yeah, I’ve tried both and it doesn’t work. Seems something else is causing the issue.</p>

InnovArul: <p>This looks tricky to achieve with multiple optimizers.</p>
<p>I am not sure if this is the reason. But you are calling multiple <code>optimizer.step()</code>'s in between and the weights of the encoder, decoder are modified in-place due to that. What if you try the weight updates in reverse order (discriminator, then decoder, then encoder):</p>
<pre><code class="lang-auto">calculate encoder loss, decoder loss, discriminator loss

# discriminator update
optimizer_disc.zero_grad()
lossD.backward(retain_graph=True)
optimizer_disc.step()

# decoder update
optimizer_decoder.zero_grad()
loss_decoder.backward(retain_graph=True)
optimizer_decoder.step()

# encoder update
optimizer_encoder.zero_grad()
loss_ele.backward(retain_graph=True)
optimizer_encoder.step()

# to release the computation graph
lossD.backward()
</code></pre>

InnovArul: <p>When you use <code>.detach()</code>, you are disconnecting the decoder from the discriminator. I am not sure if that’s what you want to do.</p>

MUnique: <p>Yeah, you’re right. I want them to be connected to each other.<br>
BTW, I tried what you mentioned and I am getting a similar error message:<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [60000, 1]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
<p>Here I copy my discriminator class in case it is the reason for the issue:</p>
<p>class Discriminator(nn.Module):<br>
def <strong>init</strong>(self, cfg):<br>
super(Discriminator, self).<strong>init</strong>()</p>
<pre><code>    self.seq_max = cfg.Dataset.Seq_Max
    self. num_joints = cfg.Dataset.Num_joint

    if cfg.Dataset.Num_dimension == 3:
        self. input_size = self.num_joints * 3
    else:
        self.input_size = self.num_joints * 2

    self.hidden_size = cfg.TRAIN.Hidden_size
    self.batch_size = cfg.TRAIN.Batch_Size
    self.layers = cfg.TRAIN.Num_Disc_Layers
    self.dnn_layers = cfg.TRAIN.Encoder_Dnn_Layers
    self.dropout = cfg.TRAIN.Encoder_Dropout
    self.bi = cfg.TRAIN.Bidirectional_Disc
    if self.dnn_layers &gt; 0:
        for i in range(self.dnn_layers):
            self.add_module('dnn_' + str(i), nn.Linear(
                in_features=self.input_size if i == 0 else self.hidden_size,
                out_features=self.hidden_size
            ))
    gru_input_dim = self.input_size if self.dnn_layers == 0 else self.hidden_size
    self.rnn = nn.GRU(
        gru_input_dim,
        self.hidden_size,
        self.layers,
        dropout=self.dropout,
        bidirectional=self.bi,
        batch_first=True)
    self.relu = nn.LeakyReLU(0.2, inplace=False)
    self.fc = nn.Linear(self.hidden_size * self.seq_max * self.layers, 1)
    self.nonlinear = nn.Tanh()

def run_dnn(self, x):
    for i in range(self.dnn_layers):
        x = F.relu(getattr(self, 'dnn_'+str(i))(x))
    return x

def forward(self, inputs, hidden):
    output, embedding = self.rnn(inputs, hidden)
    output = self.relu(output).contiguous()
    output = output.view(output.size()[0], -1)
    output = self.fc(output)
    output = self.nonlinear(output)

    return output
</code></pre>
<p>The decoder and the encoder have similar architectures.</p>

InnovArul: <p>I see. Even in this case, the final <code>lossD.backward()</code> faces this <code>variable modified in-place</code> scenario.</p>
<p>From <a class="mention" href="/u/alband">@albanD</a> 's answer here:</p><aside class="quote quote-modified" data-post="2" data-topic="9255">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-free-graph-manually/9255/2">How to free graph manually?</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi, 
Whenever the output Variable will go out of scope in python, the whole graph will be deleted. 
By default, some intermediary buffers are freed even before that to reduce peak memory usage (this is what is disabled when using retain_graph=True). But the graph and all intermediary buffers are only kept alive as long as they are accessible from python (usually from the output Variable), so running the last backward with retain_graph=True will only keep the intermediary buffers alive until they…
  </blockquote>
</aside>

<p>You can use <code>del lossD</code> instead of final <code>lossD.backward()</code> (to release computational graph). Can you try that?</p>
<p><strong>Edit:</strong> Can you pack encoder and decoder into one optimizer (or) backward them together, if possible? The Encoder grad calculation is dependent on Decoder parameters as well. So, you can’t <code>optimizer_decoder.step()</code> before <code>loss_ele.backward()</code>. One solution is as follows:</p>
<pre><code class="lang-auto">calculate encoder loss, decoder loss, discriminator loss

# discriminator update
optimizer_disc.zero_grad()
lossD.backward(retain_graph=True)
optimizer_disc.step()

# encoder and decoder update
optimizer_encoder.zero_grad()
optimizer_decoder.zero_grad()

loss_generator = loss_encoder + loss_decoder
loss_generator.backward()

optimizer_decoder.step()
optimizer_encoder.step()

# to release the computation graph of the discriminator
del lossD
</code></pre>

MUnique: <p>Thanks a lot for your responses.</p>
<p>This configuration seems to solve the problem:</p>
<pre><code>        optimizer_encoder.zero_grad()
        optimizer_decoder.zero_grad()

        loss_ele.backward(retain_graph=True)
        loss_decoder.backward(retain_graph=True)

        optimizer_encoder.step()
        optimizer_decoder.step()

        # discriminator update
        optimizer_disc.zero_grad()
        lossD.backward()
        optimizer_disc.step()
</code></pre>
<p>BTW, should I still do “del lossD” after all optimization steps? What does it do?</p>

InnovArul: <p><code>del lossD</code> is not really needed if you are not storing any references of it after the <code>for</code> loop.<br>
I have put it to explicitly make sure it is deleted.</p>