Multiple model backpropagations in a loop

David_Gomez: <p>I’ve seen similar posts but their problem is different than mine and their solutions don’t seem to apply.</p>
<p>I have 3 models: source_generator, target_generator and a classifier (discriminator). The generators are feature extractors that process images and output an embedding. Each generator is intended to process only images from its domain, either source or target. The discriminator calssifies the embeddings into target or source labels. The goal of this training is to implement Adversarial Discriminative Domain Adaptation for the target_generator model.</p>
<p>The problem comes with the training loop:</p>
<pre><code class="lang-python"> discriminator_optimizer = optim.Adam(discriminator.parameters(), lr=args.learning_rate)
 generator_optimizer = optim.Adam(target_generator.parameters(), lr=args.learning_rate)

 loss = torch.nn.BCEWithLogitsLoss()

 # Train loop
 print('Training...')
 for epoch in range(args.epochs):
     # Train step
     for i, batch in enumerate(train_dl):
         inputs = batch['image']
         labels = batch['fake']
         
         # Separate the inputs by label
         source_index = labels == 1
         target_index = labels == 0
         source_inputs = inputs[source_index].float().to(device)
         target_inputs = inputs[target_index].float().to(device)

         # Zero grads
         discriminator_optimizer.zero_grad()

         # Process each input with corresponding generator
         source_outputs = source_generator(source_inputs)[0]
         target_outputs = target_generator(target_inputs)[0] 

         # Compute discriminator
         source_labels = torch.Tensor(np.array([np.ones(1) for _ in range(len(source_inputs))])).to(device)
         target_labels = torch.Tensor(np.array([np.zeros(1) for _ in range(len(target_inputs))])).to(device)

         source_loss = loss(discriminator(source_outputs), source_labels)
         target_loss = loss(discriminator(target_outputs), target_labels)
         discriminator_loss = source_loss + target_loss

         # Do backpropagation for discriminator
         discriminator_loss.backward(retain_graph=True)
         discriminator_optimizer.step()

         # Do backpropagation for generator
         target_loss.backward()
         generator_optimizer.step() 
</code></pre>
<p>I get RuntimeErrors related to a variable that has been modified:</p>
<pre data-code-wrap="console"><code class="lang-plaintext">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [16, 1]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>So I get the problem is something related with the fact that the discriminator model and the target_generator model are somehow related because the latter is used to compute the input of the first.<br>
¿Is there another approach to this? ¿Can I somehow backpropagate only a model at a time?</p>

ptrblck: <p>You are most likely running into <a href="https://discuss.pytorch.org/t/struggling-with-a-runtime-error-related-to-in-place-operations/102222/6">this issue</a> trying to compute gradients with stale forward activations after a parameter update was performed.<br>
The link shows a code snippet and describes the issues in more detail. Based on your code I would assume:</p>
<pre><code class="lang-python">         # Do backpropagation for discriminator
         discriminator_loss.backward(retain_graph=True)
         discriminator_optimizer.step()

         # Do backpropagation for generator
         target_loss.backward()
</code></pre>
<p>fails, since <code>target_loss.backward()</code> would need the original parameters of the <code>discriminator</code> to compute the gradients, while these were already updated in <code>discriminator_optimizer.step()</code> thus creating stale intermediate forward activations.</p>

KFrank: <p>Hi David!</p>
<p>There are a couple of different things going on here.</p>
<p>Most importantly, it looks like you are training your <code>target_generator</code> backwards,<br>
see below.</p>
<aside class="quote no-group" data-username="David_Gomez" data-post="1" data-topic="182811" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/david_gomez/48/61600_2.png" class="avatar"> David_Gomez:</div>
<blockquote>
<p>I get RuntimeErrors related to a variable that has been modified:</p>
<pre data-code-wrap="console"><code class="lang-console">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [16, 1]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
</blockquote>
</aside>
<p>First, <a class="mention" href="/u/ptrblck">@ptrblck</a>’s explanation of the inplace-modification error is almost<br>
certainly correct.</p>
<pre data-code-wrap="python"><code class="lang-python">         discriminator_optimizer.step()
</code></pre>
<p>performs an inplace modification of <code>discriminator</code>’s parameters.</p>
<p>With:</p>
<pre data-code-wrap="python"><code class="lang-python">         target_loss.backward()
</code></pre>
<p>you then backward through <code>discriminator</code> again.  It is highly likely that at least<br>
some of <code>discriminator</code>’s parameters will be needed to perform that backward<br>
pass, so, because they’ve been modified, you get the error.</p>
<p>But what to do about it?</p>
<p>Here’s the issue about training <code>target_generator</code> “backwards:”</p>
<aside class="quote no-group">
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">         # Process each input with corresponding generator
         source_outputs = source_generator(source_inputs)[0]
         target_outputs = target_generator(target_inputs)[0] 
</code></pre>
</blockquote>
</aside>
<p><code>target_generator</code> is supposed to generate a <code>target_outputs</code> that is <em>hard</em><br>
for <code>discriminator</code> to tell apart from <code>source_outputs</code>.</p>
<aside class="quote no-group">
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">         source_labels = torch.Tensor(np.array([np.ones(1) for _ in range(len(source_inputs))])).to(device)
         target_labels = torch.Tensor(np.array([np.zeros(1) for _ in range(len(target_inputs))])).to(device)

         source_loss = loss(discriminator(source_outputs), source_labels)
         target_loss = loss(discriminator(target_outputs), target_labels)
</code></pre>
</blockquote>
</aside>
<p>If <code>discriminator</code> correctly recognizes that <code>target_outputs</code> is “fake,” it will<br>
produce zeros (or small values) for its predictions for <code>target_labels</code>, which<br>
are zeroes.  If it does so, it will make <code>target_loss</code> small.  Using <code>target_loss</code><br>
to train <code>discriminator</code> makes sense, as a small value for <code>target_loss</code><br>
“rewards” <code>discriminator</code> for identifying <code>target_outputs</code> as fake.</p>
<p>But we want to <em>penalize</em> <code>target_generator</code> for having produced a<br>
<code>target_outputs</code> that <code>discriminator</code> recognized as fake.  (This is the<br>
“adversarial” part.)</p>
<aside class="quote no-group">
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">         # Do backpropagation for generator
         target_loss.backward()
         generator_optimizer.step() 
</code></pre>
</blockquote>
</aside>
<p>By also training <code>target_generator</code> with <code>target_loss</code> we actually <em>reward</em><br>
<code>target_generator</code> for producing a detectably-fake <code>target_outputs</code>.  This<br>
is backwards of what we want.</p>
<aside class="quote no-group">
<blockquote>
<p>¿Can I somehow backpropagate only a model at a time?</p>
</blockquote>
</aside>
<p>As an aside, yes you can perform your “backwards” logic in a way where<br>
you (more or less) backpropagate one model at a time.  (But, as explained<br>
above, this is not what you want to do.)</p>
<p>Here’s how:</p>
<pre data-code-wrap="python"><code class="lang-python">         # Process each input with corresponding generator
         source_outputs = source_generator(source_inputs)[0]
         target_outputs = target_generator(target_inputs)[0] 

         # Compute discriminator
         source_labels = torch.Tensor(np.array([np.ones(1) for _ in range(len(source_inputs))])).to(device)
         target_labels = torch.Tensor(np.array([np.zeros(1) for _ in range(len(target_inputs))])).to(device)

         # Zero grads
         generator_optimizer.zero_grad()
         discriminator_optimizer.zero_grad()

         target_loss = loss(discriminator(target_outputs), target_labels)

         # Do backpropagation for generator and PART OF discriminator
         target_loss.backward()
         generator_optimizer.step() 
</code></pre>
<p>This accumulates the gradients of <code>target_loss</code> into the parameters of both<br>
<code>target_generator</code> and <code>discriminator</code> and performs an optimization step<br>
on <code>target_generator</code>, modifying it inplace.  The graph for <code>target_loss</code> is<br>
no longer needed, so it is not retained.</p>
<pre><code class="lang-auto">         source_loss = loss(discriminator(source_outputs), source_labels)

         # Do backpropagation for THE REST OF discriminator
         source_loss.backward()
         discriminator_optimizer.step()
</code></pre>
<p>The parameters of <code>discriminator</code> still have their <code>target_loss</code> gradients.<br>
This now accumulates the <code>source_loss</code> gradients into <code>discriminator</code>’s<br>
parameters (so the total gradient becomes that of <code>discriminator_loss</code><br>
which is not explicitly computed in this version) and performs an optimization<br>
step.</p>
<p><code>source_loss</code> does not depend on <code>target_generator</code> so<br>
<code>source_loss.backward()</code> does not backpropagate anything through<br>
<code>target_generator</code>.  Therefore no inplace-modification error occurs,<br>
even though <code>target_generator</code> had been modified inplace.</p>
<p>Even though you <em>don’t</em> want to do this, it’s worth understanding how this<br>
scheme implements your “backwards” logic without an inplace-modification<br>
error.</p>
<p>Now to what you most likely actually want to do:</p>
<p>At the cost of applying <code>discriminator</code> twice to <code>target_outputs</code> we will<br>
compute two versions of <code>target_loss</code> – one that rewards <code>target_generator</code><br>
for fooling <code>discriminator</code> and a second that rewards <code>discriminator</code> for not<br>
getting fooled.</p>
<pre data-code-wrap="python"><code class="lang-python">         # Process each input with corresponding generator
         source_outputs = source_generator(source_inputs)[0]
         target_outputs = target_generator(target_inputs)[0]

         target_labelsA = torch.Tensor(np.array([np.ones(1) for _ in range(len(target_inputs))])).to(device)    # reward target_generator for fooling discriminator
         source_labels = torch.Tensor(np.array([np.ones(1) for _ in range(len(source_inputs))])).to(device)
         target_labelsB = torch.Tensor(np.array([np.zeros(1) for _ in range(len(target_inputs))])).to(device)   # reward discriminator for not getting fooled

         # Zero grads
         generator_optimizer.zero_grad()

         target_lossA = loss(discriminator(target_outputs), target_labelsA)
         target_lossA.backward()
         generator_optimizer.step()

         # clear out target_lossA grads which we don't want
         discriminator_optimizer.zero_grad()

         # detach so that we will not backpropagate through target_generator
         target_outputs = target_outputs.detach()

         source_loss = loss(discriminator(source_outputs), source_labels)
         target_lossB = loss(discriminator(target_outputs), target_labelsB)
         discriminator_loss = source_loss + target_lossB

         # Do backpropagation for discriminator
         discriminator_loss.backward()
         discriminator_optimizer.step()
</code></pre>
<p>To recap:  This scheme uses two versions of <code>target_loss</code> – one to train<br>
<code>target_generator</code> and a second to train <code>discriminator</code>.  Because these<br>
two versions are, in a sense, the opposite of one another, this logic correctly<br>
trains <code>target_generator</code> to fool <code>discriminator</code>, rather than training<br>
<code>target_generator</code> backwards.</p>
<p>Best.</p>
<p>K. Frank</p>

David_Gomez: <p>Thank you so much <a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/kfrank">@KFrank</a> for your help and for taking the time to answer so clearly and helping me understand not only the solution but the problem itself. This helped me a lot, and for a first experience using forums this was excellent.</p>
<p><a class="mention" href="/u/kfrank">@KFrank</a> The target_generator has a custom GradientInversor  function which is supposed to invert the gradients at the beginning of the backwards pass. This was intended to make the error of the target_generator -= the error of the discriminator.</p>
<pre><code class="lang-python">class GradientInversor(torch.autograd.Function):
    '''Gradient inversor to make the feature extractor loss
    the negative of the discriminator loss.'''
    @staticmethod
    def forward(self, x):
        return x
    
    @staticmethod
    def backward(self, grad_output):
        # Retorna el gradiente inverso
        return -grad_output
</code></pre>
<p>I was worried about this approach and I think the solution given makes a lot of sense, thank you so much, this will help me a lot for my Masters Thesis.</p>

KFrank: <p>Hi David!</p>
<aside class="quote no-group" data-username="David_Gomez" data-post="4" data-topic="182811" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/david_gomez/48/61600_2.png" class="avatar"> David_Gomez:</div>
<blockquote>
<p>The target_generator has a custom GradientInversor  function which is supposed to invert the gradients at the beginning of the backwards pass. This was intended to make the error of the target_generator -= the error of the discriminator.</p>
</blockquote>
</aside>
<p>Well that’s different.  Using your <code>GradientInversor</code> is a logically sound scheme<br>
to train <code>target_generator</code> in the “right” direction.  I think the approach that I<br>
outlined in my previous post where you first optimize <code>target_generator</code> and<br>
then call <code>source_loss.backward()</code> should therefore work.</p>
<p>Having said that, using <code>GradientInversor</code> trains `target_generator with a loss<br>
function, in essence, of:</p>
<pre data-code-wrap="python"><code class="lang-python">target_lossBNegative = -loss(discriminator(target_outputs), target_labelsB)
</code></pre>
<p>(where <code>target_labelsB</code> are zeros).</p>
<p>On the other hand (without <code>GradientInversor</code>), my training proposal trains<br>
<code>target_generator</code> with</p>
<pre data-code-wrap="python"><code class="lang-python">target_lossA = loss(discriminator(target_outputs), target_labelsA)
</code></pre>
<p>(where <code>target_labelsA</code> are ones).</p>
<p>While <code>target_lossA</code> and <code>target_lossBNegative</code> play the same role and have<br>
some similarities, they are not equal to one another.</p>
<p>I have never tested the two against one another, but my intuition tells me that<br>
<code>target_lossA</code> is likely to work better because it has a logarithmic divergence<br>
when <code>target_generator</code>  is not doing well in fooling <code>discriminator</code>.  This sort<br>
of logarithmic divergence seems to me to be very helpful in effective training.</p>
<p>If you have the time and energy, it might be useful to compare the two methods<br>
head-to-head on your specific concrete problem and see if one or the other is<br>
clearly superior.</p>
<p>Good luck!</p>
<p>K. Frank</p>

louvere_sam: <p>What you’ve done here gives me an error unless I specify retain_graph=True for target loss. However you’ve specified that it is not necessary, can you please explain?</p>
<pre><code class="lang-auto"># Process each input with corresponding generator
         source_outputs = source_generator(source_inputs)[0]
         target_outputs = target_generator(target_inputs)[0] 

         # Compute discriminator
         source_labels = torch.Tensor(np.array([np.ones(1) for _ in range(len(source_inputs))])).to(device)
         target_labels = torch.Tensor(np.array([np.zeros(1) for _ in range(len(target_inputs))])).to(device)

         # Zero grads
         generator_optimizer.zero_grad()
         discriminator_optimizer.zero_grad()

         target_loss = loss(discriminator(target_outputs), target_labels)

         # Do backpropagation for generator and PART OF discriminator
         target_loss.backward()
         generator_optimizer.step()
         source_loss = loss(discriminator(source_outputs), source_labels)

         # Do backpropagation for THE REST OF discriminator
         source_loss.backward()
         discriminator_optimizer.step()
</code></pre>