GAN loss backward

dragonHyeon: <p>Hello.</p>
<p>I was training CycleGAN.<br>
When I train discriminator,</p>
<p>case 1.</p>
<pre><code class="lang-auto">loss_D_A.backward()
loss_D_B.backward()
</code></pre>
<p>case 2.</p>
<pre><code class="lang-auto">loss_D = loss_D_A + loss_D_B
loss_D.backward()
</code></pre>
<p>Without thinking of retain_graph=True stuff, assuming this code doesn’t occur any error, are both case have same effects?</p>
<p>My question is that for case 2., does loss get accumulated twice (not just loss A. loss A + loss B for Discriminator A. Same case for Discriminator B.) so it update the gradient twice more then they should have been? Or does loss automatically affects only to its related tensors even though they are added like case 2.?</p>
<p>Thank you.</p>

KFrank: <p>Hi Hyeon!</p>
<aside class="quote no-group" data-username="dragonHyeon" data-post="1" data-topic="183214" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/e8c25b/48.png" class="avatar"> dragonHyeon:</div>
<blockquote>
<p>case 1.</p>
<pre><code class="lang-auto">loss_D_A.backward()
loss_D_B.backward()
</code></pre>
<p>case 2.</p>
<pre><code class="lang-auto">loss_D = loss_D_A + loss_D_B
loss_D.backward()
</code></pre>
</blockquote>
</aside>
<p>This will work properly.  <code>loss_D_A.backward()</code> accumulates the gradient<br>
of <code>loss_D_A</code> into whatever parameters <code>loss_D_A</code> depends on.<br>
<code>loss_D_B.backward()</code> then accumulates the gradient of <code>loss_D_B</code><br>
into its relevant parameters.</p>
<p>If <code>loss_D_A</code> and <code>loss_D_B</code> both depend on some of the same<br>
parameters, those parameters will have the sum of the two gradients<br>
accumulated into their <code>.grad</code> properties.</p>
<p>But gradients are linear – that is, the gradient of the sum is the sum of<br>
the gradients.  So accumulating the gradient of <code>loss_D_A + loss_D_B</code><br>
into the relevant parameters does, indeed, accumulate the sum of the<br>
two gradients into the parameters.  (But by doing so in one <code>.backward()</code><br>
call, it will be somewhat cheaper.)</p>
<aside class="quote no-group">
<blockquote>
<p>Without thinking of retain_graph=True stuff</p>
</blockquote>
</aside>
<p>Just to be clear, in case 1, if <code>loss_D_A</code> and <code>loss_D_B</code> share<br>
part of the same computation graph, you will have to use<br>
<code>loss_D_A.backward (retain_graph = True)</code> in order for<br>
<code>loss_D_B.backward()</code> to work.</p>
<p>(<code>retain_graph = True</code> won’t be needed for case 2 because there<br>
is only one <code>.backward()</code> call.)</p>
<p>Best.</p>
<p>K. Frank</p>

dragonHyeon: <p>Hello Frank!<br>
Thank you for your fast and kind reply!</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="183214">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>This will work properly. <code>loss_D_A.backward()</code> accumulates the gradient<br>
of <code>loss_D_A</code> into whatever parameters <code>loss_D_A</code> depends on.<br>
<code>loss_D_B.backward()</code> then accumulates the gradient of <code>loss_D_B</code><br>
into its relevant parameters.</p>
</blockquote>
</aside>
<p>So do you mean that also for case 2, it will accumulate the gradient of <code>loss_D_A</code> only into its relevant parameters and accumulate the gradient of <code>loss_D_B</code> only into its relevant parameters even if we call only one <code>.backward()</code> to sum of those two losses (<code>loss_D_A + loss_D_B</code>) but not separate <code>.backward()</code> call to each losses?</p>

KFrank: <p>Hi Hyeon!</p>
<aside class="quote no-group" data-username="dragonHyeon" data-post="3" data-topic="183214" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/e8c25b/48.png" class="avatar"> dragonHyeon:</div>
<blockquote>
<p>even if we call only one <code>.backward()</code> to sum of those two losses (<code>loss_D_A + loss_D_B</code>) but not separate <code>.backward()</code> call to each losses?</p>
</blockquote>
</aside>
<p>Yes, this is correct (with the proviso that you might need to use<br>
<code>retain_graph = True</code> in case 1).</p>
<p>You can easily test that cases 1 and 2 produce the same gradients.</p>
<p>Run case 1 and <code>clone()</code> the <code>.grad</code>s of all of the parameters that affect<br>
either <code>loss_D_A</code> or <code>loss_D_B</code> for future comparison.  Then repeat the<br>
loss computations from scratch (using the same data, of course) and<br>
run case 2.  You can now compare the <code>.grad</code>s from case 2 with those<br>
you saved from case 1 and you will see that they are equal (up to numerical<br>
round-off error).</p>
<p>Best.</p>
<p>K. Frank</p>

dragonHyeon: <p>Thank you so much.<br>
So kind and clear solution!</p>
<p>Have a nice day and always good luck Frank!</p>

LamKser: <aside class="quote no-group" data-username="dragonHyeon" data-post="1" data-topic="183214">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/e8c25b/48.png" class="avatar"> dragonHyeon:</div>
<blockquote>
<pre><code class="lang-auto">loss_D = loss_D_A + loss_D_B
loss_D.backward()
</code></pre>
</blockquote>
</aside>
<p>what happens if we use:</p>
<pre><code class="lang-auto">loss_D = (loss_D_A + loss_D_B) / 2
loss_D.backward()
</code></pre>
<p>instead of <code>loss_D = loss_D_A + loss_D_B</code><br>
Does it work the same?</p>

KFrank: <p>Hi Lam!</p>
<aside class="quote no-group" data-username="LamKser" data-post="6" data-topic="183214" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lamkser/48/64437_2.png" class="avatar"> LamKser:</div>
<blockquote>
<p>what happens if we use:</p>
<pre><code class="lang-auto">loss_D = (loss_D_A + loss_D_B) / 2
loss_D.backward()
</code></pre>
<p>instead of <code>loss_D = loss_D_A + loss_D_B</code></p>
</blockquote>
</aside>
<p>This reduces <code>loss_D</code> by a factor of two so, because the gradients of the<br>
loss are linear in the loss, the gradients computed by <code>loss_D.backward()</code><br>
will also be reduced by a factor of two.</p>
<p>(All the previous discussion about how calling <code>.backward()</code> twice for two<br>
pieces of the loss and calling <code>.backward()</code> once for the combined loss<br>
being essentially the same is unaffected by your factor of two.)</p>
<p>Best.</p>
<p>K. Frank</p>