3 backprop vs 1 backprop for gan discriminator training

40uf411: <p>I am trying to train a 3D gan using 2D discriminator that take slices of the original data.<br>
And wanted to get your opinion on two points:<br>
1- is it better to have 3 discriminators, one per plane. Or a single discriminator and takes the embedding of the plane as input.<br>
2-my current implementation is something like this:</p>
<ul>
<li>disc real training backprop</li>
<li>disc fake training backprop</li>
<li>r1 regularisation backprop</li>
<li>gen training backprop</li>
</ul>
<p>What would the expected effect of summing up the losses and doing one back prop per model? which method is better.</p>

Dhia-naouali: <p>Hii</p>
<p>for the first question<br>
I  would say each strategy has it’s drawbacks, if you went with 3 discriminators that will cost more compute and you’ll have to carefully balance them else you may have one discriminator dominating the others, given the GANs reputation, you’ll need careful tuning<br>
if you chose to use one discriminator, how you design your aggregation will matter most</p>
<p>one idea I would suggest giving a shot:<br>
for each layer of the 2D discriminator</p>
<ul>
<li>pass all slices through the discriminator layer,</li>
<li>pass all slices as a 3D block through an NdLinear block, acting as a communication layer</li>
</ul>
<p>repeat<br>
I think NdLinear layers offer a good trade-off in this case, you’ll have regular communication point beetween D layers with minimal params (check the <a href="https://arxiv.org/pdf/2503.17353" rel="noopener nofollow ugc">paper</a>, of course other intermediate layers can do the trick too</p>
<p>for your second question<br>
I would recommend summing all losses and back propagating once per model<br>
(real_loss + fake_loss + r1_reg).backward(); gen_loss.backward()</p>

KFrank: <p>Hi Dhia (and Ali)!</p>
<aside class="quote no-group" data-username="Dhia-naouali" data-post="2" data-topic="221670" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dhia-naouali/48/77193_2.png" class="avatar"> Dhia-naouali:</div>
<blockquote>
<p>I would recommend summing all losses and back propagating once per model<br>
(real_loss + fake_loss + r1_reg).backward(); gen_loss.backward()</p>
</blockquote>
</aside>
<p>As written, this won’t work (at least not without further explanation of the intended details).</p>
<p>When you call <code>fake_loss.backward()</code>, you will populate <code>gen</code>’s parameters with gradients<br>
from <code>fake_loss</code> that won’t be correct for training <code>gen</code>.  This will also free the computation<br>
graph built during the forward pass through <code>gen</code>.  (Calling <code>gen_loss.backward()</code> will<br>
analogously populate <code>disc</code> with incorrect gradients.)</p>
<p>(However, you <em>can</em> call <code>(real_loss + fake_loss + r1_reg).backward()</code>, if done so <em>solely</em><br>
for the purpose of training <code>disc</code>, for example by calling <code>opt_disc.step()</code> after <code>disc</code>’s<br>
gradients have been populated .)</p>
<p>Best.</p>
<p>K. Frank</p>

Dhia-naouali: <p>thanks for pointing that out<br>
I didn’t mention the forward props<br>
here is what I meant in more details</p>
<h2><a name="p-473172-discriminator-step-1" class="anchor" href="#p-473172-discriminator-step-1"></a>Discriminator step</h2>
<pre data-code-wrap="python"><code class="lang-python"># forward
real_logits = D(real_samples.requires_grad_()) # for r1
with torch.no_grad():
    fake_samples = G(noise)
fake_logits = D(fake_samples)

real_loss = ...
fake_loss = ...
r1 = ...
D_loss = real_loss + fake_loss + r1

D.zero_grad()
D_loss.backward()
D_optimizer.step()
</code></pre>
<h2><a name="p-473172-generator-step-2" class="anchor" href="#p-473172-generator-step-2"></a>Generator step</h2>
<pre data-code-wrap="python"><code class="lang-python"># forward
fake_samples = G(noise)
fake_logits = D(fake_samples)

G_loss = ...

G.zero_grad()
G_loss.backward()
G_optimizer.step()
</code></pre>

40uf411: <p>Thanks Dhia and Frank,</p>
<p>1-So far I managed to have some decent results with 3 discriminators, though, not perfect.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e4c9ef0762e315f37ff1aed5ee0eb9785fedcc3.png" data-download-href="https://discuss.pytorch.org/uploads/default/2e4c9ef0762e315f37ff1aed5ee0eb9785fedcc3" title="depth_sample_2_epoch_57-5799"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e4c9ef0762e315f37ff1aed5ee0eb9785fedcc3.png" alt="depth_sample_2_epoch_57-5799" data-base62-sha1="6BA7D4M1LLRasocParizNrfvODN" width="332" height="332"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">depth_sample_2_epoch_57-5799</span><span class="informations">332×332 56.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
the images tend to have a lot of noise and they are not sharp; that on top of the ballencing issues, and params finetuning.<br>
Working with one model is not done yet.</p>
<blockquote>
<p>I think NdLinear layers offer a good trade-off in this case</p>
</blockquote>
<p>I am not sure about the use of NdLinear here, because the 3d images are quite big in size and multiply that by the batch size and you will fill the memory quit rapidely.</p>
<p>As for the second one, the tradeoff is that the original nvidia code was doing 3 backprops for the disc, which is memory intensive and you have to retain the computation graph in memory for a long time, but, you do three steps.</p>
<p>To my knowledge loss accum, should lead to the same direction but in a lesser magnitude. So, I was a bit confused about their choice. Maybe they want to have the multi disc step methodology but without skipping any generator training iteration?</p>