Why not switching between train and eval mode between the networks in DCGAN?

ZimoNitrome: <p>Looking at the <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html#training" rel="noopener nofollow ugc">PyTorch implementation of DCGAN</a> but also GANs in general…</p>
<p>To me it would seem intuitive to set G in train and D in eval when training G, and vice versa while training D. However, I don’t see this done in any GAN implementations.</p>
<p>Not switching to train/eval mode, it would seem that doing backprop on D(G(noise)) would train both D and G simultaneously. Is this incorrect?</p>
<p>Furthermore, trying to switch between eval and train like I suggested above seems to destabilize the GAN, where D gets 100% accuracy for real samples but 0% accuracy for fake samples.</p>
<p>Maybe .eval() makes D too powerful or makes G too weak.<br>
Maybe freezing D and G with <code>param.requires_grad = False</code> is better? Trying that out seems to make little difference if any to the original training scheme though.</p>

ptrblck: <aside class="quote no-group" data-username="ZimoNitrome" data-post="1" data-topic="131746">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zimonitrome/48/27591_2.png" class="avatar"> ZimoNitrome:</div>
<blockquote>
<p>Not switching to train/eval mode, it would seem that doing backprop on D(G(noise)) would train both D and G simultaneously. Is this incorrect?</p>
</blockquote>
</aside>
<p>No, since <code>model.eval()</code> and <code>model.train()</code> do not change the gradient calculation or the backpropagation in general. Instead the behavior of certain layers is changed. E.g. dropout layers are disabled during <code>eval()</code> and batchnorm layers will use their internal running stats to normalize the input activation.</p>
<aside class="quote no-group" data-username="ZimoNitrome" data-post="1" data-topic="131746">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zimonitrome/48/27591_2.png" class="avatar"> ZimoNitrome:</div>
<blockquote>
<p>Furthermore, trying to switch between eval and train like I suggested above seems to destabilize the GAN, where D gets 100% accuracy for real samples but 0% accuracy for fake samples.</p>
</blockquote>
</aside>
<p>I think this is the reason why this particular tutorial keeps both models in <code>train()</code> mode.</p>
<aside class="quote no-group" data-username="ZimoNitrome" data-post="1" data-topic="131746">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zimonitrome/48/27591_2.png" class="avatar"> ZimoNitrome:</div>
<blockquote>
<p>Maybe freezing D and G with <code>param.requires_grad = False</code> is better? Trying that out seems to make little difference if any to the original training scheme though.</p>
</blockquote>
</aside>
<p>I don’t think this would make any difference, since the output of the generator is already detached when the discriminator is trained as seen <a href="https://github.com/pytorch/examples/blob/cedca7729fef11c91e28099a0e45d7e98d03b66d/dcgan/main.py#L239">here</a>.</p>

ZimoNitrome: <p>Thanks for clearing up some points for me ptr <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Further, like you said, the reason for not freezing G is because we detatch its output when training D to discriminate fake samples as fake. So freezing G at any time would be pointless.</p>
<p>However:</p>
<aside class="quote no-group" data-username="ZimoNitrome" data-post="1" data-topic="131746">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zimonitrome/48/27591_2.png" class="avatar"> ZimoNitrome:</div>
<blockquote>
<p>Maybe freezing D and G with <code>param.requires_grad = False</code> is better? Trying that out seems to make little difference if any to the original training scheme though.</p>
</blockquote>
</aside>
<p>While this was my original sentiment, some further training by freezing D <a href="https://github.com/pytorch/examples/blob/cedca7729fef11c91e28099a0e45d7e98d03b66d/dcgan/main.py#L250" rel="noopener nofollow ugc">here</a> and unfreezing D <a href="https://github.com/pytorch/examples/blob/cedca7729fef11c91e28099a0e45d7e98d03b66d/dcgan/main.py#L222" rel="noopener nofollow ugc">here</a> seems to speed up the “convergence”.</p>
<p>If we look at the code I think this makes sense. What the DCGAN code does is essentially:</p>
<p>Train D that real → 1</p>
<p>Train D that fake → 0</p>
<p>Train D(G) that fake → 1</p>
<p>In one iteration it seems like we backprop D so that it should learn that fake → 0 but also that fake → 1 (in the case where we still have the graph of fake being produced by G). Maybe this would make D in general learn that fake → 0.5 or 0.25 or something like that. By freezing D at said points this might be mitigated.</p>
<p>I am currently at work so I can’t share trainig results but this does indeed look like an improvement.</p>
<p>On another note, can we still train networks then even if they are in eval mode?</p>

ptrblck: <aside class="quote no-group" data-username="ZimoNitrome" data-post="3" data-topic="131746">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zimonitrome/48/27591_2.png" class="avatar"> ZimoNitrome:</div>
<blockquote>
<p>I am currently at work so I can’t share trainig results but this does indeed look like an improvement.</p>
</blockquote>
</aside>
<p>Freezing the parameters of the discriminator could yield a speed improvement but shouldn’t change the convergence since the gradients in <code>D</code> are zeroed out in each iteration (but you are right that the gradients are computed without being used).</p>
<aside class="quote no-group" data-username="ZimoNitrome" data-post="3" data-topic="131746">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zimonitrome/48/27591_2.png" class="avatar"> ZimoNitrome:</div>
<blockquote>
<p>On another note, can we still train networks then even if they are in eval mode?</p>
</blockquote>
</aside>
<p>Yes, that’s possible.</p>

ZimoNitrome: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="131746">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>but shouldn’t change the convergence since the gradients in <code>D</code> are zeroed out in each iteration</p>
</blockquote>
</aside>
<p>Aha, so <a href="https://github.com/pytorch/examples/blob/cedca7729fef11c91e28099a0e45d7e98d03b66d/dcgan/main.py#L251-L253" rel="noopener nofollow ugc">this backprop</a> is not affecting D’s parameters because we don’t call <code>optimizerD.step()</code> after it?</p>

ptrblck: <p>Yes, exactly and in the new iteration the gradients in <code>netD</code> will be zeroed <a href="https://github.com/pytorch/examples/blob/cedca7729fef11c91e28099a0e45d7e98d03b66d/dcgan/main.py#L224">here</a>. So you might see a speed difference but shouldn’t see any difference in the parameter updates or convergence.</p>