DCGAN strange losses despite visually good fakes

jeremyfix: <p>Hi,</p>
<p>I did some experiments reimplementing a DCGAN like for generating MNIST digits (full code at <a href="https://github.com/jeremyfix/gan_experiments" class="inline-onebox" rel="noopener nofollow ugc">GitHub - jeremyfix/gan_experiments</a>). It seems to me the generated digits are pretty good (see <a href="https://raw.githubusercontent.com/jeremyfix/gan_experiments/main/mnist.gif" rel="noopener nofollow ugc">https://raw.githubusercontent.com/jeremyfix/gan_experiments/main/mnist.gif</a>)</p>
<p>And indeed, even interpolating in the latent space seems to produce a large variety of digits (to produce the plot below, I generated 3 random noises and then interpolated the noise to feed in the generator) (see the github page. I cannot put more than 1 image or more than 2 links)</p>
<p>But, when it turns to plot the losses, these seems to me unexpected.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b965eb5aef485a4d447a7f4b88e45f1496ab8b4e.png" data-download-href="https://discuss.pytorch.org/uploads/default/b965eb5aef485a4d447a7f4b88e45f1496ab8b4e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b965eb5aef485a4d447a7f4b88e45f1496ab8b4e.png" alt="image" data-base62-sha1="qs6GYQ8SMLvuMtK1cy8nmQxYCkK" width="690" height="483" data-dominant-color="15051C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1500×1050 29.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>They are all consistent : a high generator loss (that I want to be minimized actually…), almost 90% accuracy for the discriminator. This indicates the discriminator is able to distinguish the fakes from the reals 90 % of the time but when I see the digits the generator generates, I see no obvious things and it seems to generate a variety of shapes which seems to indicate this is not a mode collapse.</p>
<p>Do you have any idea on this behavior ?</p>

jeremyfix: <p>A follow-up . I did try on other datasets (MNIST, FashionMNIST, EMNIST, SVHN), and below are generated samples on these datasets.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/d/1dee1928b5c57c894df4849d24d7d32a987cf21c.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1dee1928b5c57c894df4849d24d7d32a987cf21c" title="generated"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/d/1dee1928b5c57c894df4849d24d7d32a987cf21c_2_690x218.jpeg" alt="generated" data-base62-sha1="4gLY86m0SgVS7nOU0DNJLIfWvA8" width="690" height="218" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/d/1dee1928b5c57c894df4849d24d7d32a987cf21c_2_690x218.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/d/1dee1928b5c57c894df4849d24d7d32a987cf21c_2_1035x327.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/d/1dee1928b5c57c894df4849d24d7d32a987cf21c.jpeg 2x" data-dominant-color="6D6C6B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">generated</span><span class="informations">1339×424 205 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I would say that all the samples seem reasonably ok. And they all show similar behaviors in the metrics which I still find very surprising.</p>
<p>For example, if I plot the accuracy of the discriminator  (I would have expected it to be at 50% when the fakes look real):</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd88136b930f3e6350f900a19b759bec5f66f9dc.png" alt="critic_accuracy" data-base62-sha1="tkdqcA4W861pdjy4zYQPUY2cDuY" width="318" height="220"></p>
<p>And if I plot the loss of the generator (I would have expected it to be low when the generated samples look real):</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/7/f7518775e848a3d7457918057d12834bac7c8b5b.png" alt="generator_loss" data-base62-sha1="zhSBR5fhr4nCWNfAeI9Ikup5uht" width="310" height="210"></p>
<p>And finally, because I really find it funny (even impressive actually), if I take the model trained on FashionMNIST and SVHN and let me generate samples by interpolating in the latent space using 3 seeds (for the top-left, top-right and bottom-right).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b78017d01557f35ea1d944bcde66e079a19f59f2.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b78017d01557f35ea1d944bcde66e079a19f59f2" title="generated2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b78017d01557f35ea1d944bcde66e079a19f59f2_2_500x500.jpeg" alt="generated2" data-base62-sha1="qbjOxjLFNngH5JHXmMX3ff55gK6" width="500" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b78017d01557f35ea1d944bcde66e079a19f59f2_2_500x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/b/7/b78017d01557f35ea1d944bcde66e079a19f59f2.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/7/b78017d01557f35ea1d944bcde66e079a19f59f2.jpeg 2x" data-dominant-color="232323"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">generated2</span><span class="informations">602×602 126 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/0/105eb11881d006aa38701e82355a19a785cc1c37.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/105eb11881d006aa38701e82355a19a785cc1c37" title="generated2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/105eb11881d006aa38701e82355a19a785cc1c37_2_500x500.jpeg" alt="generated2" data-base62-sha1="2kOvYt2HzesRZE8TvsLA1J8VQW3" width="500" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/105eb11881d006aa38701e82355a19a785cc1c37_2_500x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/1/0/105eb11881d006aa38701e82355a19a785cc1c37.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/0/105eb11881d006aa38701e82355a19a785cc1c37.jpeg 2x" data-dominant-color="8A8278"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">generated2</span><span class="informations">682×682 183 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

jeremyfix: <p>A follow-up. I was suggested that the discriminator might be overfitting. Indeed, it seems to be the case.</p>
<p>I evaluated separately the cross entropy loss of the discriminator on the positive samples (coming the train split of the dataset), on the negative samples (generated by the generator) and on a validation positive samples (coming from the validation split of the dataset) :</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/2/9295ac2c7e282339261315c04f87f65b7d8d1d87.png" alt="Critic_p_loss" data-base62-sha1="kUKnIV0X5oZywivxH66dPM2nezJ" width="341" height="242"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/e/be26991d731f11f9a5fada883fbfb174e2ab87be.png" alt="Critic_n_loss" data-base62-sha1="r89pKYe43LlWp4NpXLdQNgrlEfQ" width="348" height="240"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/1/813fd232056402d5bef21f474113194c41147823.png" alt="Critic_v_loss" data-base62-sha1="irokrpD9fUHVI2VUFcQbM3WulVN" width="346" height="243"></p>
<p>I interpret this as the discriminator being overspecialized on the training set positive samples considering everything else as a negative (like a one class classifier) and therefore I suppose this is a failure since the discriminator would train the generator to generate only the samples that are exactly (or almost) similar to the samples from the training set.</p>
<p>To be honest, I have the feeling this is not exactly what is happening as interpolating in the latent space produces quite diverse images and, by the way, the generator loss is pretty high. For completeness, the generator loss and some generated samples are displayed below; Even though the generator loss is high , I feel like the samples are again not that bad.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/9/7976cfaa006d4edef735235c921c45fa57cc29a5.png" alt="gen_loss" data-base62-sha1="hkwkc4eoNrT2ovysea2u6iKhVWt" width="346" height="237"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/9/2971b852211d52b36880a4cbb3a185e69ba31987.jpeg" alt="gen_samples" data-base62-sha1="5UDdEISTDybZkPN7zbsWWgFwY6j" width="360" height="363"></p>
<p>For complete completeness, this is generated with <a href="https://github.com/jeremyfix/gan_experiments" class="inline-onebox" rel="noopener nofollow ugc">GitHub - jeremyfix/gan_experiments</a> , tag v2, and running <code>python3 main.py train --dataset SVHN --dropout 0.0 --wdecay 0.0</code></p>
<p>In the next post, soon, I will post some results when regularizing the discriminator.</p>