DCGAN mode collapse into geometric blobs

username: Topsoil
cooked: <p>I have a DCGAN (code available here: <a href="https://github.com/Grubzerusernameisavailable/DCGAN-pytorch/blob/master/pytorch_gan.ipynb" rel="noopener nofollow ugc">https://github.com/Grubzerusernameisavailable/DCGAN-pytorch/blob/master/pytorch_gan.ipynb</a>), but it has a big problem: it doesnt work. In the beginning if the generator is fed some random noise, outputs are also a uniform noise as expected, but at least they change with whatever we input in. But almost immidiately after training starts it falls into something that looks like a mode collapse, but worse: images not only stay almost completly identical no matter the input (they may have some variance, but it is imperceptible for me), but they also have no traits of original dataset (256 * 256 rgb living room images):<br>
Here is how typical output looks (from one of the experiments i ran with reduced (64*64 RGB) resolution):</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc22903a0f7f3005c5ee07a9d8cd03526363a9d6.png" alt="Без названия (3)" data-base62-sha1="t7RsfEmulllP8ELL2u0vRDWTYl8" width="231" height="231"></p>
<p>And loss from one of the runs, sampled each batch (graph looks kinda the same, no matter what i do):<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4d32b0ca91932ff3f1336560ec768b48f0e11576.png" alt="Без названия (2)" data-base62-sha1="b0VmTE3O3DouYF9h8BRTIdaJLzE" width="378" height="248"></p>
<p>Things that i tried to fix it:</p>
<ul>
<li>Deeper/shallower generator. Change structure of generator to 16-&gt;3 or 128-&gt;3 Conv2d channels at output, no effect</li>
<li>Change first linear layer output depth and resolution: all kinds of combinations between 2x2, 4x4, and 8x8 resolutions and 256,512, 1024 and 2048 depth</li>
<li>Varying input noise space, 50,100,300</li>
<li>Feeding noise directly through .view omitting linear layer as input (noise size 65536 → 1024,8,8)</li>
<li>Vary discriminator depth, dropout values and channels at Conv2d layers</li>
<li>Playing around with loss functions in a despair</li>
</ul>
<p>None of it worked. One behavior that i noticed is a “blobby” nature of some outputs, where output would typically have a blob of one color with a background of another color, and a high-res grid pattern overlayed on top of that, colors of which change in the “blob” and backgground</p>
<p>I am kinda new to ML community, and might be missing some key element, but this amount of absence of any change in behavior no matter what i do, is starting to drive me kinda crazy, help me please</p>

username: simaiden
cooked: <p>Have you taken a look to <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="nofollow noopener">DCGAN Pytorch tutorial</a> ?</p>

username: Topsoil
cooked: <p>Even though i wrote my dcgan from another tutorial, yeah, i read this one too, verified that my training loop is ok, found a method for random weight initialization, and remembered that i should BachNorm2d my layers. Still didnt help though</p>

username: simaiden
cooked: <p>Maybe the input are not normalized as the Generator output? ([-1,1] using tanh) .</p>

username: Topsoil
cooked: <p>done that, ported it from my MNIST tensorflow GAN as the first measure</p>

username: simaiden
cooked: <p>I noticed when you plot the image you don’t do <code>gen.eval()</code>. This is important because of the BatchNorm. Also why using dropout with so low p? Have you tried with p=0.5 , or using BatchNorm in both generator and discriminator?</p>

username: Topsoil
cooked: <p>I thought that bad generator performance was due to weak discriminator, so i tried making it ‘stronger’ by reducing how much it forgets during training</p>
<p>Completly forgot about .eval, ill try that tomorrow, and Discriminator BatchNorm too, will post results here</p>

username: Topsoil
cooked: <p>Alright, i’ve done some tests, and results are as following:<br>
on the same architecture as before, no changes using .eval before image sampling, though i kept using it since that is how i should’ve done it from the beginning</p>
<p>Also i’ve done three more tests: discriminator dropout 0.4, discriminator batchnorm, and both at the same time, here is how they look, results from:</p>
<p>Dropout 0.4 with no batchnorm 10 epochs:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/9/89d548cf2862c6ccb521228c8ff4415a5eafba6f.png" alt="10 ep dropout 0.4" data-base62-sha1="jFkn8yIeIeXw1XG4t9DYfdkIMjl" width="251" height="251"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/7/87d631d38f44f8edcce716f30ef1a013dadfeeec.png" alt="10ep 0.4 dropout" data-base62-sha1="jnFmQ7DG13VeoqYWBaShIvesyMA" width="372" height="248"></p>
<p>Batchnorm with 0.4 dropout 36 epochs:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cff16496a317398b5ce9f9090f9438de7ee013b4.png" alt="36ep 0.4 dropout  BN" data-base62-sha1="tFy18u55O58jyCmMe1nWYHetxvm" width="231" height="231"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1eeedcfdb7e196eb0a10f55c0baade4c3eacb3b1.png" alt="36ep 0.4 dropout  BN loss" data-base62-sha1="4pE5lzS2zgO52kgqpRhEYGHmi2J" width="372" height="248"></p>
<p>Batchnorm only 36 epochs:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/a/3ae26beace7458822c83aa0e4c47f33f09c9444b.png" alt="36ep only BN" data-base62-sha1="8oUNydeUFOAANZ3wJKrTPa1rZSb" width="231" height="231"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9a695ea25f158d60cbdd39f3a28548f99b7fe21e.png" alt="36ep only BN loss" data-base62-sha1="m1ZhBn03NFdO0siMuqeRzfiN9VQ" width="372" height="248"></p>
<p>I feel like batchnorm with no dropout works the best, even at 10th epoch:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/9/19cf59dedef1f1167c517de1a9042b6bd50a55b8.png" alt="bnonly 10ep" data-base62-sha1="3GkbqEeMxq3afQXjRsIfK0BszrO" width="231" height="231"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f08cb23bb59c05b9177db52df73baa2b52b69773.png" alt="bnonly 10ep loss" data-base62-sha1="yk023wUuOwjzxYqfiujtJlE6j2X" width="375" height="248"></p>
<p>But the problem of mode collapse still remains unsolved. No matter the input, generator creates the same pictures (up to pixel-by-pixel similarity). Any remedeis for that? I’ve heard that Unrolled GANs solve this problem, but maybe there is something else? Also, i can see the problem of big repetitive patterns. Is this something similar to checkerboard artifacts that transpose layers produce, just carried over from earlier layers?</p>
<p>Gif from batchnorm only run<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/0/7098d37dcbeea11bc60ffc0336350112d07c6ba9.gif" alt="dcgan bn only 36 ep" data-base62-sha1="g44U7w4MQu6oqYxQBXqJi3aH86J" width="128" height="128" class="animated"></p>

username: Topsoil
cooked: <p>So, after some more testing, i came to a conclusion:</p>
<p>My model (and model from dcgan pytorch tutorial i switched to) did not have enough capacity for dataset i was using. After switching to CelebA dataset, model worked as is it was suppose to. My theory is that because of high variety of images in living room dataset, different environments, camera angles, etc. and low model capacity for this task, it was forced into creating blurry approximations of them, hardly capturing even a color scheme, and falling into mode collapse, since size of latent vector and model capacity were not big enough to capture all parameters that it needed.</p>
<p>In this tutorial: <a href="https://machinelearningmastery.com/practical-guide-to-gan-failure-modes/" rel="nofollow noopener">https://machinelearningmastery.com/practical-guide-to-gan-failure-modes/</a><br>
Author decreases the size of latent vector from 100 to 1, which causes mode collapse.</p>
<p>Ideas to take with me to my other experiments:</p>
<ul>
<li>Start with simple datasets, for example, CelebA, and test on them. If the problem is in the model’s architecture, it would show it here</li>
<li>If model experiences mode collapse, increase its capacity, and size of latent vectors</li>
<li>In my case i’ve noticed a phenomenon of “bouncing” in the beginning of training process: images start as noise, then become more realistic (Sort of. There are some color patches, in case of CelebA dataset there is a blurry kinda-face figure in the center), but after that they come back to noise, then back to sort-of realistic images, and then going back and forth a few times, before settling at final result that improves over time. Not sure if it is normal, but in my case it went through that stage, and then worked as expected, hopefully someone can explain to me whether it is normal or not</li>
</ul>
<p>Thanks to everyone who helped me, problem is solved</p>