Artifact in GAN generated results

nile649: <p>Hi, in GAN results i am getting very weird pixel artifact patch in results… so wanted some ideas how can i get rid it off. Also, wanted to discuss reasons behind such artifacts as well.!<img src="https://discuss.pytorch.org/uploads/default/original/2X/b/bd71a51787caa0cb344ddf10e380961e3c0529f2.jpeg" alt="Epoch_(31)_(10000of30000)" data-base62-sha1="r1TIWMvpXboJycFCfgflEroirHc" width="392" height="132"><br>
There is a dark spot in the image.</p>
<p>Input image is 1st image, Generated Image is 2nd, and 3rd is GT.</p>
<p>Thank you,<br>
Nilesh Pandey</p>

pytorchnewbie: <p>I’ve had similar results …<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/79dc20041848bf3164f2b3a7298a55e390e864b8.png" data-download-href="https://discuss.pytorch.org/uploads/default/79dc20041848bf3164f2b3a7298a55e390e864b8" title="31%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/79dc20041848bf3164f2b3a7298a55e390e864b8_2_690x348.png" alt="31%20PM" data-base62-sha1="ho1o9ZnEqxWEJDKIapjbZFJxH2E" width="690" height="348" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/79dc20041848bf3164f2b3a7298a55e390e864b8_2_690x348.png, https://discuss.pytorch.org/uploads/default/optimized/2X/7/79dc20041848bf3164f2b3a7298a55e390e864b8_2_1035x522.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/7/79dc20041848bf3164f2b3a7298a55e390e864b8_2_1380x696.png 2x" data-dominant-color="9FA0A4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">31%20PM</span><span class="informations">1890×954 291 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Maybe this is simply due to the instability of the GAN? I’ve heard this downside thrown around a lot when discussing GANs</p>

Nilesh_Pandey1: <p>I have seen such artifacts only when discriminator gets trained faster, and it causes model collapse. But, in my case network is still being trained normally with the artifact as well.</p>
<p>The other reason could be overfitting, celebA dataset is well aligned to eyes and nose in same position in almost all images.</p>
<p>while testing I did realize that my GAN learns spatial key points descriptors, it does SOTA results on testing but fails on being presented with images outside from dataset, which gives hint maybe the GAN is over fitting in someway.</p>
<p>On changing dataset loader slightly, the pixel corruption does changes as well.</p>
<p>I am still not sure for the cause of pixel corruption.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/9a85b27c957dba17cf9d99f3e0430e5ed1320d5b.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/9a85b27c957dba17cf9d99f3e0430e5ed1320d5b" title="Screenshot%20from%202019-06-07%2023-57-43"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/9/9a85b27c957dba17cf9d99f3e0430e5ed1320d5b_2_690x388.jpeg" alt="Screenshot%20from%202019-06-07%2023-57-43" data-base62-sha1="m2XYtt2t3Fc3J2tN8oakByoPj8T" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/9/9a85b27c957dba17cf9d99f3e0430e5ed1320d5b_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/9/9a85b27c957dba17cf9d99f3e0430e5ed1320d5b_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/9/9a85b27c957dba17cf9d99f3e0430e5ed1320d5b_2_1380x776.jpeg 2x" data-dominant-color="2E2523"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-06-07%2023-57-43</span><span class="informations">1920×1080 221 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

pytorchnewbie: <p>Doesn’t model collapse just mean that many images in the starting domain get collapsed to the same image in the output domain? X1-&gt;Y1 and X2-&gt;Y1 etcetera? I don’t follow how modal collapse has to do with the artifacts… sorry! Could you please explain?</p>

Balamurali_M: <p>For the task of inpainting, various loss functions have been proposed till now. In the images shown, the black box artifact is present in the area where image data is already available. So ideally you should not get that kind of artifact. I would be interested to know the loss function used. If it was simple L1 between the whole predicted image and gt image, I believe this can happen. One way to handle it using loss function is you calculate loss only for regions which are being filled. So that data which is already available can be used. Something like (mask*predicted) - (mask * GT).</p>

Nilesh_Pandey1: <ol>
<li>Generator starts producing images with artifacts and corrupt pixels, and its not learning anything new.</li>
<li>Generator starts producing garbage.</li>
<li>Generator starts producing average of the result in all cases.</li>
</ol>

Nilesh_Pandey1: <p>True, simple “L1 between the whole predicted image and gt image” function won’t work here, It is simple L2 loss function.</p>

Balamurali_M: <p>You should use some kind of context aware gan to get better inpainting results. The setup you mentioned looks very basic.</p>

pytorchnewbie: <p>Is this a list of things that people consider to be included in “mode collapse”? I read that it’s <span class="hashtag">#3</span> but didn’t see the first two included in mode collapse…</p>