Artefacts when using a perceptual loss term

PatrickKr: <p>Hi everybody,</p>
<p>I have a question regarding some kind of checkerboard artefacts when using a perceptual loss function.<br>
You can see the artefacts in the following image, these tiny white dots, it looks like the surface of a basketball.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/574e2939b9573492084ef0381e323be4ced7b995.png" data-download-href="https://discuss.pytorch.org/uploads/default/574e2939b9573492084ef0381e323be4ced7b995" title="220310_Artefact_checkerboard"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/7/574e2939b9573492084ef0381e323be4ced7b995.png" alt="220310_Artefact_checkerboard" data-base62-sha1="cskZWQf9uha3OPqYum9IKXtc1VP" width="260" height="250" data-dominant-color="3B3F44"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">220310_Artefact_checkerboard</span><span class="informations">595×571 35.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>My model:</strong><br>
I’m using an encoder-decoder architecture.<br>
Downsampling is done with a <code>nn.Conv2d()</code> Layer with stride 2.<br>
Upsampling is done with a <code>nn.ConvTranspose2d()</code> Layer with stride 2.</p>
<p><strong>Loss function</strong><br>
First of all, these artefacts only appear when I’m using a perceptual loss term.<br>
Using only L1 or L2 no artefacts are visible.<br>
I’m using a VGG-19 perceptual loss term for which I’m using the output of the conv layer before the Relu activation function and the max pooling layer.</p>
<pre><code class="lang-auto">loss = vgg_loss + L1_loss
</code></pre>
<p><strong>What I’ve tried</strong><br>
I’ve tried different weights for the content and perceptual loss without any improvement.<br>
I’ve tried adding a total variation loss term without any improvement.<br>
(I used this implementation for the TVLoss on the generated image: <a href="https://discuss.pytorch.org/t/implement-total-variation-loss-in-pytorch/55574/2" class="inline-onebox">Implement total variation loss in pytorch - #2 by Xinxiang7</a>)</p>
<p><em>Update</em><br>
I’ve also tried a Multi-level Wavelet-CNN for better up- and downsampling without any improvement.<br>
[<a href="https://arxiv.org/abs/1805.07071" class="inline-onebox" rel="noopener nofollow ugc">[1805.07071] Multi-level Wavelet-CNN for Image Restoration</a>]</p>
<p>I’ve read this paper <a href="https://arxiv.org/abs/2002.02117" class="inline-onebox" rel="noopener nofollow ugc">[2002.02117] Fixed smooth convolutional layer for avoiding checkerboard artifacts in CNNs</a> but couldn’t find any code implementation. Since I don’t know how to implement the authors method I havent tried this approach. Does someone know how to implement the authors method?</p>
<p><strong>Does someone know the cause of these artefacts and how to solve them?</strong></p>
<p>Thank you very much,<br>
Patrick</p>

Srivathsa_Pasumarthi: <p>What is the image size you are inputting to get the VGG19 features? I found that if the image size is not close to the original vgg19 network (224, 224, 3), then it produces such artifacts</p>