VGG Perceptual Loss for really High Definition images

mohit117: <p>Hello,</p>
<p>The <a href="https://arxiv.org/abs/1603.08155" rel="nofollow noopener">perceptual loss</a> has become very much prevalent with an example shown in this <a href="https://github.com/pytorch/examples/blob/d91adc972cef0083231d22bcc75b7aaa30961863/fast_neural_style/neural_style/vgg.py#L7" rel="nofollow noopener">code</a>. However mostly I see people using VGG16 and not VGG19. This could be because generally people use low to medium resolution images such as 400x600 and so the depth of VGG16 may be sufficient. However the output of my denoising network is a high definition image which is <code>2048 x 4096</code> and has perfectly registered corresponding ground truth image. Of course at the time of training I just use a patch of 512 x 512 and perceptual loss will be computed over this patch only.</p>
<p>So I am not clear on the following issue regarding which will be the best practice:-</p>
<ul>
<li>
<p>Use VGG16 or VGG19. Since VGG19 has more layers I naively believe that it should be used although most implementations on GitHub still use VGG16.</p>
</li>
<li>
<p>Even in VGG16 people do not use all the layers. Mostly middle layers. But since I want my restored image to be an identical replica of the ground truth, in the best possible case can I naively use L1norm difference on all max pool layers of VGG? Or should I just use deep layers and skip the shallow layers or just use the middle layers?</p>
</li>
<li>
<p>Also many implementations such as <a href="https://gist.github.com/alper111/8233cdb0414b4cb5853f2f730ab95a49" rel="nofollow noopener">this one</a> resize the image to 244 x 244. But since mine is a much larger image; bilinearly interpolating <code>512 x 512</code> images to this small resolution may be problematic. So what is better? Passing the full resolution image through VGG or the downsized version for computing perceptual loss.</p>
</li>
</ul>
<p>Thank you very much.</p>

Kushaj: <p>There is not much benefit in the last layers. As these layers become more and more task specific, while we want something that can guide our model to generalize. Also, most of the basic things in the image is easily incorporated from low-to-middle layers of the model. But if you want, you can use last layers, but it is more of a diminishing returns thing.</p>
<p>Also, this also confuses me why people use VGG16, whereas we have much better classification models. One reason that I once found was because it just works and so nobody bothers to change it, as you would have to choose the layers from where you want features if you use a new architecture.</p>

mohit117: <aside class="quote no-group" data-username="Kushaj" data-post="2" data-topic="82760">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kushaj/48/58838_2.png" class="avatar"> Kushaj:</div>
<blockquote>
<p>There is not much benefit in the last layers. As these layers become more and more task specific</p>
</blockquote>
</aside>
<p>Thankyou for this understanding. I now feel to take loss from shallow and middle layers only.</p>
<blockquote>
<p>as you would have to choose the layers from where you want features</p>
</blockquote>
<p>The famous paper <a href="https://arxiv.org/pdf/1603.08155.pdf" rel="noopener nofollow ugc">Perceptual Losses for Real-Time Style Transfer<br>
and Super-Resolution</a> has the following diagram</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c3a56320aba1ca85057bcf5e052aeea30a59603.png" alt="Screenshot from 2020-05-26 08-56-40" data-base62-sha1="41Im0xffE07A577B5aX7DLskwBd" width="585" height="202"></p>
<p>According to this for content loss <strong>relu3_3</strong> is used but the in the description the paper says,</p>
<blockquote>
<p>For all style transfer experiments we compute feature reconstruction loss at layer relu2_2</p>
</blockquote>
<p>So can you please additionally help me which one will be more suitable for denoising task when perfectly aligned GT is available.</p>
<p>Thankyou<br>
Mohit</p>

Kushaj: <p>I think it comes to experimentation. Hard to say anything in deep learning without training a model.</p>