Perceptual Loss with Vgg19 and normalization

spnova12: <p>Hello.<br>
My question is about image reconstruction and Perceptual Loss and normalization.<br>
The question is a little long, but I think this is an interesting question if you are interested in image reconstruction and Perceptual Loss and feature extraction.</p>
<p>I am making Image restoration model.<br>
The model restores the image from the damaged image to the original image.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/a/a7658739a84e86d7761e18d49d4f11f988f534b0.jpg" alt="ori" data-base62-sha1="nSRgarljsVUmaA3lFM4kGmruPXq" width="432" height="432"><br>
[damaged image]</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/e/eadef4a3c9a8ed62ca0082fa9bb0acd728a96c84.png" alt="art" data-base62-sha1="xvLpjwXRIw4LmUZwVHeK5QQJQtm" width="432" height="432"><br>
[original image]</p>
<p>To do it well, I decided to use perceptual Loss.<br>
I use pytorch’s vgg19 as the feature extractor.<br>
I normalize image [0:255] to [-1:1]  with the input of my model, because i am using tanh() activate function.<br>
The pytorch’s vgg19 model is using [0:1] image.<br>
So I normalize it [-1:1] to [0:1]  just before entering the Perceptual Loss.<br>
But I found something strange in this part.<br>
I observed that the training results differed depending on where they were normalized.<br>
Below link is the code.</p>
<aside class="onebox githubgist" data-onebox-src="https://gist.github.com/spnova12/ab4aa294673eb5e1af0c99995ef68add">
  <header class="source">

      <a href="https://gist.github.com/spnova12/ab4aa294673eb5e1af0c99995ef68add" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/spnova12/ab4aa294673eb5e1af0c99995ef68add" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/spnova12/ab4aa294673eb5e1af0c99995ef68add</a></h4>

  <h5>pytorch 실험 002</h5>
  <pre><code class="">    class FeatureExtractor(nn.Module):
        def __init__(self, cnn, feature_layer=11):
            super(FeatureExtractor, self).__init__()
            self.features = nn.Sequential(*list(cnn.features.children())[:(feature_layer + 1)])

        def normalize(self, tensors, mean, std):
            if not torch.is_tensor(tensors):
                raise TypeError('tensor is not a torch image.')
            for tensor in tensors:
                for t, m, s in zip(tensor, mean, std):</code></pre>
   This file has been truncated. <a href="https://gist.github.com/spnova12/ab4aa294673eb5e1af0c99995ef68add" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I’ve marked the number (1) and (2) on the document.<br>
That (1) and (2) mean normalization position.<br>
And I found that the model’s result differed when doing normalization in 1) and in (2).</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/9/9c8a5ea8a283c8b73e8b4e13484ff38174eeff65.png" alt="204_2000" data-base62-sha1="mkOWm6F6MGZ3y65lxWXqVD1miP3" width="432" height="432"><br>
This is the result when the normalization position is in (1) after sufficient training.<br>
I feel something very bright in the picture…</p>
<p>(<img src="https://discuss.pytorch.org/uploads/default/original/2X/e/ed75b6312959e8a4bfc61ea9f366e1642076841c.jpg" alt="204_5000" data-base62-sha1="xSFm6RvngDwD8PhGkJWlANFiCSg" width="432" height="432"><br>
This is the result when the normalization position is in (2) after sufficient training.<br>
This result is closer to the original than the (1) result.</p>
<p>What is the difference between (1) and (2)?</p>
<p>When (1), normalization is in  nn.Module.<br>
When (2), normalization is outside nn.Module.<br>
I guess this is the reason…?</p>
<p>Thank you.</p>

gauku: <p>I find both to be the same and can’t think of any reason why doing this normalization at (1) or (2) should matter!<br>
Did you figure out some other reason for this behavior later?</p>