VGGPerceptualLoss in mixed precision mode

username: FilipAndersson245
cooked: <p>When i try to train my model using the nightly auto mixed precision mode i get this error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "src/train.py", line 101, in &lt;module&gt;
    init_phase(
  File "src/train.py", line 48, in init_phase
    loss = content_loss(fakes, images)
  File "E:\program\anaconda\envs\torch_n\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "D:\C_GAN\src\loss.py", line 92, in forward
    l = self.base_loss(x, y)
  File "E:\program\anaconda\envs\torch_n\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "E:\program\anaconda\envs\torch_n\lib\site-packages\torch\nn\modules\loss.py", line 813, in forward
    return F.smooth_l1_loss(input, target, reduction=self.reduction)
  File "E:\program\anaconda\envs\torch_n\lib\site-packages\torch\nn\functional.py", line 2581, in smooth_l1_loss
    ret = _smooth_l1_loss(input, target)
  File "E:\program\anaconda\envs\torch_n\lib\site-packages\torch\nn\functional.py", line 2557, in _smooth_l1_loss
    return torch.where(t &lt; 1, 0.5 * t ** 2, t - 0.5)
RuntimeError: expected scalar type float but found struct c10::Half
</code></pre>
<p>The remaining code is quite similar to that of <a href="https://pytorch.org/docs/stable/notes/amp_examples.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/amp_examples.html</a></p>
<p>The loss is found here: <a href="https://gist.github.com/brucemuller/37906a86526f53ec7f50af4e77d025c9" rel="nofollow noopener">https://gist.github.com/brucemuller/37906a86526f53ec7f50af4e77d025c9</a></p>
<p>How should i change the loss to make it work correctly?</p>

username: ptrblck
cooked: <p>I cannot reproduce this issue using your criterion and this minimal code snippet:</p>
<pre><code class="lang-python">
criterion = VGGPerceptualLoss().cuda()

x = torch.randn(1, 3, 24, 24, device='cuda', requires_grad=True)
y = torch.randn(1, 3, 24, 24).cuda()

with torch.cuda.amp.autocast():
    loss = criterion(x, y)
loss.backward()

print(x.grad)
</code></pre>
<p>Could you check my code and compare it to yours, which is raising this issue?</p>

username: FilipAndersson245
cooked: <p>Thanks for the reply.<br>
Upon rerunning the code from op the code does not crash for unknown reasons,<br>
but when we apply autocast it produces unexpected results.</p>
<p>This is the essence of the training and evaluation code.</p>
<pre data-code-wrap="python"><code class="lang-python">scaler = torch.cuda.amp.GradScaler()
for epoch in range(epochs):
    for images in train_photo_dataloader:
        images = images.to(device)

        optimizer.zero_grad()

        with torch.cuda.amp.autocast():
            fakes = G(images)
            loss = content_loss(fakes, images)
        scaler.scale(loss).backward()

        scaler.step(optimizer)

        scaler.update()

    G.eval()
    with torch.no_grad():
        fakes = G(test_images)
        loss_test = content_loss(fakes, test_images)

        logger.log_images("Test images output", fakes, normalize=False)
    G.train()
</code></pre>
<p>With the logger using make_grid like this</p>
<pre data-code-wrap="python"><code class="lang-python">class ExperimentLogger(object):
    def __init__(self, exp_id: str, init_phase: bool):
        self.global_step = 0
        self.exp_id = exp_id
        log_dir = f"./logs/{exp_id}/{'pre_training' if init_phase else 'main'}"
        self.writer = SummaryWriter(log_dir=log_dir)

    def log_images(self, tag: str, tensor: torch.Tensor, normalize=False):
        tensor = vutils.make_grid(tensor.cpu(), normalize=normalize)
        # TODO Maybe check here what phase we are in and set 'tag' accordingly.
        self.writer.add_image(tag, tensor, self.global_step)
</code></pre>
<p>G is a Unet defined using <a href="https://github.com/qubvel/segmentation_models.pytorch" rel="noopener nofollow ugc">segmentation_models.pytorch</a></p>
<pre data-code-wrap="python"><code class="lang-python">smp.Unet(encoder_name="resnet18", encoder_depth=5, encoder_weights="imagenet",
                         decoder_channels=(32, 32, 32, 32, 16),
                         decoder_use_batchnorm=True, in_channels=3, classes=3, activation="sigmoid")
</code></pre>
<p>When using <code>autocast</code> this is our result <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/e/bec5d78eea50076629435936d88ac7426aaad926.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/bec5d78eea50076629435936d88ac7426aaad926" title="individualImage (1)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/bec5d78eea50076629435936d88ac7426aaad926_2_517x129.jpeg" alt="individualImage (1)" data-base62-sha1="rdEAPdasNUux0YbRQ7652TC5SF8" width="517" height="129" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/bec5d78eea50076629435936d88ac7426aaad926_2_517x129.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/bec5d78eea50076629435936d88ac7426aaad926_2_775x193.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/e/bec5d78eea50076629435936d88ac7426aaad926.jpeg 2x" data-dominant-color="73888A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">individualImage (1)</span><span class="informations">906×228 194 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
compared to not not wrapping the two code lines in <code>with torch.cuda.amp.autocast():</code><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de96026c09c406e040fac4b2880b3f2b04d66857.png" data-download-href="https://discuss.pytorch.org/uploads/default/de96026c09c406e040fac4b2880b3f2b04d66857" title="individualImage (2)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de96026c09c406e040fac4b2880b3f2b04d66857_2_517x129.png" alt="individualImage (2)" data-base62-sha1="vL5ojIQeTk976go0BfLxQzL9n0P" width="517" height="129" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de96026c09c406e040fac4b2880b3f2b04d66857_2_517x129.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de96026c09c406e040fac4b2880b3f2b04d66857_2_775x193.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/e/de96026c09c406e040fac4b2880b3f2b04d66857.png 2x" data-dominant-color="7E6877"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">individualImage (2)</span><span class="informations">906×228 423 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>We are simply trying to recreate the following images <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c61997efcdf8be2f75a87369cbd5f9afafc0ff02.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/c61997efcdf8be2f75a87369cbd5f9afafc0ff02" title="individualImage"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c61997efcdf8be2f75a87369cbd5f9afafc0ff02_2_517x129.jpeg" alt="individualImage" data-base62-sha1="sgtn7IBKZ7eNKEuy20jRkoeKyiK" width="517" height="129" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c61997efcdf8be2f75a87369cbd5f9afafc0ff02_2_517x129.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c61997efcdf8be2f75a87369cbd5f9afafc0ff02_2_775x193.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/6/c61997efcdf8be2f75a87369cbd5f9afafc0ff02.jpeg 2x" data-dominant-color="726C6D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">individualImage</span><span class="informations">906×228 128 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Another problem is that the produced are oversaturated, compared to the originals.</p>

username: ptrblck
cooked: <p>Is this issue reproducible, i.e. if you are rerunning the training script several times, the amp training fails to converge, while the “standard” training yields these oversaturated images?</p>

username: FilipAndersson245
cooked: <p>I redid the experiment using a sample of 600 images, where each iteration where ran for 5 epochs,<br>
here is a few samples of the output of each case, 5 experiment where done for both and with and without, and i took 2 random samples from both to display the difference. It seem to be reproducible.</p>
<p>with <code>torch.cuda.amp.autocast()</code><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d903859a65dc10f6518f91e589243fbb6a7283b.png" data-download-href="https://discuss.pytorch.org/uploads/default/0d903859a65dc10f6518f91e589243fbb6a7283b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d903859a65dc10f6518f91e589243fbb6a7283b_2_517x129.png" alt="image" data-base62-sha1="1VZceoyY6XMq3HYFy3dsN2PbvVp" width="517" height="129" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d903859a65dc10f6518f91e589243fbb6a7283b_2_517x129.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d903859a65dc10f6518f91e589243fbb6a7283b_2_775x193.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d903859a65dc10f6518f91e589243fbb6a7283b.png 2x" data-dominant-color="8A884E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">906×228 497 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> `<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/5/65081a2830c75392ed235f25352dcaa8597df390.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/65081a2830c75392ed235f25352dcaa8597df390" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65081a2830c75392ed235f25352dcaa8597df390_2_517x129.jpeg" alt="image" data-base62-sha1="epLAanTcOgAu0nQ3hkcXBL8vI1W" width="517" height="129" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65081a2830c75392ed235f25352dcaa8597df390_2_517x129.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65081a2830c75392ed235f25352dcaa8597df390_2_775x193.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/5/65081a2830c75392ed235f25352dcaa8597df390.jpeg 2x" data-dominant-color="8E7790"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">906×228 308 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>without amp:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/7/97aea42613b40b754077d34778ac3852ecf57afd.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/97aea42613b40b754077d34778ac3852ecf57afd" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97aea42613b40b754077d34778ac3852ecf57afd_2_517x129.jpeg" alt="image" data-base62-sha1="lDQgr1O9SVvOkx3gBNylxBMhRzn" width="517" height="129" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97aea42613b40b754077d34778ac3852ecf57afd_2_517x129.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97aea42613b40b754077d34778ac3852ecf57afd_2_775x193.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/7/97aea42613b40b754077d34778ac3852ecf57afd.jpeg 2x" data-dominant-color="806463"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">906×228 243 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/0/9069ec2d103855e41d8bc171ef23479f7a9357ba.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/9069ec2d103855e41d8bc171ef23479f7a9357ba" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/9069ec2d103855e41d8bc171ef23479f7a9357ba_2_517x129.jpeg" alt="image" data-base62-sha1="kBxH0Vd0kDfWGG54xnfPmCvYOzE" width="517" height="129" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/9069ec2d103855e41d8bc171ef23479f7a9357ba_2_517x129.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/9069ec2d103855e41d8bc171ef23479f7a9357ba_2_775x193.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/0/9069ec2d103855e41d8bc171ef23479f7a9357ba.jpeg 2x" data-dominant-color="836C67"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">906×228 248 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: FilipAndersson245
cooked: <p>I fixed the oversaturation by changing how i did the inv_norm. The fix me and <a class="mention" href="/u/zimonitrome">@ZimoNitrome</a> did can be seen in <a href="https://discuss.pytorch.org/t/how-to-normalize-a-tensor-to-0-mean-and-1-variance/18766/19">here</a>, mixed precision still produces noise and fail to converge.</p>

username: ptrblck
cooked: <p>I haven’t seen these failures before and other GAN arhcitectures, such as <a href="https://github.com/NVIDIA/pix2pixHD/blob/5a2c87201c5957e2bf51d79b8acddb9cc1920b26/train.py">Pix2PixHD</a> don’t seem to need any changes additional changes for mixed-precision training.<br>
CC <a class="mention" href="/u/mcarilli">@mcarilli</a> in case he’s seen some GAN failures before.</p>