Convolutional Autoencoder, Autocast, and loss of information from BCEWithLogits

username: cpeters
cooked: <p>I have a convolutional autoencoder, where the output layer is:</p>
<pre><code class="lang-auto">        self.t_conv5 = nn.ConvTranspose2d(128, 3, 3, stride=2, padding=1, output_padding=1)
        ...
        x = torch.sigmoid(self.t_conv5(x))
</code></pre>
<p>and the loss criterion is given by <code>nn.BCELoss()</code>. When trying to utilise autocast for fp16 conversion, the error that BCELoss is unsafe is given and the program suggests using BCEWithLogits instead to combine the final sigmoidal layer and BCE loss. This obviously stacks another sigmoidal layer on top of the desired output. If I remove the sigmoidal activation and re-apply it to the model output (i.e. <code>output = model(images)</code> becomes <code>output = torch.sigmoid(model(images))</code>, the calculated loss after 20 epochs of training remains the same but there’s a drastic difference in the output images:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/3/c3efa99d03b615f4c12132862f8488c44e7a3992.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/c3efa99d03b615f4c12132862f8488c44e7a3992" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c3efa99d03b615f4c12132862f8488c44e7a3992_2_690x319.jpeg" alt="image" data-base62-sha1="rXkA2QSq019f1XG4kHZcjV3exns" width="690" height="319" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c3efa99d03b615f4c12132862f8488c44e7a3992_2_690x319.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c3efa99d03b615f4c12132862f8488c44e7a3992_2_1035x478.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/3/c3efa99d03b615f4c12132862f8488c44e7a3992.jpeg 2x" data-dominant-color="AFAEAC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1256×582 134 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>But as this is the same data and shuffling is disabled, the rearrangement shouldn’t have changed the output in this way. Is there some loss of relevant weights by having to rearrange the location of the sigmoid function in this way, and is there some way to counteract this? The autocasting, to my knowledge, should not reduce the final accuracy.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="cpeters" data-post="1" data-topic="111658">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/e274bd/48.png" class="avatar"> cpeters:</div>
<blockquote>
<p>If I remove the sigmoidal activation and re-apply it to the model output (i.e. <code>output = model(images)</code> becomes <code>output = torch.sigmoid(model(images))</code>, the calculated loss after 20 epochs of training remains the same but there’s a drastic difference in the output images:</p>
</blockquote>
</aside>
<p>If I understand the workflow correctly, you are now feeding the raw logits to <code>nn.BCEWithLogitsLoss</code>, which yields the same loss as in the <code>float32</code> training, and use the <code>output = torch.sigmoid(model(images))</code> tensor only to visualize the samples?<br>
If so, do you see any difference by converting the output to <code>.float()</code> before feeding it into the <code>torch.sigmoid</code> operation?</p>

username: cpeters
cooked: <p>That recovers the colours. Do you know why that is?</p>

username: ptrblck
cooked: <p>The mixed-precision training utility (<code>amp</code>) raises the error when you are using <code>sigmoid</code> + <code>nn.BCEWithLogitsLoss</code>, as its usage was determined to not be numerically stable in <code>float16</code>. Thus passing the raw logits to <code>nn.BCEWithLogitsLoss</code> is the right approach, since internally <code>log_sigmoid</code> will be applied, if I’m not mistaken.<br>
I had a guess that using <code>torch.sigmoid</code> on the output might have exactly the same issue when you are trying to predict the outputs and it seems to be indeed the case.</p>

username: cpeters
cooked: <p>By the same problem, you mean numerical instability?</p>

username: ptrblck
cooked: <p>Yes, numerical instability, rounding etc. which might be caused by passing <code>float16</code> values directly to <code>sigmoid</code>.</p>