How to calculate loss for binary images?

Vishal-S-P: <p>Hi,</p>
<p>I am trying to use encoder-decoder architecture to predict binary images. What is the appropriate loss function for this ? Input is a binary image and output is a binary image &amp; gt is a binary image.</p>

KFrank: <p>Hi Vishal!</p>
<aside class="quote no-group" data-username="Vishal-S-P" data-post="1" data-topic="158987" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vishal-s-p/48/62996_2.png" class="avatar"> Vishal-S-P:</div>
<blockquote>
<p>What is the appropriate loss function for this ?</p>
</blockquote>
</aside>
<p>I assume that you want to use your loss function to train your network.<br>
I also assume that “output” is the output of your network, that is, the<br>
predictions your network makes.</p>
<p>If output is, indeed, a binary image, e.g., an image where the pixel values<br>
are either <code>0.0</code> and <code>1.0</code> (or, say, <code>0</code> and <code>255</code>), then you won’t be able to<br>
construct a suitable loss function.</p>
<p>This is because – in order to train your network with a gradient-based<br>
optimization algorithm – your loss needs to be usefully differentiable.<br>
Because your output is a binary image, it can’t be usefully differentiable.<br>
Even if you define it in a way that is differentiable, its gradients will be<br>
zero almost everywhere.</p>
<p>I would  recommend that you use the output of your final <code>Linear</code> layer,<br>
without any non-linear activation or anything that binarizes it, as your<br>
prediction, and then use <code>BCEWithLogitsLoss</code> as your loss criterion<br>
that compares your prediction with your ground truth (which can be<br>
binary).</p>
<p>Best.</p>
<p>K. Frank</p>

Vishal-S-P: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="158987">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>near activation or anyth</p>
</blockquote>
</aside>
<p>I was able to backpropagate through the “thresholding” layer by defining a custom autograd function. The output of the model has a pixel value of either 0 or 1. I trained this model using L1 loss, MSE loss and BCEwithLogits loss. None of these losses I was able to get desired output.</p>

azhanmohammed: <aside class="quote no-group" data-username="Vishal-S-P" data-post="3" data-topic="158987">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vishal-s-p/48/62996_2.png" class="avatar"> Vishal-S-P:</div>
<blockquote>
<p>I was able to backpropagate through the “thresholding” layer by defining a custom autograd function.</p>
</blockquote>
</aside>
<p>A better way would be to use a linear layer followed by a sigmoid output, and then train the model using BCE Loss. The sigmoid activation would make sure that the output stays between 0 and 1, and as the model trains, the output would move towards binary outputs. You can later use binary thresholding to get the final output.</p>

Vishal-S-P: <aside class="quote no-group" data-username="azhanmohammed" data-post="4" data-topic="158987">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/azhanmohammed/48/49537_2.png" class="avatar"> azhanmohammed:</div>
<blockquote>
<p>linear layer followed by a sigmoid output,</p>
</blockquote>
</aside>
<pre><code class="lang-auto">class Decoder(nn.Module):
	
	def __init__(self, input_dim=256, output_dim=3, n_ups=2):
		super(Decoder, self).__init__()
		
		model = []
		
		ch = input_dim
		for i in range(n_ups):
			model += [nn.Upsample(scale_factor=2, mode='bilinear', align_corners=False)]
			model += [nn.Conv2d(ch, ch // 2, 3, 1, 1)]
			model += [get_norm_layer(ch // 2)]
			model += [nn.ReLU()]
			ch = ch // 2
		out_channels = 1
		model += [nn.Conv2d(ch, out_channels, 3, 1, 1)]
		# model += [nn.Tanh()]
		# # model += [nn.Sigmoid()]
		model += [nn.Linear(512, 512)]
		model += [nn.Sigmoid()]
		self.model = nn.Sequential(*model)
	
	def forward(self, x):
		out = self.model(x)
		# print("[INFO] out shape : ", out.shape)
		# exit()
		return out
</code></pre>
<p>This is the decoder I have and I am calculating the BCELoss with reduction as the mean. The output images are not at all close to the ground truth.</p>
<p>For example the ground-truth looks like this -<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b77c6392e5e9c50f4e3f79b8205cd48f897e48cd.png" alt="Capture" data-base62-sha1="qbbSvGs8zYxPB2r2s3vIf71pHZP" width="297" height="269"><br>
I believe that no loss can match such kind of ground truth.</p>

KFrank: <p>Hi Vishal!</p>
<aside class="quote no-group" data-username="Vishal-S-P" data-post="3" data-topic="158987" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vishal-s-p/48/62996_2.png" class="avatar"> Vishal-S-P:</div>
<blockquote>
<p>I was able to backpropagate through the “thresholding” layer by defining a custom autograd function. The output of the model has a pixel value of either 0 or 1.</p>
</blockquote>
</aside>
<p>If your predictions – the output of your model – are binary (presumably<br>
because they are binarized by your “thresholding” layer), you will get<br>
zero gradients.  You then try to fix this up by using a custom autograd<br>
function that gives you some sort of non-zero gradients.</p>
<p>After thresholding, the damage has largely been done, and trying to<br>
repair that damage with your choice of loss function is not likely to work.</p>
<p>The standard approach – that works quite well in many use cases – is<br>
to train with the output of your final <code>Linear</code> layer as probabilistic logit<br>
predictions (with no “thresholding” layer or other non-linear activation),<br>
and use <code>BCEWithLogitsLoss</code> to compute the loss between your logit<br>
predictions and your binary ground truth.</p>
<p>Then, at inference time, if you want to visualize you predicted images<br>
or compute an accuracy metric, you can binarize your your predicted<br>
images by thresholding the logits against zero, e.g.,</p>
<pre data-code-wrap="python"><code class="lang-python">binary_predicted_image = (predicted_image &gt; 0).long()
</code></pre>
<p>Try this first, and see whether you get good results (or at least better<br>
results than you get from your other schemes).</p>
<p>Best.</p>
<p>K. Frank</p>

KFrank: <p>Hi Mohammed!</p>
<aside class="quote no-group" data-username="azhanmohammed" data-post="4" data-topic="158987" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/azhanmohammed/48/49537_2.png" class="avatar"> azhanmohammed:</div>
<blockquote>
<p>A better way would be to use a linear layer followed by a sigmoid output, and then train the model using BCE Loss</p>
</blockquote>
</aside>
<p><code>Sigmoid</code> feeding into <code>BCELoss</code> is mathematically the same as<br>
<code>BCEWithLogitsLoss</code>, but is numerically less stable.</p>
<p><code>BCELoss</code> really shouldn’t be used, and feeding the output of the final<br>
<code>Linear</code> layer directly into <code>BCEWithLogitsLoss</code> (without any <code>Sigmoid</code><br>
or other intervening non-linear activation) is the way to go.</p>
<p>Best.</p>
<p>K. Frank</p>