How can entropy be negative?

blackbirdbarber: <p>Example:</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

z = torch.tensor([3, -1, 2.])

y=torch.tensor([1,0,2]).float()

yhat = F.binary_cross_entropy_with_logits(z, y, reduction='none')
print(yhat)
</code></pre>
<p>Out:</p>
<pre><code class="lang-auto">tensor([ 0.0486,  0.3133, -1.8731])
</code></pre>
<p>Can you clarify why PyTorch supports negative entropy solutions?</p>

ptrblck: <p>You are passing the targets, which are out of bounds as the <code>target</code> should be a probability. From the docs:</p>
<blockquote>
<p>target: Tensor of the same shape as input with values between 0 and 1</p>
</blockquote>

blackbirdbarber: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="158563">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>target</code> should be a probability</p>
</blockquote>
</aside>
<p>I know it is something strange. How target should be probability? I haven’t found this word in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.binary_cross_entropy_with_logits.html" rel="noopener nofollow ugc">docs</a></p>
<p>What is the meaning if target is 0.5?</p>

ptrblck: <p>A target value of <code>0.5</code> would indicate that the sample belongs to both classes with the same weight.<br>
You can interpret the target values as probabilities as they are supposed to be in <code>[0, 1]</code> or as any weights if it fits your understanding better. In any case, you should not pass targets out of this range as this would yield invalid losses such as your negative one.</p>

blackbirdbarber: <p>Is this the case when I do image matting and when some pixels do have alpha channel?</p>

ptrblck: <p>I don’t fully understand the issue with the alpha channel. Wouldn’t your target be a simple mask or are you adding the alpha channel to the (binary) mask as well? In the latter case, wouldn’t it be possible to let the alpha values be in <code>[0, 1]</code> (same as the mask values)?</p>

blackbirdbarber: <p>I am currently running your U-net from the git repo to do image matting. While I do that I am tinkering.</p>
<p>The tiny dataset I am using has mask images with either zeros or ones. No alpha channel. I am in the comfort zone with the targets this way since I used CE.</p>
<p>But I evaluated using BCE and CE and the advantage of BCE would be the case there there is alpha channel in the image (like the one on the left - <em>full opacity is 1.0</em>)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/4/94c3dd175d263c1d733939c6a97958c241d614f3.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/94c3dd175d263c1d733939c6a97958c241d614f3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/4/94c3dd175d263c1d733939c6a97958c241d614f3.jpeg" alt="image" data-base62-sha1="le2iH6eB89rs9d1dy6VQIpQdqo3" width="690" height="265" data-dominant-color="95988C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">696×268 42 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I can just imagine the case where I have transparent mask images with pixels where alpha is in between zero and one at some pixels. This way the matting may look even better comparing to the coarse zero and ones.</p>
<p>This corresponds to the targets with alpha you are mentioned. But then, if you use <code>argmax</code>, how would you predict these subtle foreground edges BCE trains?</p>

ptrblck: <p>You would not use <code>argmax</code> in this case, since <code>nn.BCEWithLogitsLoss</code> is used for a multi-label classification use case, where a threshold is used on the logits.<br>
If you want to work on a multi-class segmentation, use <code>nn.CrossEntropyLoss</code>, but this would not fit your use case predicting the alpha channel additionally to the pixel class label.</p>

blackbirdbarber: <p>OK, it took me some time but now I see the image matting task is different than semantic segmentation in so much that some pixels may belong to foreground as well as background (partial or mixed pixels).</p>