Question about BCE* losses interface and features

eddiewrc: <p>Dear pytorch guys <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> ,<br>
first of all thanks for developing such a conceptually smart, easy to use and nice ML framework.</p>
<p>I am a pytorch user since the first versions and I have few questions about how the NN losses are conceptually implemented:</p>
<ol>
<li>
<p>BCE and BCEWithLogitLoss require Float labels (also called targets if you prefer). Why is that, precisely? The CrossEntropyLoss, for example, requires Long labels (integers) and it’s a more general case than BCE, so why the BCE* need floats? This choice influences also the second point I am going to raise</p>
</li>
<li>
<p>BCE and BCEWithLogit don’t (currently) support ignore_index, which is IMO an extremely useful feature. Implementing the ignore_index is easy, but the fact that these losses take in Float label make it a bit awkward to do, because due to numeric problems, the float representation of the ignore_index value could be very similar but different enough to fail the</p>
</li>
</ol>
<p>mask = labels.ne(ignore_index)</p>
<p>clause that I could use to implement such a feature. (e.g. -1.0000000001 instead of -1).</p>
<ol start="3">
<li>Minor issue: in the pytorch 1.1.0 BCEWithLogitLoss documentation, I think that there is a problem in the pos_weight example: the (10,64) tensor is said to contain 10 batches of 64 classes, but from what I see, if there are 64 classes, then 10 must be the samples (with batch omitted). Conversely, if 10 is the batch size, the 64 should be the samples per batch (with only 1 binary classification going on).</li>
</ol>
<p>To wrap up my comments:<br>
Would it then be possible maybe to uniform a bit more the “interface” of the losses functions, ensuring that:<br>
-losses for classification use integer labels/targets<br>
-all the losses for classification provide ignore_index<br>
-all the losses for classification provide class_weights</p>
<p>thanks in advance and keep up with the good work!<br>
cheers,<br>
Daniele</p>

ptrblck: <p>Hi Daniele,</p>
<p>I think the reason behind using a floating point target is given in the docs:</p>
<blockquote>
<p>This is used for measuring the error of a reconstruction in for example an auto-encoder. Note that the targets t[i] should be numbers between 0 and 1.</p>
</blockquote>
<p>I stumbled upon this issue <a href="https://github.com/pytorch/pytorch/issues/5157#issue-295920753">a while ago</a> and was mistakenly thinking its being only used for a binary or multi-label classification. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<p>This would also answer the second point. Since there are no discrete classes, the implementation of an <code>ignore_index</code> wouldn’t really make sense.</p>
<p>The documentations mentions</p>
<blockquote>
<p>target = torch.ones([10, 64], dtype=torch.float32)  # 64 classes, batch size = 10</p>
</blockquote>
<p>which seems correct.<br>
Note that <code>nn.BCEWithLogitsLoss</code> can be used in a multi-label classification use case, i.e. a single sample has more than a single correct (active) class.<br>
In this example all 64 classes would be active for all 10 sample in the batch.</p>

eddiewrc: <p>thanks for your kind reply.</p>
<p>Yes, I noticed that the doc was mentioning the autoencoder use, but then what should I (we) use as loss for “vanilla” classification? What is the “official” recommendation?</p>
<p>I ask this because the fact that BCE* are “open to the use” in autoencoders are lowering their effective usefulness for classic classification tasks (e.g. no ignore index and float labels as input).</p>
<p>Wouldn’t be better to “specialize” these functions, for autoencoder and classification use? I suppose that the number of times they are used for vanilla classification &gt;&gt; the number of times they are used into autoencoders, since the first use case is the most likely frequent.</p>

ptrblck: <p>This is my personal opinion and others might have different preferences, so take it with a grain of salt <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>For a multi-class classification, I would use <code>nn.CrossEntropyLoss</code>, which also provides the <code>ignore_index</code> argument. This makes sense, as e.g. if I’m dealing with 1000 classes, I might just want to ignore a certain one.</p>
<p>In a binary classification, you could still use <code>nn.CrossEntropoyLoss</code> with two outputs (possibly more, if you ignore this class) or alternatively <code>nn.BCE(WithLogits)Loss</code>.<br>
An <code>ignore_index</code> argument doesn’t really make sense in the latter case, since we are dealing with float values, and we are just using a single output neuron, which should give us the probability (logit) of the positive class. Ignoring a class in a binary setup seems a bit strange, and it might be simpler to just calculate the loss of a single class instead (if that’s the use case).</p>
<p>For a multi-label classification, I would use also use <code>nn.BCE(WithLogits)Loss</code>, where each neuron corresponds to the probability (logit) of the corresponding class.<br>
Ignoring certain classes in this use case could in fact make sense.</p>

M.sadegh_Saberian: <p>I have a similar question and I was wondering if you could help me. assume that I have a multi-label classification task, but for each sample some of the labels are not known, NaN. having ignore_index in this case would help me masking the loss for each binary classification. since there is no ignore_index for BCELoss, what do you suggest I do to handle the missing labels?</p>
<p>thank you so much!</p>

ptrblck: <p>You could create the unreduced loss via <code>reduction='none'</code>, create a mask with 1s for all valid outputs/targets and 0s for the invalid ones, multiply the loss with it, reduce it (e.g. via <code>mean</code>), and calculate the gradients via <code>backward()</code>.</p>

M.sadegh_Saberian: <p>great idea! thanks for your help!</p>