Looking for a cross entropy loss that accepts two tensors of the same shape

username: Mehran_Ziadloo
cooked: <p>The cross-entropy loss function in <code>torch.nn.CrossEntropyLoss</code> takes in inputs of shape <code>(N, C)</code> and targets of shape <code>(N)</code>. This means that targets are one integer per sample showing the index that needs to be selected by the trained model.</p>
<p>In my case, I’ve already got my target formatted as a one-hot-vector. And also, the output of my model has already gone through a softmax function. Now, I’m looking for a loss function to calculate the loss between my model’s predictions and the gold targets. Is there already such a loss function in Pytorch? Or do I need to develop my own? If I have to develop my own, could you please point me in the right direction?</p>
<p>How about this:</p>
<pre><code class="lang-auto">def cross_entropy(output: torch.Tensor, target: torch.Tensor):
    return -1.0 * (output.log() * target).mean()
</code></pre>

username: ptrblck
cooked: <p>You could easily transform the one-hot encoded tensors to class index tensors using <code>target = torch.argmax(one_hot, 1)</code> and just remove the softmax at the end of your model.<br>
Generally I wouldn’t recommend to rewrite PyTorch function, if you don’t have a valid use case for it, as e.g. internally the log-sum-exp trick is used to stabilize the calculations numerically.</p>
<p>Anyway, I think the <code>mean</code> in your formula would calculate the mean over all classes, which should most likely yield a lower loss than the corresponding one using <code>nn.CrossEntropyLoss</code>.</p>

username: Mehran_Ziadloo
cooked: <p>Thanks. As a matter of fact, the reason why I turned to this approach is a use case which is not supported by the cross entropy currently implemented. And I’ll be really surprised if I’m the first to complain why Pytorch developers have decided to implement a special case of cross entropy instead of the real one.</p>
<p>Anyways, the current implementation of cross entropy in Pytorch does not support multi class targets. These are the problems where there could be none or more than one class be active in the target at the same time. I guess I’ll go with my own function then.</p>

username: KFrank
cooked: <p>Hi Mehran!</p>
<aside class="quote no-group" data-username="Mehran_Ziadloo" data-post="3" data-topic="61302" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mehran_ziadloo/48/17822_2.png" class="avatar"> Mehran_Ziadloo:</div>
<blockquote>
<p>Anyways, the current implementation of cross entropy in Pytorch does not support multi class targets. These are the problems where there could be none or more than one class be active in the target at the same time. I guess I’ll go with my own function then.</p>
</blockquote>
</aside>
<p>Let me draw a distinction between two different problems.  When I<br>
say “multiclass” I mean more than two classes, that is, not binary.<br>
(So, for example, classes A, B, C, and D, rather than just “yes” or<br>
“no.”)  But each sample is of exactly one of the classes.</p>
<p>When I say “multilabel” I mean that a given sample can have more<br>
than one class label (e.g., this image has both a cat and a dog in<br>
it, but not a fish).</p>
<p>This last use case that you talk about is what I would call multilabel:<br>
“none or more than one class be active in the target at the same time.”</p>
<p>If you think about it, a multilabel problem (even for only two classes)<br>
is really a binary classification problem, just with an additional “class”<br>
dimension.  (That is, cat – yes or no, dog – yes or no, fish – yes or no.)</p>
<p><a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BCELoss" rel="noopener nofollow ugc">BCELoss</a> (and probably better to use, <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BCEWithLogitsLoss" rel="noopener nofollow ugc">BCEWithLogitsLoss</a>) supports<br>
this extra class dimensions.  I think this is the “cross entropy loss that<br>
accepts two tensors of the same shape” that you are looking for.</p>
<p>If you don’t want to think about it terms of a binary classification<br>
problem, you could use the equivalent, but differently named<br>
<a href="https://pytorch.org/docs/stable/nn.html#torch.nn.MultiLabelSoftMarginLoss" rel="noopener nofollow ugc">MultiLabelSoftMarginLoss</a>.</p>
<p>Here is a pytorch (version 0.3.0) session that shows that these two<br>
loss functions are really the same.  (You can get rid of the 0.3.0 oddities,<br>
and this will run with an up-to-date pytorch.  It will probably even run<br>
with an up-to-date pytorch as-is.)</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; l1 = torch.nn.BCEWithLogitsLoss()
&gt;&gt;&gt; l2 = torch.nn.MultiLabelSoftMarginLoss()
&gt;&gt;&gt; pred = torch.randn ((3, 5))
&gt;&gt;&gt; targ = torch.bernoulli (0.5 * torch.ones ((3, 5)))
&gt;&gt;&gt; pred = torch.autograd.Variable (pred)
&gt;&gt;&gt; targ = torch.autograd.Variable (targ)
&gt;&gt;&gt; l1 (pred, targ)
Variable containing:
 0.8563
[torch.FloatTensor of size 1]

&gt;&gt;&gt; l2 (pred, targ)
Variable containing:
 0.8563
[torch.FloatTensor of size 1]
</code></pre>
<p>You would probably want to replace the call to:</p>
<pre data-code-wrap="python"><code class="lang-python">targ = torch.bernoulli (0.5 * torch.ones ((3, 5)))
</code></pre>
<p>with:</p>
<pre data-code-wrap="python"><code class="lang-python">targ = torch.randint (1, (3, 5))
</code></pre>
<p>(The old 0.3.0 version didn’t have <code>randint()</code> yet.)</p>
<p>Note that the input tensor, <code>pred</code>, and target tensor, <code>targ</code>, are indeed<br>
of the same shape, in this example, <code>(3, 5)</code>.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: Mehran_Ziadloo
cooked: <p>Thanks K. Frank,</p>
<p>That was very informative. You are right, using the right terminology is crucial. Thanks for correcting me.</p>
<p>I guess I’ll go with MultiLabelSoftMarginLoss even though it seems just like BCEWithLogitsLoss (I’m not sure what are their differences at this point!).</p>
<p>Thanks again,<br>
Mehran</p>