Combining sparmax with CrossEntropyLoss

yuri: <p>it is known that torch.nn.CrossEntropyLoss() method integrates the softmax method for calculating the loss. in my case, I want to use sparsemax method instead and still keep the CrossEntropyLoss as a criterion. how would this be possible?<br>
ps: my model is a FCNN that has 6 inputs and 125 output classes and 5 hidden layers each with 128 neurons. I welcome any critics of the model (since I am not fully convinced about it)<br>
thank you</p>

KFrank: <p>Hi Yuri!</p>
<aside class="quote no-group" data-username="yuri" data-post="1" data-topic="183371" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/71c47a/48.png" class="avatar"> yuri:</div>
<blockquote>
<p>I want to use sparsemax method instead and still keep the CrossEntropyLoss as a criterion.</p>
</blockquote>
</aside>
<p>First off, I don’t think you want to do this.  <code>CrossEntropyLoss</code> has a<br>
logarithmic divergence when you predict a probability of zero for what<br>
your <code>target</code> labels as the correct class.  You will therefore get <code>inf</code> for<br>
your loss function, which will quickly pollute your backpropagation and<br>
future training.</p>
<p>Note that what I believe is the <a href="https://arxiv.org/pdf/1602.02068.pdf" rel="noopener nofollow ugc">original sparsemax paper</a> also proposes<br>
a companion “sparsemax loss” that one would presumably use in place<br>
of <code>CrossEntropyLoss</code>.  I haven’t tried any of this, but perhaps it would<br>
make sense for your use case.</p>
<aside class="quote no-group">
<blockquote>
<p>how would this be possible?</p>
</blockquote>
</aside>
<p>With the proviso that it’s probably not a good idea, I see two ways:</p>
<p>You could write your own version of <code>CrossEntropyLoss</code> that applies just<br>
<code>log()</code> rather than <code>log_softmax()</code> internally and takes probabilities as its<br>
<code>input</code>. Then pass the output of sparsemax into your custom cross-entropy<br>
loss.</p>
<p>You could also use the fact that <code>log()</code> is the inverse of <code>softmax()</code> (in the<br>
sense that <code>t.softmax (0).log().softmax (0) == t.softmax (0)</code>)<br>
and apply <code>log() </code> to the output of sparsemax before feeding it into pytorch’s<br>
<code>CrossEntropyLoss</code>.</p>
<p>Note that in both cases you will be applying <code>log()</code> to the output of<br>
sparsemax, which will yield <code>inf</code> when the output of sparsemax is<br>
zero.</p>
<p>You could try clamping the output of sparsemax away from zero, but<br>
when you’re in the clamped regime, your gradient for the clamped<br>
probability will be zero, defeating the benefit you get from the logarithmic<br>
divergence when your prediction is quite wrong, and degrading training.</p>
<p>Best.</p>
<p>K. Frank</p>

yuri: <p>thanks for your answer KFrank, first the log in CrossEntropyLoss is preseeded by the softmax therefore no zeros.<br>
the idea of writing an adapted version of CrossEntropyLoss sounds necessary since Pytorch does not integrate it yet.<br>
Best regards</p>

KFrank: <p>Hi Yuri!</p>
<aside class="quote no-group" data-username="yuri" data-post="3" data-topic="183371" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/71c47a/48.png" class="avatar"> yuri:</div>
<blockquote>
<p>the log in CrossEntropyLoss is preseeded by the softmax therefore no zeros.</p>
</blockquote>
</aside>
<p>Just to be clear about the logic of my previous post:</p>
<p>The sparsemax function outputs probabilities.  (They range over <code>[0.0, 1.0]</code>,<br>
inclusive, and sum to <code>1.0</code>.)</p>
<p>Pytorch’s <code>CrossEntropyLoss</code> is <em>not</em> designed to take probabilities as its<br>
input.  Instead, it takes (unnormalized) log-probabilities (that range over<br>
<code>(-inf, inf)</code>).</p>
<p>If you want to pass the output of sparsemax to a cross-entropy function,<br>
you have two choices:  You can pass the outputs of sparsemax to pytorch’s<br>
<code>CrossEntropyLoss</code>, but this is a known mistake, because passing<br>
probabilities to <code>CrossEntropyLoss</code> doesn’t train well.</p>
<p>In the analogous <code>softmax()</code> situation, this would be like:</p>
<pre data-code-wrap="python"><code class="lang-python">logprobs = torch.nn.Linear (10, 10) (somedata)
probs = logprobs.softmax (dim = 0)
loss = torch.nn.CrossEntropyLoss() (probs, target)   # error, should have passed in logprobs
</code></pre>
<p>You can do this and it will “work,” but it won’t train well.</p>
<p>It is true that when sparsemax outputs a zero, the <code>softmax()</code> that is internal<br>
to pytorch’s <code>CrossEntropyLoss</code> will protect <code>CrossEntropyLoss</code>’s <code>log()</code><br>
and you won’t get that <code>inf</code>.  But it’s still a mistake, because pytorch’s<br>
<code>CrossEntropyLoss</code> doesn’t work properly when passed probabilities.</p>
<p>Or you can pass the output of sparsemax to a version of cross entropy that<br>
accepts probabilities.  Internally such a cross-entropy function will take the<br>
<code>log()</code> of its inputs (because that it’s how it’s defined).  But now when you<br>
pass in a probability that is exactly zero (rather than just very small), you<br>
will get an <code>inf</code> that breaks things unless you stand on your head to patch<br>
it up somehow.</p>
<p>The key problem is that sparsemax <em>is designed</em> to output probabilities that<br>
are frequently exactly zero, whereas a cross-entropy loss doesn’t work<br>
with probabilities that are exactly zero.  (Mathematically, you could say that<br>
cross entropy isn’t defined when an input probability is zero, or you could<br>
say that it’s defined to be <code>inf</code>, but either way, your training breaks.)</p>
<aside class="quote no-group">
<blockquote>
<p>the idea of writing an adapted version of CrossEntropyLoss sounds necessary</p>
</blockquote>
</aside>
<p>Well, this depends on what you mean by “adapted.”  But if your “adapted<br>
version” remains close to a standard cross-entropy function – for example,<br>
by adding epsilons or clamping probabilities away from zero – it won’t train<br>
well.</p>
<p>The original sparsemax paper proposes a sparsemax-loss function (that<br>
I would say is <em>not</em> just an “adapted” version of cross entropy).  I’ve never<br>
tried it, but it seems conceptually sound and is likely to be better than<br>
trying to patch over the inherent incompatibility between cross entropy<br>
and exactly-zero probabilities.</p>
<p>Best.</p>
<p>K. Frank</p>

yuri: <p>Dear K. Frank<br>
once again thank you very much for your informative answer.<br>
unfortunately, I could not find any implementation of the sparesmax loss function that could be easily exploited.<br>
by “adapted” I mean to skip the log softmax part and only run the  -∑ ground_truth_i * p_i part.<br>
Regards</p>