Loss For Low Confidence Classification

jafioti: <p>Hi,</p>
<p>I’m working on a system in which I want the network to predict one of the classes with a very high score and the others a very low score. These scores are attention weights, and they select different sections of the network downstream, so I want the network to choose a single downstream section at once and not a mixture of different ones. I have lowered the temperature on the softmax, but I am wondering how to implement an additional loss to incentivize the network to favor a single class, and not a mixture of many. Please let me know if you need any clarification.</p>

KFrank: <p>Hello Joe!</p>
<aside class="quote no-group quote-modified" data-username="jafioti" data-post="1" data-topic="74709" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jafioti/48/19400_2.png" class="avatar"> jafioti:</div>
<blockquote>
<p>I want the network to predict one of the classes with a very high score and the others a very low score.<br>
…<br>
I have lowered the temperature on the softmax, but I am wondering how to implement an additional loss to incentivize the network to favor a single class, and not a mixture of many.</p>
</blockquote>
</aside>
<p>Part of me wonders whether this is a good idea.  You’re not able<br>
to train your network to make these predictions with confidence,<br>
so you want to fake it, and I wonder if this isn’t a little fishy.</p>
<p>Having said that:</p>
<p>I would have thought that tweaking the softmax temperature would<br>
help.  How did that not give you what you were looking for?</p>
<p>Working with probabilities (after the softmax, not logits), you could<br>
simply take<br>
<code>new_preds = torch.nn.functional.softmax (alpha * old_preds)</code>.<br>
Increasing <code>alpha</code> will drive one of your predicted probabilities to<br>
1, and the others to 0.</p>
<p>If you really want to add a loss term:</p>
<p>Thinking in terms of probabilities, for a problem with <code>nClass</code> classes,<br>
all probabilities being equal to <code>1 / nClass</code> is the least “confident”<br>
prediction, while one probability being <code>1</code> and the others <code>0</code> is the<br>
most “confident.”  So we build a function of each individual probability,<br>
<code>p</code>, that peaks at <code>1 / nClass</code> and goes to zero at <code>0</code> and <code>1</code>.</p>
<p>First, <code>p - p**2</code> peaks at <code>p = 1/2</code> and goes to zero at <code>0</code> and <code>1</code>.</p>
<p>Next, <code>p**(log (2) / log (nClass))</code> maps the interval<br>
<code>[0.0, 1.0]</code> to itself (monotonically), while mapping<br>
<code>(1 / nClass)</code> to <code>1/2</code>.</p>
<p>So, for a tensor of predicted probabilities, <code>p</code>, set<br>
<code>q = p**(log (2) / log (nClass))</code>, calculate your additional loss as<br>
<code>loss = alpha * (q - q**2).sum()</code>, where <code>alpha</code> just tunes how<br>
strong you want this effect to be.</p>
<p>This loss function has the properties you want.  As to whether your<br>
overall scheme will work, I don’t know.</p>
<p>Good luck.</p>
<p>K. Frank</p>

jafioti: <p>Ok, I will try that out. Thanks for the help!</p>
<p>I know it sounds odd, but this classification is not the final answer of the network, only an intermediate selection of other downstream parts of the network. I am currently using a lower temperature on the softmax, and it does pretty well, but I don’t want the network to try and compensate for the colder softmax and still achive a mixture of answers. This is why I want to incentivize the network to stick with one choice and not a mixture. Thanks again for everything!</p>