Custom loss for single-label, multi-class problem

Saswat: <p>I have a single-label, multi-class classification problem, i.e., a given sample is in exactly one class (say, class 3), but for training purposes, predicting class 2 or 5 is still okay to not penalise the model that heavily.</p>
<p>For example, the ground truth for 1 sample is [0,1,1,0,1] of 5 classes, instead of a one-hot vector. This implies that, the model predicting <strong>any one</strong> of the above classes (2,3 or 5) is fine.</p>
<p>For every batch, the predicted output dimension is of the shape <code>bs x n x nc</code>, where bs is the batch size, n is the number of samples per point and nc is the number of classes. The ground truth is also of the same shape as the predicted tensor.</p>
<p>For every batch, I’m expecting my loss function to compare <code>n</code> tensors across <code>nc</code> classes and then average it across <code>n</code>.</p>
<p>Eg: When dimensions are 32 x 8 x 5000. There are 32 <em>batch points</em> in a batch (for bs=32). Each batch point has 8 <em>vector points</em>, and each vector point has 5000 classes. For a given batch point, I wish to compute loss across all (8) <em>vector points</em>, compute their average and do so for the rest of the <em>batch points</em> (32). Final loss would be loss over all losses from each <em>batch point</em>.</p>
<p>How can I approach designing such a loss function? Any help would be deeply appreciated</p>
<p>P.S.: Let me know if the question is ambiguous</p>

KFrank: <p>Hi Saswat!</p>
<p>You’ve never really told us what your use case is, that is what real-world<br>
problem you’re trying to solve or what your input data, predictions, and<br>
ground-truth labels are supposed to mean.</p>
<aside class="quote no-group" data-username="Saswat" data-post="1" data-topic="163271" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saswat/48/49862_2.png" class="avatar"> Saswat:</div>
<blockquote>
<p>For example, the ground truth for 1 sample is [0,1,1,0,1] of 5 classes, instead of a one-hot vector. This implies that, the model predicting <strong>any one</strong> of the above classes (2,3 or 5) is fine.</p>
</blockquote>
</aside>
<p>I doubt that this is really exactly what you want to be doing.</p>
<p>But to answer your specific question:</p>
<p>Taking guidance from cross entropy, I would suggest <code>loss = -log (P_all)</code>,<br>
where <code>P_all</code> is the sum of the predicted probabilities for all of the classes<br>
that “are fine.”  Note, when you only have a single ground-truth class, this is<br>
just the conventional cross entropy.</p>
<p>(For reasons of numerical stability, it will be best to predict log-probabilities<br>
and use the “log-sum-exp” trick to compute <code>log (P_all)</code>.)</p>
<p>Best.</p>
<p>K. Frank</p>

Saswat: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a><br>
Sorry for the confusion</p>
<p>I’m working on developing an ML-based packer which aims at solving multi-dimensional knapsack problems (I.e., given x items, try to fit them in as less knapsacks as possible). I’m trying to supervise it by using a conventional knapsack solver. The solver gives me the top 50 knapsacks of varying capacities that can suitably pack the given requests.</p>
<p>The input is a set of <code>n</code> requests which need to be fit in the knapsacks. The ground truth output is the set of bin indices which the solver has given me (hence, multiple 1s).</p>
<p>Let me know if I’m clear on the problem definition</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="163271">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>(For reasons of numerical stability, it will be best to predict log-probabilities<br>
and use the “log-sum-exp” trick to compute <code>log (P_all)</code>.)</p>
</blockquote>
</aside>
<p>I didn’t quite follow you here. Did you mean, to take exp, divide by the sum of all exps and take a log of it all?</p>
<p>Also, how would you recommend averaging across a particular dimension?</p>

KFrank: <p>Hi Saswat!</p>
<aside class="quote no-group" data-username="Saswat" data-post="3" data-topic="163271" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saswat/48/49862_2.png" class="avatar"> Saswat:</div>
<blockquote>
<p>I’m working on developing an ML-based packer which aims at solving multi-dimensional knapsack problems</p>
</blockquote>
</aside>
<p>I do not have any intuition about how one might use a neural network to<br>
learn to solve the knapsack problem.  (My gut reaction is that it might be<br>
hard and that approaching it as a classification problem doesn’t feel right<br>
to me.)</p>
<aside class="quote no-group">
<blockquote>
<p>I didn’t quite follow you here. Did you mean, to take exp, divide by the sum of all exps and take a log of it all?</p>
</blockquote>
</aside>
<p>Take a look at Wikipedia’s explanation of the <a href="https://en.wikipedia.org/wiki/LogSumExp#log-sum-exp_trick_for_log-domain_calculations" rel="noopener nofollow ugc">log-sum-exp trick.</a></p>
<p>Also, pytorch offers a built-in <a href="https://pytorch.org/docs/1.12/generated/torch.logsumexp.html" rel="noopener nofollow ugc">logsumexp()</a> function.</p>
<p>Best.</p>
<p>K. Frank</p>