Need Guidance on Loss Function for Multi-Label/Multi-Output Classification

MarioAvolio: <p>Hello everyone,</p>
<p>I’m encountering a challenge with the choice of the loss function for a multi-label/multi-output classification problem in PyTorch. I hope you can provide some insights and guidance.</p>
<p><strong>Problem Description:</strong> I’m working on a problem where we have 47 labels, and each label can belong to one of three possible classes (0, 1, -1). So essentially, it’s a multi-label classification problem with three classes. I’ve never worked on such a problem before and I’m a bit confused about which loss function to use.</p>
<p><strong>Example Data:</strong> Here’s a simplified example of my data:</p>
<pre><code class="lang-auto">y: tensor([[[ 1., -1., -1., 1., -1., 1., ... ],
           ...
           [ 0., 1., -1., 0., 1., -1., ... ]],

y_pred: tensor([[ 0.0156, 0.0109, 0.0048, 0.0125, 0.0005, ... ],
                ...
                [ 0.0172, 0.0048, 0.0098, 0.0057, -0.0036, ... ]],

</code></pre>
<p><strong>Data Shapes:</strong></p>
<ul>
<li>y shape: torch.Size([8, 47])</li>
<li>y_pred shape: torch.Size([8, 47])</li>
</ul>
<p><strong>Dilemma:</strong> I’m aware that the Binary Cross Entropy loss function technically can’t be used here because there are three classes. Cross-entropy seems like the right choice for classification problems like this, but I’m struggling with the fact that I need to compare probabilities (ranging from 0 to 1) with labels (-1, 0, 1).</p>
<p>I’m also used to working with one-hot encoding, but I’m unsure if it’s applicable in this case.</p>
<p>I’d greatly appreciate any advice or suggestions on how to approach this problem. What loss function should I use, and how should I handle the class labels and predictions?</p>
<p>Thank you in advance for your help!</p>

KFrank: <p>Hi Mario!</p>
<p>The short story:  Treat this as what pytorch’s <a href="https://pytorch.org/docs/2.1/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">CrossEntropyLoss</a> calls the<br>
“K-dimensional case” (with K = 1), where the trailing “channels” dimension<br>
is of length 47.</p>
<aside class="quote no-group" data-username="MarioAvolio" data-post="1" data-topic="189373" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marioavolio/48/63977_2.png" class="avatar"> MarioAvolio:</div>
<blockquote>
<p>we have 47 labels, and each label can belong to one of three possible classes (0, 1, -1). So essentially, it’s a multi-label classification problem with three classes.</p>
</blockquote>
</aside>
<p>Yes, this is an appropriate description.  You have 47 three-class classification<br>
problems that all take the same input so you get better throughput and likely<br>
better training if you build one model that performs the 47 classifications at the<br>
same time.</p>
<p>We will group these 47 problems together into a trailing “labels” dimension<br>
that I will call the “channels” dimension.</p>
<aside class="quote no-group">
<blockquote>
<p>I’ve never worked on such a problem before and I’m a bit confused about which loss function to use.</p>
<pre><code class="lang-auto">y: tensor([[[ 1., -1., -1., 1., -1., 1., ... ],
           ...
           [ 0., 1., -1., 0., 1., -1., ... ]],
</code></pre>
</blockquote>
</aside>
<p>First, shift the values in your <code>y</code> tensor (the ground-truth <code>target</code> tensor) so that<br>
they run over <code>{0, 1, 2}</code> and will be proper categorical class labels from the<br>
perspective of pytorch’s <code>CrossEntropyLoss</code>.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">y_pred: tensor([[ 0.0156, 0.0109, 0.0048, 0.0125, 0.0005, ... ],
                ...
                [ 0.0172, 0.0048, 0.0098, 0.0057, -0.0036, ... ]],

</code></pre>
</blockquote>
</aside>
<p>You will want your <code>y_pred</code> tensor (the output of your model that will be the<br>
<code>input</code> to <code>CrossEntropyLoss</code>) to be (unnormalized) log-probabilities for your<br>
three classes.</p>
<p>(Note that pytorch’s <code>CrossEntropyLoss</code> has started referring to its <code>input</code><br>
as “unnormalized logits”.  This is technically incorrect –  they are actually<br>
log-probabilities – but the point being made – that <code>input</code> lives in “log space”<br>
rather than “probability space” – is basically correct.)</p>
<aside class="quote no-group">
<blockquote>
<p><strong>Data Shapes:</strong></p>
<ul>
<li>y shape: torch.Size([8, 47])</li>
</ul>
</blockquote>
</aside>
<p>I assume that the <code>8</code> is your batch dimension.  So for each batch element and<br>
each of your 47 “channels” you have a three-class categorical class label (that<br>
takes values in <code>{0, 1, 2}</code>).  (Note, the example <code>y</code> you gave above appears to<br>
be a three-dimensional tensor.  You do want a two-dimensional tensor of shape<br>
<code>[nBatch = 8, channels = 47]</code>.)</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li>y_pred shape: torch.Size([8, 47])</li>
</ul>
</blockquote>
</aside>
<p>Your <code>y_pred</code> should have three (unnormalized) class log-probabilities (which<br>
is to say, one for each of your three classes) for each batch element and channel.<br>
It should therefore be of shape <code>[nBatch = 8, nClass = 3, channels = 47]</code><br>
and have values that run from <code>-inf</code> to <code>inf</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>Cross-entropy seems like the right choice for classification problems like this</p>
</blockquote>
</aside>
<p>Correct.</p>
<aside class="quote no-group">
<blockquote>
<p>I’m struggling with the fact that I need to compare probabilities (ranging from 0 to 1) with labels (-1, 0, 1).</p>
</blockquote>
</aside>
<p>Conceptually you do this by predicting (for each batch element and channel) a<br>
set of three probabilities (one for each class) that sum to one.</p>
<p>In practice you do this by predicting three <em>unnormalized</em> log-probabilities.  They<br>
are unnormalized in the sense that when converted to probabilities, they <em>don’t</em><br>
necessarily sum to one.  The conversion to probabilities and normalization (that<br>
is, that they sum to one) occurs, in effect, inside of <code>CrossEntropyLoss</code>.</p>
<p>Probably the easiest way to do this is to have the final layer of your model be<br>
<code>lin = Linear (in_features = in_features, out_features = 3 * 47)</code><br>
(for your use case of <code>nClass = 3</code> and <code>channels = 47</code>)  and group the classes<br>
together by reshaping the output: <code>output = lin (t).reshape (-1, 3, 47)</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>I’m also used to working with one-hot encoding</p>
</blockquote>
</aside>
<p>Clear your mind of <strong>ALL</strong> thoughts of “one-hot encoding.”</p>
<p>Best.</p>
<p>K. Frank</p>