CrossEntropyLoss for multiple output classification

username: sureshj
cooked: <p>Given an input, I would like to do multiple classification tasks.</p>
<p>Each input needs to be classified into one of 5 classes. There are 6 such classification tasks to be done. I could build six separate Linear(some_number, 5) layers and return the result as tuple in the forward() function. Then call the loss function 6 times and sum the losses to produce the overall loss.</p>
<p>I am wondering if I could do this better than this. For example, can I have a single Linear(some_number, 5*6) as the output. Then reshape the logits to (6,5) and use. The documentation for CrossEntropyLoss mentions about “K-dimensional loss”. Is it for use cases as here? If so, can anyone share a sample code (When I tried to use I got confused about how to reshape the output).</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html</a></p>

username: KFrank
cooked: <p>Hi Suresh!</p>
<aside class="quote no-group" data-username="sureshj" data-post="1" data-topic="111660" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/7c8e57/48.png" class="avatar"> sureshj:</div>
<blockquote>
<p>I could build six separate Linear(some_number, 5) layers and return the result as tuple in the forward() function. Then call the loss function 6 times and sum the losses to produce the overall loss.</p>
</blockquote>
</aside>
<p>This is a perfectly reasonable approach and will work fine.  It may well<br>
introduce de minimis inefficiency, but any such cost will be negligible<br>
compared with the cost of backpropagation.</p>
<aside class="quote no-group">
<blockquote>
<p>Then reshape the logits to (6,5) and use. The documentation for CrossEntropyLoss mentions about “K-dimensional loss”.</p>
</blockquote>
</aside>
<p>This is also fine.  I would probably use this second approach – not for<br>
efficiency reasons, but because it seems a little more organized to me.</p>
<p>Note, for the “K-dimensional loss” case, you will want to reshape your<br>
logits to have shape <code>[nBatch, 5, 6]</code> (not <code>[nBatch, 6, 5]</code>).</p>
<p>Let me use the following terms: <code>nBatch</code> (batch size), <code>nClass = 5</code>,<br>
and <code>nTask = 6</code>.</p>
<p>The output of your model should have shape<br>
<code>[nBatch, nClass * nTask]</code>, you will reshape it, e.g.,<br>
<code>output.reshape (-1, nClass, nTask)</code>, to have shape<br>
<code>[nBatch, nClass, nTask]</code>, and then pass it as the <code>input</code> (prediction)<br>
into <code>CrossEntropyLoss</code>.  When you do this, the <code>target</code> you pass into<br>
<code>CrossEntropyLoss</code> must have shape <code>[nBatch, nTask]</code> (no class<br>
dimension), and consist of integer class labels that run from <code>0</code> to<br>
<code>nClass - 1</code>.</p>
<p>(This second scheme only works, of course, if your multiple classification<br>
tasks all have the same number of classes.  If not, you would use your<br>
first scheme.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: sureshj
cooked: <p>Frank,<br>
Thanks for the detailed reply.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="111660">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Note, for the “K-dimensional loss” case, you will want to reshape your<br>
logits to have shape <code>[nBatch, 5, 6]</code> (not <code>[nBatch, 6, 5]</code>).</p>
</blockquote>
</aside>
<p>I am somehow finding this order less intuitive for me - <code>nClass</code> appearing before <code>nTask</code> .</p>
<p>I will be using <code>torch.nn.functional.softmax</code> to use the logits outside the loss function. I will use the <code>dim</code> parameter to go along the correct dimension.</p>
<p>Thanks again.</p>