Multiple categorical outcomes and each has multiple classes

username: zhenhua-wang
cooked: <p>Hi everyone,<br>
I am working on a classification question, where the outcomes contain more than one categorical variable. Each of them has multiple classes. For example, y could be [y1, y2, y3, y4, y5] and each y_i is a categorical variable with multiple classes.</p>
<p>The only solution that I could think of is to have separate output layers (softmax) for each categorical variable. This is very tedious. I wonder is there any other easier methods to deal with this type of question? Thank you!</p>

username: ptrblck
cooked: <p>Your use case sounds like a multi-label classification, where each sample might belong to zero, one or multiple classes.<br>
If that’s the case, you could use a final linear layer with <code>out_features=nb_classes</code>, use a one-hot encoded target in the same shape, and apply <code>nn.BCEWithLogitsLoss</code> as your loss function.</p>

username: KFrank
cooked: <p>Hello Zhenhua!</p>
<aside class="quote no-group" data-username="zhenhua-wang" data-post="1" data-topic="87406" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhenhua-wang/48/26201_2.png" class="avatar"> zhenhua-wang:</div>
<blockquote>
<p>For example, y could be [y1, y2, y3, y4, y5] and each y_i is a categorical variable with multiple classes.</p>
</blockquote>
</aside>
<p>Let me tell you how I understand your question.</p>
<p>I don’t see the use case you describe as being a multi-label,<br>
multi-class problem.  I guess I would call it something like<br>
“multiple multi-class.”</p>
<p>That is (and to make things simpler, let me speak of just <code>y_1</code><br>
and <code>y_2</code>), let’s say <code>y_1</code> indicates one of three colors<br>
(<code>0 = red</code>, <code>1 = green</code>, <code>2 = blue</code>), and that <code>y_2</code> indicates<br>
one of four animals (<code>0 = parrot</code>, <code>1 = cat</code>, <code>2 = cobra</code>,<br>
<code>3 = shark</code>).  You classify images according to <code>y_1</code> and <code>y_2</code>.<br>
So each image is give exactly one color label and exactly one<br>
animal label.  Therefore this is a set of two single-label,<br>
multi-class classification problems (running the same network<br>
on the same data).</p>
<p>Is this what you mean?</p>
<aside class="quote no-group">
<blockquote>
<p>The only solution that I could think of is to have separate output layers (softmax) for each categorical variable.</p>
</blockquote>
</aside>
<p>If my description above represents your use case, I don’t think<br>
that there is anything <em>entirely</em> built in to pytorch that will do<br>
what you want.  But doing it in pieces should be straightforward.</p>
<p>For the <code>y_1</code>, <code>y_2</code> example, I would do something like this:</p>
<p>Let your final layer have 7 outputs (3 for <code>y_1</code> and 4 for <code>y_2</code>).<br>
Don’t pass it through any kind of <code>softmax()</code>.  Let your target<br>
data be a pair of integer categorical class labels, the first<br>
for <code>y_1</code> that runs over <code>{0, 1, 2}</code>, and the second for <code>y_2</code><br>
that runs over {0, 1, 2, 3}.</p>
<p>The output of your model, <code>pred</code>, will have shape <code>[nBatch, 7]</code><br>
and your batch of targets, <code>targ</code>, will have shape <code>[nBatch, 2]</code>.<br>
Then calculate:</p>
<pre><code class="lang-auto">loss_y_1 = cross_entropy (pred[:, :3], targ[:, 0])
loss_y_2 = cross_entropy (pred[:, 3:], targ[:, 1])
loss = loss_y_1 + loss_y_2
</code></pre>
<p>(Remember, <code>cross-entropy()</code> has, in effect, <code>softmax()</code> built in.<br>
So it might better have been called <code>cross_entropy_with_logits()</code>.)</p>
<p>As an aside, you could – mathematically equivalently – use two<br>
final output layers.</p>
<p>Let’s say your second-to-last layer has <code>32</code> outputs.  Then you<br>
could feed these outputs to both a <code>Linear (32, 3)</code> and a<br>
<code>Linear (32, 4)</code>.  Then use the outputs of these two <code>Linear</code><br>
layers to calculate <code>loss_y_1</code> and <code>loss_y_2</code>, as above.</p>
<p>But, again, there is no* mathematical difference between these<br>
two approaches.  To me, the first approach of using a single final<br>
layer with 7 outputs seems a little cleaner and less error-prone.</p>
<p>*) I believe that the default initialization of the <code>Linear</code>s’ weights<br>
will scale the random initial weights differently.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: zhenhua-wang
cooked: <p>Thank you, this is exactly what I mean!</p>

username: zhenhua-wang
cooked: <p>Thank you. Even though this is not a multi-label classification, it is still good to learn this!</p>

username: Vishay_Raina
cooked: <p>how is<br>
loss = loss_y_1 + loss_y_2<br>
loss.backward()</p>
<p>different than</p>
<p>loss_y_1.backward()<br>
loss_y_2.backward()</p>

username: KFrank
cooked: <p>Hi Vishay!</p>
<aside class="quote no-group" data-username="Vishay_Raina" data-post="6" data-topic="87406" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vishay_raina/48/25733_2.png" class="avatar"> Vishay_Raina:</div>
<blockquote>
<p>how is<br>
loss = loss_y_1 + loss_y_2<br>
loss.backward()</p>
<p>different than</p>
<p>loss_y_1.backward()<br>
loss_y_2.backward()</p>
</blockquote>
</aside>
<p>In the first case you backpropagate through the network once; in the<br>
second, twice.  Backpropagating once saves on computation, and these<br>
savings can be significant.</p>
<p>Also, if you want to use the second approach where you backpropagate<br>
twice, you will have to monkey around with setting <code>retain_graph=True</code>.</p>
<p>I can’t think of a use case where I would prefer the second approach.</p>
<p>Best.</p>
<p>K. Frank</p>