Cross entropy for multivariate classification

username: hash-ir
cooked: <p>I have a classification problem where each sample output consists of four values each of which is one of the 6 classes, i.e <code>y</code> has shape <code>(batchsize, 4)</code>. So far, I have created a toy model which outputs a logits tensor <code>out</code> having the shape <code>(batchsize, 6(=num_classes), 4)</code>. I am calculating the loss for each of the target value and summing up, followed by backprop:</p>
<pre><code class="lang-python">out = model(in)
out.shape #(batchsize, 6, 4)

out1 = out[:,:,0].squeeze(-1)
out1.shape #(batchsize, 6)
out2 = out[:,:,1].squeeze(-1)
out3 = out[:,:,2].squeeze(-1)
out4 = out[:,:,3].squeeze(-1)
y1 = y[:,0].squeeze(-1)
y1.shape #(batchsize)
y2 = y[:,1].squeeze(-1)
y3 = y[:,2].squeeze(-1)
y4 = y[:,3].squeeze(-1)

total_loss = criterion(out1, y1) + criterion(out2, y2) + criterion(out3, y3) + criterion(out4, y4)

optimizer.zero_grad()
total_loss.backward()
optimizer.step()
</code></pre>
<p>I want to know if this is the correct workaround for multivariate classification</p>

username: KFrank
cooked: <p>Hi Hashir!</p>
<aside class="quote no-group quote-modified" data-username="hash-ir" data-post="1" data-topic="67471" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hash-ir/48/28920_2.png" class="avatar"> hash-ir:</div>
<blockquote>
<p>I have a classification problem where each sample output consists of four values each of which is one of the 6 classes, i.e <code>y</code> has shape <code>(batchsize, 4)</code>. So far, I have created a toy model which outputs a logits tensor <code>out</code> having the shape <code>(batchsize, 6(=num_classes), 4)</code>.</p>
<p>…</p>
<p>I want to know if this is the correct workaround for multivariate classification</p>
</blockquote>
</aside>
<p><em>If</em> I understand what you are trying to do here (and I am not at all<br>
sure that I do), then <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">torch.nn.CrossEntropyLoss</a> should do what<br>
you want without further manipulation.  (Look at the parts where<br>
the documentation talks about the “K-dimensional case.”)</p>
<p>Let the output of your model be a tensor of shape <code>(batchsize, 6, 4)</code>.<br>
I will call this tensor <code>prediction</code>.  I will call the last dimension (the<br>
“4” dimension) the “channel.”  (So in your use case you have four<br>
“channels.”)  For a given sample in the batch and a given channel,<br>
your <code>prediction</code> consists of six logits, one for each of your six classes.</p>
<p>You use your loss function to compare your <code>prediction</code> with your<br>
<code>target</code>.  These are the known class labels that you use for training.<br>
<code>target</code> will be a tensor of shape <code>(batchsize, 4)</code>, and for a given<br>
sample within the batch and for a given channel (the “4” dimension)<br>
you will have a <em>single</em> value that is an integer class label ranging<br>
from 0 to 5.</p>
<p>(Note that <code>prediction</code> and <code>target</code> are not of the same shape:<br>
<code>prediction</code> carries all of the batchsize, class, and channel dimensions,<br>
while <code>target</code>, because it uses a single integer class label per item,<br>
carries only the batchsize and channel dimensions.)</p>
<p>By default, <code>CrossEntropyLoss</code> will average your loss over both the<br>
batchsize and channel dimensions.  If you prefer the sum, then pass<br>
in the optional <code>reduction = 'sum'</code> argument when you construct<br>
your <code>CrossEntropyLoss</code> loss-function object.</p>
<p>Best.</p>
<p>K. Frank</p>

username: hash-ir
cooked: <p><a class="mention" href="/u/kfrank">@KFrank</a> I had a look over the “K-dimensional case” in <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss" rel="nofollow noopener">torch.nn.CrossEntropyLoss</a> and it really helped solve my problem. What I was essentially doing can be done with <code>criterion = torch.nn.CrossEntropyLoss(reduction='sum')</code>.</p>
<p>Thanks for the help!</p>