For crossentropy loss function why output and target size cannot be the same

Leonardo_Ma: <p>I’m learning to use PyTorch to solve a multi-item, multi-feature, time sequence prediction problem.<br>
In brief, my question is why the size of output and target of crossentropy loss function cannot be the same.<br>
For instance, size of output is (batch_size, num_items), in which each element is a value fitted to the ground true class. Like matrix A:<br>
[[ 0.5 2.1 4.8 3.2 ]<br>
[ 5.0 4.3 2.7 0.2 ]<br>
[ 3.7 0.3 2.0 1.5 ]]<br>
(each row is one batch, and each column is for an item.)<br>
Accordingly, size of the target is also (batch_size, num_items), like matrix B:<br>
[[ 1 0 2 1 ]<br>
[ 0 3 2 1 ]<br>
[ 1 1 1 0 ]]<br>
I suppose the most suitable loss function for my model in pytorch should be crossentropy(one of the pointwise methods?), but if that’s not true, please correct me.<br>
Based on the Doc of pytorch loss functions, Input(output of model): (N, C) where C = number of classes, or (N, C, d_1, d_2, …, d_K) in the case of K-dimensional loss.<br>
Target: (N) where each value is 0:C−1, or (N, d_1, d_2, …, d_K) in the case of K-dimensional loss.<br>
How can I transform or squeeze the shape of my output and target to fit the requirement of crossentropy loss function? Thanks a lot.</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> is used for multi-class classification use cases, i.e. each sample belongs to a single target class.<br>
In your example, <code>num_items</code> in the output would correspond to the number of classes.<br>
Each row would give you the logits of this sample to belong to the class at column x.<br>
Based on this, the target should only contain the target class indices, which are used to calculate the cross entropy loss.</p>
<p>If your targets might take arbitrary values for each element of your output, you might want to use some loss function like <code>nn.MSELoss</code>.</p>

Leonardo_Ma: <p>Hi there, thank you for your quick and detailed reply.<br>
Actually, each element in the output, namely A(i, j), is a single sample. So num_items is not number of classes, but just another dimension of samples like batch_size. The value of A(i, j) is the output of a linear layer, expected to be as close as possible to the ground true value of class, which is B(i, j). In the target B, element values are indeed within (0, C-1) strictly.<br>
So I think it can be seen as a multi-class classification problem. The issue is to correctly classify each element in output A, compared to target B. Problem is output A does not have num_classes dimension, can I just expand A with classes dimension in some way?</p>

ptrblck: <p>In that case your output should have the shape <code>[batch_size, nb_classes, num_items]</code>, which could be seen as a temporal signal, where each sample contains the class logits for <code>nb_classes</code>.<br>
<code>nn.CrossEntropyLoss</code> would then maximize the output of the current neuron which corresponds to the target class.<br>
Would that be possible using your model?</p>

Leonardo_Ma: <p>You mean in the last layer, make the output in shape of [batch_size, nb_classes, num_items]?<br>
That’s possible, let the dense layer to fill dimension of classes. Here in this model, num_classes is about 2000~3000, and the input features of model is just around 100, would that be a problem? Another question is how do I transform the output into same shape of the target, cause the final result need it to be.<br>
Actually the exact value of output is not important in this example. If the target is [0 1 2 3], then output [0 2 5 8] or [3 4 9 21] would be equal to the final result. In this sense, it’s a Learning to Rank problem. I just want to compare nn.CrossEntropyLoss to the other pointwise, pairwise, especially listwise loss functions.</p>

Leonardo_Ma: <p>If you know any better listwise loss functions for LtoR problem in PyTorch, please let me know. I searched the document quite a few times, nn.CrossEntropyLoss is the best I can find… Thank you very much.</p>