Error torch.nn.CrossEntropyLoss()

PatricYan: <p>loss = torch.nn.CrossEntropyLoss()<br>
loss_values = loss(train_output, train_label)</p>
<p><em><strong>train_output[128, 10, 27] , train_label[128, 10]</strong></em></p>
<p>get error: <em><strong>Expected target size (128, 27), got torch.Size([128, 10])</strong></em></p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> expects a model output containing logits in the shape <code>[batch_size, nb_classes, *]</code> and a target containing class indices in the range <code>[0, nb_classes-1]</code> and the shape <code>[batch_size, *]</code>.<br>
The <code>*</code> denotes additional dimensions.<br>
Assuming you are working with <code>27</code> classes, you would need to <code>permute</code> the model output such that the class dimension is in <code>dim1</code>.</p>

PatricYan: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="135056">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>output</p>
</blockquote>
</aside>
<p>need to change output [batch, seq, class] to [batch, class, seq], right? but how to change this?</p>

ptrblck: <p>This would work:</p>
<pre><code class="lang-python">output = ... # shape [batch_size, seq_len, nb_classes]
output = output.permute(0, 2, 1).contiguous() # shape [batch_size, nb_classes, seq_len]
</code></pre>

PatricYan: <p>Hi<br>
log_softmax = nn.LogSoftmax(dim=1) with data=[128, 10, 27], apply out=log_softmax(data), out.shape [128, 10, 27], but there are many 0s in out when data.shape [128, 1, 27],</p>

ptrblck: <aside class="quote no-group" data-username="PatricYan" data-post="5" data-topic="135056">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/patricyan/48/40086_2.png" class="avatar"> PatricYan:</div>
<blockquote>
<p>but there are many 0s in out when data.shape [128, 1, 27],</p>
</blockquote>
</aside>
<p>This shape wouldn’t make sense, since you are trying to classify a single class using a multi-class classification loss. The zeros coming from <code>log_softmax</code> indicate a <code>1.</code> probability for this single class.</p>

PatricYan: <p>data is [batch, sequence, class], it is batch 128, sequence length=1, class is 27, as what you have said, I need to change the data.shape [128, 1, 27] to data.shape [128, 27, 1] right?<br>
that is to change [batch, sequence, class] to [batch, class, sequence], right?</p>

ptrblck: <p>Yes, you have to <code>permute</code> the model output to <code>[batch_size, nb_classes, seq_len]</code>.</p>

PatricYan: <p>Thank you, I will try it now.</p>

PatricYan: <p>but after log_softmax = nn.LogSoftmax(dim=1) with data=[128, 27, 10], apply out=log_softmax(data),<br>
I change it [128, 27, 10] to [128, 10, 27], by out.permute(0, 2, 1).contiguous() but get error:<br>
AttributeError: ‘numpy.ndarray’ object has no attribute ‘permute’</p>
<p>I want to get the last sequence result that it is a length of 27 slice.</p>

ptrblck: <p>If the number of classes is <code>27</code>, <code>dim1</code> should have this size.<br>
<code>.permute</code> is a tensor method, not a numpy.array method, so you should apply it on the model output tensor before passing it to the criterion.</p>