CrossEntropyLoss - Why is it possible to have more columns in input than there are classes in the target?

SuryaKiran: <p>[Changing the contents as the question wasn’t articulated well and I didn’t produce a minimum reproducible code]</p>
<p>As per documentation of cross_entropy (<a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss" class="inline-onebox" rel="noopener nofollow ugc">CrossEntropyLoss — PyTorch 2.1 documentation</a>)</p>
<blockquote>
<p>The input is expected to contain raw, unnormalized scores for each class. input has to be a Tensor of size (minibatch,C) for batched case. The target that this criterion expects should contain class indices in the range [0, C) where C is the number of classes</p>
</blockquote>
<p>Why does the following code work -</p>
<pre data-code-wrap="python"><code class="lang-python">input = torch.cat([torch.randn(3,5),torch.randn(3,3)],dim=1)
target = torch.empty(3,dtype=torch.long).random_(5)
F.cross_entropy(input,target)
</code></pre>

Anshumaan_Dash: <p>Its correct as far as syntax is concerned. The <code>logits</code> are expected to be of shape <code>[N,C]</code> where <code>N</code> is the batch size and <code>C</code> is number of classes. The <code>targets</code> tensor is expected to be of shape <code>[N]</code> where each value is the numeric value of the expected/target class in the range <code>[0,C)</code>.</p>
<p>In this case, it seems there are 100 classes.</p>

SuryaKiran: <p>There are only 50 classes. logits00 and logits01 are concatenated along dim=1 which makes C=100 for the value of input argument. Please note that 50 is an illustrative number here. The main idea is that logits00/01 are square tensors of shape, say, CxC. After concatenation, the shape becomes Cx2C. However, target tensor still have C classes only. I hope this makes sense now.</p>

Anshumaan_Dash: <p>Could you please do a <code>target.max()</code> for a couple epochs and let me know the values. Also, which dataset are you using?</p>

SuryaKiran: <p>torch.max is 49 for target. I am using Stanford Online Product dataset</p>

ptrblck: <p>I assume the discussion is targeting the original post before the edit as the current code snippet uses 8 valid classes so I refer to:</p>
<aside class="quote no-group" data-username="SuryaKiran" data-post="1" data-topic="155223">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/suryakiran/48/50927_2.png" class="avatar"> SuryaKiran:</div>
<blockquote>
<p>Why does the following code work -</p>
<pre><code class="lang-auto">input = torch.cat([torch.randn(3,5),torch.randn(3,3)],dim=1)
target = torch.empty(3,dtype=torch.long).random_(5)
F.cross_entropy(input,target)
</code></pre>
</blockquote>
</aside>
<p>This code snippet is working, as it meets the requirements.<br>
The logits have a shape of <code>[batch_size=3, nb_classes=8]</code> while the targets have a shape of <code>[batch_size=3]</code> and contain values in <code>[0, nb_classes-1]</code>. In this case you are limiting the target values to <code>[0, 4]</code> which is inside the valid range of <code>[0, 7]</code>.</p>