2D CrossEntropyLoss for one-hot targets?

username: richieYT-wan
cooked: <p>From the definition of CrossEntropyLoss:</p>
<blockquote>
<p>input has to be a 2D Tensor of size (minibatch, C).<br>
This criterion expects a class index (0 to C-1) as the target for each value of a 1D tensor of size</p>
</blockquote>
<p>My last dense layer gives dim (mini_batch, 23*N_classes), then I reshape it to (mini_batch, 23, N_classes)<br>
So for my task, I reshape the output of the last dense layer and softmax it along dim=2, so the output and target is of the following shape :</p>
<p>I get a predicted output that has shape (minibatch, 23, N_classes), so a given outputted sample is basically the predicted one_hot vector (each of the 23 rows can only be of one class)</p>
<p>What I want to do, basically, is to use CrossEntropyLoss for each of these rows of each sample, i.e. a 2D CrossEntropyLoss.</p>
<p>for a single sample, a given output (after softmax), looks something like : (Here is an example with smaller dimension, (n=2, 3, 5) instead of (n, 23, 25)</p>
<pre><code class="lang-auto">&gt;&gt;&gt;target
tensor([[[0, 1, 0, 0, 0],
         [0, 0, 0, 0, 1],
         [0, 1, 0, 0, 0]],

        [[1, 0, 0, 0, 0],
         [0, 0, 0, 1, 0],
         [0, 0, 0, 1, 0]]])

&gt;&gt;&gt;x = torch.randn((2, 3, 5)).view(-1,3,5)
&gt;&gt;&gt;out = F.softmax(x,dim = 2)
&gt;&gt;&gt;out 
tensor([[[0.2093, 0.1281, 0.5016, 0.0836, 0.0773],
         [0.1146, 0.0575, 0.1194, 0.4064, 0.3021],
         [0.2026, 0.2265, 0.3767, 0.0556, 0.1385]],

        [[0.0473, 0.2789, 0.0782, 0.3650, 0.2306],
         [0.0054, 0.2677, 0.0643, 0.5199, 0.1427],
         [0.4490, 0.2397, 0.2088, 0.0787, 0.0237]]])
</code></pre>
<p>Is there a way to use the CrossEntropyLoss for a 2D target (so final dimension would be 3D, (batchsize, dim1, dim2)), i.e.</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
loss = criterion(target, out)
</code></pre>

username: KFrank
cooked: <p>Hi Richie!</p>
<aside class="quote no-group" data-username="richieYT-wan" data-post="1" data-topic="117545" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/richieyt-wan/48/36893_2.png" class="avatar"> richieYT-wan:</div>
<blockquote>
<p>Is there a way to use the CrossEntropyLoss for a 2D target (so final dimension would be 3D, (batchsize, dim1, dim2))</p>
</blockquote>
</aside>
<p>Yes.  <a href="https://pytorch.org/docs/1.8.1/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">CrossEntropyLoss</a> supports what it calls the “K-dimensional case.”</p>
<p>Note, pytorch’s <code>CrossEntropyLoss</code> does not accept a one-hot-encoded<br>
<code>target</code> – you have to use integer class labels instead.</p>
<p>Let’s call your value <code>23</code> <code>length</code>.  Your <code>input</code> (the prediction generated<br>
by your network) should have shape<br>
<code>[mini_batch, N_classes, length]</code></p>
<p>Your <code>target</code> (the ground-truth labels) should have shape<br>
<code>[mini_batch, length]</code> <em>without</em> an <code>N_classes</code> dimension and should<br>
be integer class labels in <code>[0, N_classes - 1]</code>, inclusive.  (In particular,<br>
they are <em>not</em> one-hot encoded.)</p>
<p>Last, do not pass the output of your network through <code>softmax()</code>;<br>
<code>CrossEntropyLoss</code> has, in effect, <code>softmax()</code> built in (and expects<br>
its <code>input</code> to be raw-score <em>logits</em>).</p>
<p>Best.</p>
<p>K. Frank</p>

username: richieYT-wan
cooked: <p>Thanks a lot for the reply!</p>
<p>I didn’t quite understand that the ground truth couldn’t be one-hot encoded, while the output of my network should have switched shape [batch, N_classes, length] instead of [batch, length, N_classes].</p>
<p>But now, say my vector also has a positional encoding, so it is no longer of dimension [batch, length, N_classes] but now [batch, length, N_classes + 4], and can take two positive values (one for N_class, and one for the positional encoding), ex :</p>
<pre><code class="lang-auto">label = [0,1,0,0,0,...] #shape [1, 1, 21] for a single item and single row
pos = [0, 0, 1, 0] #shape [1, 1, 4] for a single item and single row
final_encoding = torch.cat([label,pos], dim = 2) #shape [1,1,25] for a single item/row
</code></pre>
<p>So now, my network predict both the class AND the positional encoding. If I slice my tensor and then run two different CrossEntropyLoss for each of the subtensor, will Autograd still work ?</p>
<p>Best,<br>
Richie</p>

username: KFrank
cooked: <p>Hi Richie!</p>
<aside class="quote no-group" data-username="richieYT-wan" data-post="3" data-topic="117545" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/richieyt-wan/48/36893_2.png" class="avatar"> richieYT-wan:</div>
<blockquote>
<p>I didn’t quite understand that the ground truth couldn’t be one-hot encoded, while the output of my network should have switched shape [batch, N_classes, length] instead of [batch, length, N_classes].</p>
</blockquote>
</aside>
<p>Just to be clear, if you want to use pytorch’s <code>CrossEntropyLoss</code>, you<br>
have to do it this way.</p>
<p><code>CrossEntropyLoss</code> is written to take as its <code>target</code> integer class labels<br>
rather than one-hot-encoded labels and doesn’t have a built-in feature<br>
to use one-hot encoding.  Similarly, <code>CrossEntropyLoss</code> expects the<br>
second dimension of its <code>input</code> to be the class dimension, so that <code>input</code><br>
has shape <code>[mini_batch, N_classes, d1, d2, ...]</code>, where <code>d1</code>, etc.,<br>
are optional additional dimensions for the “K-dimensional case.”</p>
<p>(That’s just how it works.  If you need something else, you would have<br>
to write your own version or write a wrapper for <code>CrossEntropyLoss</code>.)</p>
<aside class="quote no-group">
<blockquote>
<p>If I slice my tensor and then run two different CrossEntropyLoss for each of the subtensor, will Autograd still work ?</p>
</blockquote>
</aside>
<p>Yes, doing this makes perfect sense.</p>
<p>I don’t understand your use case or your “positional encoding,” so let<br>
me use an artificial example:</p>
<p>Let’s say you have a set of images and each image contains both a<br>
picture of an animal – cat, dog, mouse – and a digit – 0 though 9.  You<br>
want your network to classify both at the same time.</p>
<p>The input to your network (not to the loss criterion) is a batch of images<br>
with shape <code>[mini_batch, height, width]</code>, and your ground-truth<br>
<code>labels</code> have shape <code>[mini_batch, 2]</code>.  The “animal” labels are<br>
<code>labels[:, 0]</code>, a vector of length <code>mini_batch</code>, and are integer class<br>
labels with values <code>0</code> (cat), <code>1</code> (dog), and <code>2</code> (mouse).  The digit labels<br>
are <code>labels[:, 1]</code> (also, of course, of length <code>mini_batch</code>) and have<br>
values <code>0</code> through <code>9</code>.</p>
<p>So far, so good.</p>
<p>Your network architecture is whatever it is – perhaps some initial<br>
convolutional layers because you’re classifying two-dimensional<br>
images – followed by some fully-connected layers.  But your <em>final</em><br>
fully-connected layer should have <code>nAnimal = 3</code> plus <code>nDigit = 10</code><br>
output features so that the output of your network (the <code>input</code> to your<br>
loss criteria) is a tensor of shape <code>[mini_batch, nAnimal + nDigit]</code>.</p>
<p>Now you slice <code>input</code> and <code>labels</code> and apply <code>CrossEntropyLoss</code> twice:</p>
<pre data-code-wrap="python"><code class="lang-python">lossAnimal = torch.nn.CrossEntropyLoss() (input[:, :nAnimal], target[:, 0])
lossDigit = torch.nn.CrossEntropyLoss() (input[:, nAnimal:], target[:, 1])
lossTotal = lossAnimal + lossDigit
lossTotal.backward()
</code></pre>
<p>It’s fully legitimate to build your total loss out of multiple losses – in this<br>
case two cross-entropy losses – and it will work just fine with autograd.</p>
<p>As you train your network by backpropagating <code>totalLoss</code>, your network,<br>
and, in particular, your final fully-connected layer, will 'learn" that the first<br>
<code>nAnimal</code> output features of the final layer are the predicted raw-score<br>
logits for the animal in the image, and that the remaining <code>nDigit</code> output<br>
features are the predicted logits for the digit in the image.</p>
<p>Best.</p>
<p>K. Frank</p>

username: richieYT-wan
cooked: <p>Ah, thank you very much for the example, it is much clearer now. Because in my head, I was stuck with doing Softmax as the activation of my last layer (instead of leaving it as the logits) and absolutely wanted my network to apply the softmax and thus couldn’t do two labels at the same time given the dimension of the last layer.</p>