Multi-target not supported, Multi-class classification

sinaabdollahi: <p>Hi,<br>
I have a problem when I want to obtain a loss value.<br>
Firstly, for example, I design my own DataLoader including 3 inputs (Each input has 4 one-hot vector) and 3 outputs (Each output has a vector with size = 33), The problem is a multiclass classification (just one class must be activated!):</p>
<p>class MyDataset(Dataset):<br>
def <strong>init</strong>(self):<br>
self.X = Variable(torch.Tensor([<br>
[[0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1], [0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]],</p>
<p>[[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0]],</p>
<p>[[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0]]<br>
]))<br>
self.Y = Variable(torch.Tensor([[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0], [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0]]))<br>
self.len = 3</p>
<pre><code>def __getitem__(self, index):
    return self.X[index], self.Y[index]

def __len__(self):
    return self.len
</code></pre>
<p>and in the forward() method of the Model class, the return line is a 33-dimensional vector :<br>
Z = torch.nn.functional.softmax(self.linear_final(Z), dim=1)</p>
<p>and with the following criterion:<br>
loss = criterion(y_hat, labels.long())<br>
where,<br>
criterion = torch.nn.CrossEntropyLoss()<br>
Then, it launches the following error:<br>
<strong>RuntimeError: multi-target not supported at /opt/conda/conda-bld/pytorch-cpu_1556653093101/work/aten/src/THNN/generic/ClassNLLCriterion.c:20</strong><br>
Does anyone knows the problem?<br>
When I print the y_hat and the labels, I get the following result (batch_size=2):</p>
<p>tensor([[0.0126, 0.0251, 0.0233, 0.0266, 0.0150, 0.0274, 0.0565, 0.0332, 0.0490,<br>
0.0282, 0.0231, 0.0313, 0.0362, 0.0102, 0.0111, 0.0923, 0.0161, 0.0842,<br>
0.0192, 0.0235, 0.0163, 0.0274, 0.0200, 0.0190, 0.0223, 0.0361, 0.0098,<br>
0.0532, 0.0126, 0.0279, 0.0771, 0.0254, 0.0085],<br>
[0.0276, 0.0481, 0.0292, 0.0282, 0.0292, 0.0101, 0.0470, 0.0183, 0.0195,<br>
0.0154, 0.0347, 0.0329, 0.0257, 0.0327, 0.0100, 0.0998, 0.0284, 0.0368,<br>
0.0137, 0.0426, 0.0376, 0.0172, 0.0099, 0.0166, 0.0305, 0.0196, 0.0275,<br>
0.0179, 0.0287, 0.0394, 0.0295, 0.0697, 0.0259]],<br>
grad_fn=)<br>
tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,<br>
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],<br>
[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,<br>
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])</p>
<p>Thank you very much in advance.</p>

KFrank: <p>Hello Sina!</p>
<aside class="quote no-group quote-modified" data-username="sinaabdollahi" data-post="1" data-topic="66466" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sinaabdollahi/48/19583_2.png" class="avatar"> sinaabdollahi:</div>
<blockquote>
<p>The problem is a multiclass classification (just one class must be activated!):<br>
…</p>
<p>and in the forward() method of the Model class, the return line is a 33-dimensional vector :<br>
…<br>
and with the following criterion:<br>
loss = criterion(y_hat, labels.long())<br>
where,<br>
criterion = torch.nn.CrossEntropyLoss()<br>
Then, it launches the following error:<br>
**RuntimeError: multi-target not supported …<br>
…<br>
When I print the y_hat and the labels, I get the following result (batch_size=2):</p>
<p>tensor([[0.0126, 0.0251, 0.0233, 0.0266, 0.0150, 0.0274, 0.0565, 0.0332, 0.0490,<br>
0.0282, 0.0231, 0.0313, 0.0362, 0.0102, 0.0111, 0.0923, 0.0161, 0.0842,<br>
0.0192, 0.0235, 0.0163, 0.0274, 0.0200, 0.0190, 0.0223, 0.0361, 0.0098,<br>
0.0532, 0.0126, 0.0279, 0.0771, 0.0254, 0.0085],<br>
[0.0276, 0.0481, 0.0292, 0.0282, 0.0292, 0.0101, 0.0470, 0.0183, 0.0195,<br>
0.0154, 0.0347, 0.0329, 0.0257, 0.0327, 0.0100, 0.0998, 0.0284, 0.0368,<br>
0.0137, 0.0426, 0.0376, 0.0172, 0.0099, 0.0166, 0.0305, 0.0196, 0.0275,<br>
0.0179, 0.0287, 0.0394, 0.0295, 0.0697, 0.0259]],<br>
grad_fn=)<br>
tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,<br>
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],<br>
[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,<br>
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])</p>
</blockquote>
</aside>
<p>The problem is that <code>CrossEntropyLoss</code> takes “labels” (often<br>
called “targets”) that are integer class labels (in the range<br>
<code>[0, nClass)</code>, i.e., <code>0 &lt;= label &lt; nClass</code>), rather than<br>
one-hot encoded vectors.</p>
<p>Your example <code>y_hat</code> and <code>labels</code> correspond to a batch size<br>
of two, and a number of classes <code>nClass = 33</code>.  Therefore a<br>
batch of labels should be a tensor of shape <code>(2, )</code>, i.e., just<br>
two class-label integers in the range <code>[0, 33)</code>.  You are passing<br>
<code>CrossEntropyLoss</code> a (one-hot encoded)<code>labels</code> vector of shape<br>
<code>(2, 33)</code>.  (Your <code>y_hat</code> has the correct shape, <code>(2, 33)</code>, i.e.,<br>
<code>(batchSize, nClass)</code>, that is expected by <code>CrossEntropyLoss</code>.)</p>
<p>Edit:</p>
<p>Also, <code>CrossEntropyLoss</code> expects “raw scores” (<em>logits</em>) for its<br>
predictions (your <code>y_hat</code>), not probabilities.  So you do <em>not</em> want<br>
a <code>softmax()</code> for the output of your model.  Just use the output<br>
of your last linear layer:</p>
<pre data-code-wrap="python"><code class="lang-python">Z = self.linear_final(Z)
return Z
</code></pre>
<p>rather than:</p>
<pre data-code-wrap="python"><code class="lang-python">Z = torch.nn.functional.softmax(self.linear_final(Z), dim=1)
return Z
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

sinaabdollahi: <p>Thanks Frank for your help.</p>

sinaabdollahi: <p>Dear Frank,<br>
How about for testing phase?<br>
How can I find accuracy and error for test set?</p>

KFrank: <p>Hello Sina!</p>
<aside class="quote no-group" data-username="sinaabdollahi" data-post="4" data-topic="66466" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sinaabdollahi/48/19583_2.png" class="avatar"> sinaabdollahi:</div>
<blockquote>
<p>How can I find accuracy and error for test set?</p>
</blockquote>
</aside>
<p>The conventional way to calculate the accuracy:</p>
<pre data-code-wrap="python"><code class="lang-python">y_hat_test = model (inputs_test)
dummy, preds_test = torch.max (y_hat_test, dim = 1)
accuracy_test = (preds_test == labels_test).long().sum().float() /  preds_test.size()[0]
</code></pre>
<p>That is, you need to convert the raw scores (<em>logits</em>) for each<br>
class output by your model into predictions for a single, specific<br>
class.  Usually you just use the class with the largest logit as<br>
the prediction.</p>
<p>(In newer versions of pytorch you can use the <code>torch.argmax()</code><br>
function, <code>preds_test = torch.argmax (y_hat_test, dim = 1)</code><br>
(and dispense with the explicit conversion to <code>.long()</code> before the<br>
<code>.sum()</code>).</p>
<p>I’m not sure what you mean by test-set error.  If you mean your loss<br>
function,you calculate it the same way you would for a training batch<br>
(but wrap it in a <code>volatile = True</code> <code>Variable</code>, or, with newer versions,<br>
use <code>with torch.no_grad():</code>).</p>
<p>Note, you do <em>not</em> want <code>softmax()</code> as the final layer of your model.<br>
I’ve edited my previous post to clarify this.</p>
<p>Good luck.</p>
<p>K. Frank</p>