RuntimeError: multi-target not supported at /Users/soumith/code/builder/wheel/pytorch-src/aten/src/THNN/generic/ClassNLLCriterion.c:21

zooraff: <p>How to map 1-D “y” labels (0-4) vector to logits output from NN for a Loss function?</p>
<p>Hello,</p>
<p>I am new to PyTorch and ML. After reading many forums and materials, I cannot find the best way to get a solution.</p>
<p>I have following INPUTS:</p>
<blockquote>
<p>train file stored in h5 format with following labels:<br>
[y] - ground truth labels (integers 0,1,2,3,4) - isolated as separate tensor for training purposes (Y), through pandas DF<br>
[x1:x120] - feature vectors x1-x120, which gives 120 features as input vector, isolated as well for training (X1-X120 tensor coming from pandas DF)<br>
around 45000 training instances (rows)</p>
</blockquote>
<p>So I have constructed fc NN like this:<br>
L1 - [120-100] nodes for input vectors<br>
L2 - [100-50]<br>
L3 - [50-20]<br>
L4 - [20-5] nodes for outputs, either logits or probabilities after applying softmax</p>
<h2><a name="each-layer-being-activated-through-relu-no-softmax-at-the-end-in-1-version-just-the-output-itself-logits-right-1" class="anchor" href="#each-layer-being-activated-through-relu-no-softmax-at-the-end-in-1-version-just-the-output-itself-logits-right-1"></a>Each layer being activated through ReLU, no softmax at the end in 1 version, just the output itself (logits, right?)</h2>
<p>class Net(nn.Module):</p>
<pre><code>def __init__(self):
    super(Net, self).__init__()
    # an affine operation: y = Wx + b
    self.fc1 = nn.Linear(120, 100)
    self.fc2 = nn.Linear(100, 50)
    self.fc3 = nn.Linear(50, 20)
    self.fc4 = nn.Linear(20, 5)

def forward(self, x):
    #x = x.view(-1, 120)
    out1 = F.relu(self.fc1(x))
    out2 = F.relu(self.fc2(out1))
    out3 = F.relu(self.fc3(out2))
    y_pred = self.fc4(out3)
    return F.log_softmax(y_pred)
    #return y_pred
</code></pre>
<h2><a name="model-net-printmodel-2" class="anchor" href="#model-net-printmodel-2"></a>model = Net()<br>
print(model)</h2>
<p>What I am struggling with, is how to convert either logits or 1-D y labels array to match dimensions for CrossEntropyLoss or even NLL when applied Softmax?</p>
<p>It gets runtime error - multi target not supported etc… (1D vs 5D)<br>
Shall I convert ground truth labels to one-hot coding, having some array like:<br>
y<br>
[00100] = 2<br>
[01000] = 1<br>
[00001] = 4<br>
???</p>
<p>My Data Loader:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/a5244432cc8843b7a1397c334870d37ef5a5004f.png" data-download-href="https://discuss.pytorch.org/uploads/default/a5244432cc8843b7a1397c334870d37ef5a5004f" title="46"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/a5244432cc8843b7a1397c334870d37ef5a5004f_2_690x205.png" alt="46" data-base62-sha1="nyUtWL6mBLbNsiObqnXYevX3CqP" width="690" height="205" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/a5244432cc8843b7a1397c334870d37ef5a5004f_2_690x205.png, https://discuss.pytorch.org/uploads/default/optimized/2X/a/a5244432cc8843b7a1397c334870d37ef5a5004f_2_1035x307.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/a/a5244432cc8843b7a1397c334870d37ef5a5004f_2_1380x410.png 2x" data-dominant-color="EDEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">46</span><span class="informations">1686×502 77 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>dataset = TrainDataset()<br>
train_loader = DataLoader(dataset=dataset,<br>
batch_size=batch_size,<br>
shuffle=True,<br>
num_workers=2)<br>
My Training loop:<br>
///<br>
criterion = torch.nn.NLLLoss()<br>
<span class="hashtag-raw">#criterion</span> = torch.nn.CrossEntropyLoss()</p>
<p>for epoch in range(num_epochs):<br>
for batch_idx, (x_train, y_train) in enumerate(train_loader):<br>
x_train, y_train = Variable(x_train), Variable(y_train)<br>
opt.zero_grad()<br>
output = model(x_train)<br>
print(output)<br>
<span class="hashtag-raw">#y_train</span> = y_train.squeeze_()<br>
loss = criterion(output, y_train)<br>
loss.backward()<br>
opt.step()<br>
print(‘Epoch %d: Loss %.5lf’ % (epoch, loss))<br>
///</p>
<p>I tried to squeeze_ the y labels, but it does not make sense and I get same values over iterations…</p>
<p>I am really lost :(( Can you help?</p>

ptrblck: <p><code>nn.NLLLoss</code> expects your model to output a tensor of the shape <code>[batch_size, nb_classes]</code> and a target of <code>[batch_size]</code> containing the class indices for a usual multi-class classification use case.</p>
<p>If you are squeezing <code>y_train</code>, which shape does it have afterwards and what do you mean by</p>
<aside class="quote no-group" data-username="zooraff" data-post="1" data-topic="44977">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/2acd7d/48.png" class="avatar"> zooraff:</div>
<blockquote>
<p>and I get same values over iterations</p>
</blockquote>
</aside>
<p>?</p>

zooraff: <p>Hi ptrblack,</p>
<p>thank you for your quick answer.</p>
<p>I am getting the shape of [batch_size, nb_classes], which is (10,5). The problem is with y_train (10,1).</p>
<p>Pls disregard for now this line:<br>
“and I get same values over iterations” - I assume firstly the training has to start working, because the loss seems to get unreal values, thus the loss seems more less the same over 10 epochs and the results do not get better.</p>
<p>Anyway, when I use the squeezed y_train, I get (batch size 10, 5 classes):<br>
tensor([2, 2, 0, 4, 1, 3, 2, 1, 1, 1])<br>
torch.Size([10])</p>
<p>I tried eventually to adjust the logits after applying sigmoid into same shape by applying:<br>
output = model(x_train)<br>
output = torch.argmax(output, dim=-1)<br>
and I get strange values:<br>
tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1])<br>
torch.Size([10])<br>
Having same shapes finally, I tried to apply BCELoss, bot no success.</p>
<p>But my wish is to get the crossEntropy working at last, I do not know if I have to convert my y_train vector to one-hot encoding somehow for cross-entropy to see it properly?</p>
<p>On the other hand, I have read somewhere, that NLL recognizes 1D vector of labels and matches it with output layer (after applying logSoftmax, 5 classes as ouput), is it true? It does not work for me either…</p>
<p>Pls let me know if something is not clear.</p>
<p>Cheers,<br>
Marek</p>

ptrblck: <aside class="quote no-group" data-username="zooraff" data-post="3" data-topic="44977">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/2acd7d/48.png" class="avatar"> zooraff:</div>
<blockquote>
<p>On the other hand, I have read somewhere, that NLL recognizes 1D vector of labels and matches it with output layer (after applying logSoftmax, 5 classes as ouput), is it true? It does not work for me either…</p>
</blockquote>
</aside>
<p>Yes, that should work as explained:</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="44977">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>nn.NLLLoss</code> expects your model to output a tensor of the shape <code>[batch_size, nb_classes]</code> and a target of <code>[batch_size]</code> containing the class indices for a usual multi-class classification use case.</p>
</blockquote>
</aside>
<p>You should also apply <code>F.log_softmax</code> on your output, but this already seems to be the case.</p>
<p>Why does it not work for you? Do you get any error?<br>
After squeezing the target, the shape looks alright.</p>

zooraff: <p>I have such shape now of input and target as before:</p>
<p>tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
torch.Size([10]) <span class="hashtag-raw">#this</span> is input<br>
tensor([4, 3, 0, 0, 1, 0, 2, 1, 3, 2])<br>
torch.Size([10]) <span class="hashtag-raw">#this</span> is target</p>
<p>I apply log_softmax to my output logits indeed:<br>
///<br>
y_pred = self.fc4(out3)<br>
return F.log_softmax(y_pred)</p>
<p>…and transform input, target as follows in training loop:<br>
input:<br>
output = model(x_train)<br>
output = torch.argmax(output, dim=-1)<br>
target:<br>
y_train = y_train.squeeze_(1)<br>
y_train = torch.transpose(y_train, 0, -1)</p>
<p>what I get, is following error now:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/c5820afea653edb2c0712c59ede09eb6af1157d9.png" data-download-href="https://discuss.pytorch.org/uploads/default/c5820afea653edb2c0712c59ede09eb6af1157d9" title="24"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/c/c5820afea653edb2c0712c59ede09eb6af1157d9_2_690x475.png" alt="24" data-base62-sha1="sbeFZXKmFgvLc8v12kazkVoRiyt" width="690" height="475" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/c/c5820afea653edb2c0712c59ede09eb6af1157d9_2_690x475.png, https://discuss.pytorch.org/uploads/default/optimized/2X/c/c5820afea653edb2c0712c59ede09eb6af1157d9_2_1035x712.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/c/c5820afea653edb2c0712c59ede09eb6af1157d9_2_1380x950.png 2x" data-dominant-color="F5F6F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">24</span><span class="informations">2040×1406 396 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The model output should be <code>[batch_size, nb_classes]</code>, so <code>[10, 5]</code> in your case, with <code>F.log_softmax(x, dim=1)</code> applied on the logits.</p>
<p>You shouldn’t call <code>torch.argmax</code> on the outputs.</p>
<p>PS: This shouldn’t be an error, but you should either use</p>
<ul>
<li>the inplace method of <code>squeeze</code>: <code>y_train.squeeze_(1)</code>
</li>
<li>or the “vanilla” method and assign the result: <code>y_train = y_train.squeeze(1)</code>
</li>
</ul>
<p>(Methods with an underscore work inplace)</p>

zooraff: <p>When I use y_train.unsqueeze_(1), I am still getting same error:<br>
<em>RuntimeError: multi-target not supported at /Users/soumith/code/builder/wheel/pytorch-src/aten/src/THNN/generic/ClassNLLCriterion.c:21</em></p>
<p>But when I try:<br>
y_train = y_train.squeeze_(1)</p>
<p>The algorithm goes on further finally, but I am getting strange loss values, e.g.:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Train Epoch: 3 [43100/45324 (95%)]</th>
<th>Loss: 1.242024</th>
</tr>
</thead>
<tbody>
<tr>
<td>Train Epoch: 3 [43200/45324 (95%)]</td>
<td>Loss: 0.432863</td>
</tr>
<tr>
<td>Train Epoch: 3 [43300/45324 (96%)]</td>
<td>Loss: 0.725757</td>
</tr>
<tr>
<td>Train Epoch: 3 [43400/45324 (96%)]</td>
<td>Loss: 0.514305</td>
</tr>
<tr>
<td>Train Epoch: 3 [43500/45324 (96%)]</td>
<td>Loss: 0.267429</td>
</tr>
<tr>
<td>Train Epoch: 3 [43600/45324 (96%)]</td>
<td>Loss: 0.490571</td>
</tr>
<tr>
<td>Train Epoch: 3 [43700/45324 (96%)]</td>
<td>Loss: 0.444628</td>
</tr>
<tr>
<td>Train Epoch: 3 [43800/45324 (97%)]</td>
<td>Loss: 0.134719</td>
</tr>
<tr>
<td>Train Epoch: 3 [43900/45324 (97%)]</td>
<td>Loss: 0.142233</td>
</tr>
<tr>
<td>Train Epoch: 3 [44000/45324 (97%)]</td>
<td>Loss: 0.333635</td>
</tr>
</tbody>
</table>
</div><p>It’s NLL after aplying log_softmax.</p>
<p>Is it normal that the loss jumps like this from like 0.13 ~ 1.24 in the training loop?<br>
I would expect the model to be greadually improving to reach like ~0.9 accuracy<br>
(of course against test set which I have not done yet)</p>

ptrblck: <p>Sorry, that was a typo and you should use <code>squeeze</code> instead.</p>
<p>It might happen sometimes and it also depends on the batch size you are using. Larger batch sizes yield a smoother loss curve generally.</p>

zooraff: <p>I got the code working finally, thank you!</p>
<p>The only problem is, the loss seems to be still terrible.</p>
<p>I am trying different learning rates =  0.01, 0.05, 0.001, optimizer SGD, Adam or Adagrad, CrossEntropy or NLL loss. Epochs = 5 or 10.</p>
<p>I am not sure if my NN works properly, the loss seems to get very strange values. What could be a realistic loss for such a multi-class classification problem? I am never lower than 0.5 for different sizes of batches (I tried 10, 50, 100, 1000), average is around 0.9:</p>
<p>…</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Train Epoch: 4 [44500/45324 (98%)]</th>
<th>Loss: 1.194766</th>
</tr>
</thead>
<tbody>
<tr>
<td>Train Epoch: 4 [44600/45324 (98%)]</td>
<td>Loss: 0.656055</td>
</tr>
<tr>
<td>Train Epoch: 4 [44700/45324 (99%)]</td>
<td>Loss: 0.755115</td>
</tr>
<tr>
<td>Train Epoch: 4 [44800/45324 (99%)]</td>
<td>Loss: 0.929295</td>
</tr>
<tr>
<td>Train Epoch: 4 [44900/45324 (99%)]</td>
<td>Loss: 0.823785</td>
</tr>
<tr>
<td>Train Epoch: 4 [45000/45324 (99%)]</td>
<td>Loss: 1.219415</td>
</tr>
<tr>
<td>Train Epoch: 4 [45100/45324 (99%)]</td>
<td>Loss: 0.845087</td>
</tr>
<tr>
<td>Train Epoch: 4 [45200/45324 (100%)]</td>
<td>Loss: 0.934597</td>
</tr>
<tr>
<td>Train Epoch: 4 [45300/45324 (100%)]</td>
<td>Loss: 0.878224</td>
</tr>
</tbody>
</table>
</div><p>///</p>
<p>When I submitted the results for our project, I got a very low baseline ( I do not have ground truth labels unfortunately, to evaluate the model on test set.</p>
<p>My evaluation loop is as follows in this case:<br>
///<br>
model.eval()<br>
y = []<br>
pred = []</p>
<p>for i, (x_test) in enumerate(test_loader):<br>
x_test = Variable(x_test)<br>
output = model(x_test)<br>
output = torch.max(output.data,1)[1]<br>
y = output.numpy()[0]<br>
pred.append(y)<br>
///</p>
<p>Would you mind having a look and shed some light to that problem?</p>
<p>Thanks again!</p>

ptrblck: <p>Are you dealing with an imbalanced dataset?<br>
If so, you could add a <code>weight</code> argument to your loss function or use a <code>WeightedRandomSampler</code> so that your model does not overfit on a specific class.</p>
<p>Also, try to overfit your model on a very small dataset (e.g. just 10 samples) and make sure the loss decreases towards zero.<br>
If that’s not the case, you might have a bug in your code or the model architecture/hyperparameters are not suitable for this use case.</p>