How to handle not measured classes

Hannes_Peter: <p>Hello. Thanks for reading. I have a dataset with different chemical assays. Each assay can be pos=1, negative = 0 and not measured =-1I have now implemented a fnn model with ReLU activation function. I tried mse loss and now I tried crossentropy loss. With mse my loss is super low really fast, but the predictions return nonsense. No value is near 0.9 or even 1. Some if the assays for sure are positive for example. With crossentropy I now have nehative loss. I now thought it is time to ask for help. All input and tipps would be helpfull to me.</p>
<p>Thank you.</p>

ptrblck: <p>Based on your description it seems you are working on a multi-class classification use case with 3 classes. In this case <code>nn.CrossEntropyLoss</code> would be the right loss function.<br>
This loss function expects a model output in the shape <code>[batch_size, nb_classes=3]</code> and a target tensor in the shape <code>[batch_size]</code> containing class indices in the range <code>[0, nb_classes-1]</code>, so <code>[0, 1, 2]</code> in your case.<br>
You would have to map your current labels from <code>[-1, 0, 1]</code> to <code>[0, 1, 2]</code> which might also explain the negative loss you are seeing.</p>

J_Johnson: <p>What is the practical difference between “not measured” and “negative”?</p>
<p>Additionally, if your classes on the train dataset are not evenly distributed, are you making use of the pos_weight argument in nn.CrossEntropyLoss?</p>

Hannes_Peter: <p>Thank you for your answers. I tried to use a mask now to mask out all “not measured” classes. I have 11 classes where they can be either 0 or 1 now. (Sorry for not explaining very well)<br>
I use 3 Linear layers, ReLU between and I am wondering now should I switch them all for Softmax or only the last? Before the last Liner or after? I am not sure how to make use of the pos_weight in CreL, since i have 11 classes and don’t know which one of them is 0 or 1. I used a Boolean mask for now before calculating the loss.</p>
<p>My new problem is now. that the loss is only fluctuating, not decreasing. It stays around 1200 loss which i have never seen before. I can´t solve my problem. Maybe I should just stop.</p>
<p>Thank you anyways for trying to help me. I am still open to inputs if someone has time for me.<br>
example classes:<br>
assay 1 assay 2 assay 3<br>
compound1: 0          -1               1</p>
<p>-1 now gets filtered via Boolean mask.</p>
<p>Thanks</p>

J_Johnson: <p>ReLU as an activation on intermediate layers serves a different purpose, giving your model the ability to be decisive. I.e. if you think of the model as a game of 20 questions: “Is it red?” An activation layer turns a spectrum of red/green into a deterministic yes or no. You want that kind of objectivity in a model.</p>

ptrblck: <aside class="quote no-group quote-modified" data-username="J_Johnson" data-post="5" data-topic="176594">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/j_johnson/48/55494_2.png" class="avatar"> J_Johnson:</div>
<blockquote>
<p>In my testing on overfitting models, softmax tends to work best before CEL, whereas raw model outputs(i.e. no softmax) can be hit and miss, depending on how the parameters are initialized. It seems that softmax puts the values in a range where CEL works consistently best with gradient descent.</p>
</blockquote>
</aside>
<p>This should not be the case since <code>nn.CrossEntropyLoss</code> expects raw logits. Internally it will apply <code>F.log_softmax</code> and <code>nn.NLLLoss</code>, so applying another <code>F.softmax</code> would use:<br>
<code>loss = F.nll_loss(F.log_softmax(F.softmax(output, 1), 1))</code> and would thus decrease the magnitude of the gradients.</p>
<aside class="quote no-group" data-username="Hannes_Peter" data-post="4" data-topic="176594">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hannes_peter/48/59096_2.png" class="avatar"> Hannes_Peter:</div>
<blockquote>
<p>I am not sure how to make use of the pos_weight in CreL</p>
</blockquote>
</aside>
<p><code>nn.CrossEntropyLoss</code> does not provide the <code>pos_weight</code> argument, but <code>nn.BCEWithLogitsLoss</code> does, which is used for a multi-label or binary classification.</p>
<aside class="quote no-group" data-username="Hannes_Peter" data-post="4" data-topic="176594">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hannes_peter/48/59096_2.png" class="avatar"> Hannes_Peter:</div>
<blockquote>
<p>since i have 11 classes and don’t know which one of them is 0 or 1.</p>
</blockquote>
</aside>
<p>This sounds indeed like a multi-label classification use case where each sample can belong to zero, one, or multiple classes. If so, you should use <code>nn.BCEWithLogitsLoss</code> instead and pass the raw logits to the loss function.</p>

J_Johnson: <aside class="quote no-group" data-username="Hannes_Peter" data-post="4" data-topic="176594">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hannes_peter/48/59096_2.png" class="avatar"> Hannes_Peter:</div>
<blockquote>
<p>I am not sure how to make use of the pos_weight in CreL, since i have 11 classes and don’t know which one of them is 0 or 1.</p>
</blockquote>
</aside>
<p>Sorry, the argument should be <code>weight</code> for CEL. See here:</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html</a></p>
<p>The weight should be based on your targets/labels. For example, let’s suppose you have only half the training samples for class 0 as other classes, you’d set that value to 2 in the weight tensor, while the others would be 1. The weight is just a tensor of multipliers to get the classes equal. It should be the same length as your number of classes. It’s only necessary if your training samples have a significant class imbalance.</p>
<p>One more possible strategy you can try is initializing the biases in your layers to 1s. For example, you could add something like this in your init after making the layers:</p>
<pre><code class="lang-auto">...
self.layer1  = nn.Linear(3, 64)
self.layer1.bias = nn.Parameter(torch.ones_like(self.layer1.bias))
...
</code></pre>
<aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="176594">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>This should not be the case since <code>nn.CrossEntropyLoss</code> expects raw logits. Internally it will apply <code>F.log_softmax</code> and <code>nn.NLLLoss</code>, so applying another <code>F.softmax</code> would use:<br>
<code>loss = F.nll_loss(F.log_softmax(F.softmax(output, 1), 1))</code> and would thus decrease the magnitude of the gradients.</p>
</blockquote>
</aside>
<p>Correct. But in another thread, adding a softmax activation on the outputs before CEL provided consistently better inference on randomly initialized weights, whereas without would sometimes overfit to the targets and sometimes not. I have admittedly not dug in to the reasons why that was the case, but could be worth trying if other suggestions aren’t working. Here is that thread(with a reproducible code snippet):</p>
<aside class="quote quote-modified" data-post="5" data-topic="175680">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/j_johnson/48/55494_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/very-simple-categorisation-problem/175680/5">Very simple categorisation problem</a> 
  </div>
  <blockquote>
    If, for some reason, you’d like to frame the problem to be useable for CrossEntropyLoss, here is one way you might go about it: 
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np


outputExcepted = torch.tensor([[42, 4300, 55000]])
data = torch.tensor([[0, 0, 0]], dtype=torch.float32)

class FCNet(nn.Module):
    def __init__(self):
        super(FCNet, self).__init__()
        self.layer1 = nn.Linear(3, 55001*3)

    def forward(self, x):
        x = torch.relu(s…
  </blockquote>
</aside>


J_Johnson: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="176594">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>This should not be the case since <code>nn.CrossEntropyLoss</code> expects raw logits. Internally it will apply <code>F.log_softmax</code> and <code>nn.NLLLoss</code>, so applying another <code>F.softmax</code> would use:<br>
<code>loss = F.nll_loss(F.log_softmax(F.softmax(output, 1), 1))</code> and would thus decrease the magnitude of the gradients.</p>
</blockquote>
</aside>
<p>After retesting the code snippet on that other thread, it seems softmax doesn’t work whatsoever. It must have been sigmoid that I used with good results in that example. I am currently getting decent results based on using sigmoid for that example. I’ve updated that thread accordingly.</p>

J_Johnson: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="176594">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>This should not be the case since <code>nn.CrossEntropyLoss</code> expects raw logits. Internally it will apply <code>F.log_softmax</code> and <code>nn.NLLLoss</code>, so applying another <code>F.softmax</code> would use:<br>
<code>loss = F.nll_loss(F.log_softmax(F.softmax(output, 1), 1))</code> and would thus decrease the magnitude of the gradients.</p>
</blockquote>
</aside>
<p>On further testing, setting the activation to sigmoid also was hit and miss, although it seemed slightly better than no final activation for that example. Setting the biases to 1s had very good results every time. I will remove the comments about adding a sigmoid activation function.</p>