Which Loss function for One Hot Encoded labels

username: jda
cooked: <p>I am trying to build a feed forward network classifier that outputs 1 of 5 classes. Before I was using using Cross entropy loss function with label encoding. However, I read that label encoding might not be a good idea since the model might assign a hierarchal ordering to the labels. So I am thinking about changing to One Hot Encoded labels. I’ve also read that Cross Entropy Loss is not ideal for one hot encodings. What other loss functions can I look into in this case? Also since a new column will be created for each class, how do I “collect” the labels and “feed” them to the proposed loss function?</p>
<p>For example, right now I have:</p>
<pre><code>criterion = nn.CrossEntropyLoss()
.
.
.
loss = criterion(ypred_var, labels)</code></pre>

username: ptrblck
cooked: <p><code>nn.CrossEntropyLoss</code> should be a good fit in your case.<br>
You are not really encoding the labels in a hierarchical order as this would be the case using <code>nn.MSELoss</code>.<br>
Instead you provide the class indices, which will be used to get the current class probability. So there is mathematically no difference between this approach and using one-hot encoded tensors.</p>
<p>That being said, <code>nn.CrossEntropyLoss</code> expects class indices and does not take one-hot encoded tensors as target labels.<br>
If you really need to use it for some other reasons, you would probably use <code>.scatter_</code> to create your one-hot encoded targets.</p>

username: jda
cooked: <p>Ok so to be clear, if the five classes have labels 0 - 4 after label encoding then this is the class index and it is compatible with nn.CrossEntropyLoss that I am currently using?</p>

username: ptrblck
cooked: <p>Yes, that’s correct! You should make sure the target is stored as a <code>torch.LongTensor</code>.</p>

username: shellyondiet
cooked: <p>hey, what if i really need to use one-hot encoded targets, should i define my own nn.CrossEntropyLoss ? since it is expecting label indices. Can you elaborate on what using .scatter_ means?</p>

username: ptrblck
cooked: <p>Yes, you could write your custom loss function, which could accept one-hot encoded targets.<br>
The <code>scatter_</code> method can be used to create the targets or alternatively use <code>F.one_hot</code>:</p>
<pre><code class="lang-python">nb_classes = 3
target = torch.randint(0, nb_classes, (10,))
one_hot_scatter = torch.zeros(10, nb_classes).scatter_(
    dim=1, index=target.unsqueeze(1), src=torch.ones(10, nb_classes))

one_hot = F.one_hot(target, num_classes=nb_classes)
</code></pre>

username: shellyondiet
cooked: <p>I have tried building my own loss function for one hot encoding labels as below :</p>
<pre><code class="lang-auto">log_prob = torch.nn.functional.log_softmax(logits_q, dim=1)
loss= -torch.sum(log_prob)*y_qry[i]
</code></pre>
<p>but it seems to have error as</p>
<pre><code class="lang-auto">RuntimeError: grad can be implicitly created only for scalar outputs
</code></pre>
<p>there is usage of</p>
<pre><code class="lang-auto">grad = torch.autograd.grad(loss, params)
</code></pre>
<p>what may be wrong with the code?</p>

username: ptrblck
cooked: <p>The <code>loss.backward()</code> operation would implicitly create a gradient as <code>torch.ones(1)</code>, if the <code>loss</code> is a scalar value.<br>
Otherwise, if the <code>loss</code> is a tensor with multiple values, you would either have to provide the gradient explicitly or reduce the <code>loss</code> before, e.g. via <code>loss.mean().backward()</code>.</p>