Cross entropy loss for 100+ targets [Expected floating point type/ "host_softmax" not implemented for 'Long']

phate: <p>Hi,</p>
<p>Currently, I’m facing the issue with cross entropy loss. My target variable is one-hot encoding values such as [0,1,0,…,0] then I would have <strong>RuntimeError: Expected floating point type for target with class probabilities, got Long</strong>.</p>
<p>On the other hand, if i were to not perform one-hot encoding and input my target variable as is, then i face the issue of <strong>RuntimeError: “host_softmax” not implemented for ‘Long’</strong></p>
<pre><code class="lang-auto"># MODEL DEFINED 
class BertClass(torch.nn.Module):
    def __init__(self):
        super(BertClass, self).__init__()
        self.model = BertModel.from_pretrained('bert-base-uncased')
        self.l1 = torch.nn.Linear(768, 512) # linear layer 1
        self.l2 = torch.nn.Linear(512, len(lb.classes_)) # num. of classes
        self.dropout = torch.nn.Dropout(0.2)

    def forward(self, input_ids, attention_mask, token_type_ids):
        output = self.model(input_ids=input_ids, 
                              attention_mask=attention_mask, 
                              token_type_ids=token_type_ids)
        hidden_state = output[0]
        output = hidden_state[:, 0]
        output = self.l1(torch.nn.ReLU()(output))
        output = self.dropout(output)
        output = self.l2(output)
        return output

# INSTANTIATE MODEL
model = BertClass()
model = model.to(device)

# IMBALANCED DATA
class_weights = class_weight.compute_class_weight(class_weight = 'balanced', 
                                  classes = np.unique(available_df['controlId']), 
                                  y = available_df['controlId'])
class_weights = torch.tensor(class_weights)

loss_function = torch.nn.CrossEntropyLoss()(weight=class_weights)
optimizer = torch.optim.Adam(params =  model.parameters(), lr=LEARNING_RATE)

# TRAIN THE MODEL
def train(epoch):
    tr_loss = 0
    n_correct = 0
    nb_tr_steps = 0
    nb_tr_examples = 0
    model.train()
    for _,data in enumerate(train_loader, 0):
        print(data)
        ids = data['ids'].to(device)
        mask = data['mask'].to(device)
        token_type_ids = data['token_type_ids'].to(device)
        targets = data['targets'].to(device)
        print(targets) 
        
        outputs = model(ids, mask, token_type_ids).to(device)
        outputs = outputs.long()
        print(outputs.size())# torch.Size([8, 325])
        print(targets.size()) # with one hot encoding torch.Size([8]). Without one-hot encoding torch.Size([8, 325])
        
        loss = loss_function(outputs, targets.long())
        tr_loss += loss.item()
        big_val, big_idx = torch.max(outputs.data, dim=1)
        n_correct += calculate_accuracy(big_idx, targets)

        nb_tr_steps += 1
        nb_tr_examples+=targets.size(0)
        
        if _%1000==0:
            loss_step = tr_loss/nb_tr_steps
            accu_step = (n_correct*100)/nb_tr_examples 
            print(f"Training Loss per 5000 steps: {loss_step}")
            print(f"Training Accuracy per 5000 steps: {accu_step}")

        optimizer.zero_grad()
        loss.backward()
        # # When using GPU
        optimizer.step()

    print(f'The Total Accuracy for Epoch {epoch}: {(n_correct*100)/nb_tr_examples}')
    epoch_loss = tr_loss/nb_tr_steps
    epoch_accu = (n_correct*100)/nb_tr_examples
    print(f"Training Loss Epoch: {epoch_loss}")
    print(f"Training Accuracy Epoch: {epoch_accu}")

    return epoch_accu 

EPOCHS = 1
for epoch in range(EPOCHS):
    train(epoch)


</code></pre>

ptrblck: <aside class="quote no-group" data-username="phate" data-post="1" data-topic="179998">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/e0b2c6/48.png" class="avatar"> phate:</div>
<blockquote>
<p>My target variable is one-hot encoding values such as [0,1,0,…,0] then I would have <strong>RuntimeError: Expected floating point type for target with class probabilities, got Long</strong>.</p>
</blockquote>
</aside>
<p>That’s expected as mentioned in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss">docs</a>.<br>
If you want to pass probabilities (in this case as a one-hot encoded tensor) the target has to be a floating point tensor.</p>
<aside class="quote no-group" data-username="phate" data-post="1" data-topic="179998">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/e0b2c6/48.png" class="avatar"> phate:</div>
<blockquote>
<p>On the other hand, if i were to not perform one-hot encoding and input my target variable as is, then i face the issue of <strong>RuntimeError: “host_softmax” not implemented for ‘Long’</strong></p>
</blockquote>
</aside>
<p>I’m unsure why you want to apply a <code>softmax</code> on the target, which should contain class indices in case it’s a <code>LongTensor</code>, so could you explain your use case a bit more, please?</p>

phate: <p>Thanks for your prompt response. I’m still new to this so thank you for being patient.<br>
I’ve been reading how for Cross entropy loss, we shouldn’t convert to one-hot coding. In this case, would you recommend leave the 100+ labels as is and perform prediction? In addition, in the case of getting the indices, i assume we would select the index highest probabilities for each batch after model(ids, mask, token_type_ids) which makes the output length to be 8?</p>

phate: <p>also, the reason why I’m having softmax via cross entropy here is because i want to get the probabilities for the classes then I would perform the loss calculation between my one-hot encoding ground truth the probability output from my prediction. But please correct me if my understanding is incorrect</p>

ptrblck: <aside class="quote no-group" data-username="phate" data-post="3" data-topic="179998">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/e0b2c6/48.png" class="avatar"> phate:</div>
<blockquote>
<p>In addition, in the case of getting the indices, i assume we would select the index highest probabilities for each batch after model(ids, mask, token_type_ids) which makes the output length to be 8?</p>
</blockquote>
</aside>
<p>In case your target is one-hot encoded you can create the class indices via <code>torch.argmax</code>, yes.<br>
Don’t apply it on the model output, but the targets.</p>
<aside class="quote no-group" data-username="phate" data-post="4" data-topic="179998">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/e0b2c6/48.png" class="avatar"> phate:</div>
<blockquote>
<p>also, the reason why I’m having softmax via cross entropy here is because i want to get the probabilities for the classes then I would perform the loss calculation between my one-hot encoding ground truth the probability output from my prediction.</p>
</blockquote>
</aside>
<ol>
<li><code>nn.CrossEntropyLoss</code> expects raw logits as the model output, so don’t use a softmax on it and pass the raw logits to the criterion.</li>
<li>The error message indicates you are applying softmax on the target, which is why I asked for more details.</li>
</ol>

phate: <p>Thank you for all the insights. I’m not clear which line of my codes indicates that I apply softmax on my target. Can you help me point it out?</p>

ptrblck: <aside class="quote no-group" data-username="phate" data-post="1" data-topic="179998">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/e0b2c6/48.png" class="avatar"> phate:</div>
<blockquote>
<p>RuntimeError: “host_softmax” not implemented for ‘Long’</p>
</blockquote>
</aside>
<p>Indicates that you are are applying a <code>softmax</code> operation on a <code>LongTensor</code>, which should be the target unless you are explicitly casting the floating point outputs of your model to <code>.long()</code>.</p>