BCEWithLogitsLoss() giving negative loss

TheOraware: <p>Hi ,</p>
<p>I am training NN using pytorch 1.7.0 , when i use CrossEntopyLoss() loss function then i dont have any negative loss in any epochs, since this competition evaluation metrics is multi-class logarithmic loss which i believe BCEWithLogitsLoss() in pytorch serve this logarithmic loss for multi class (correct me if i am wrong).</p>
<p>My question is why negative loss is coming when using BCEWithLogitsLoss()? How can i prevent it, i dont wanna use CrossEntopyLoss() , Please see below code , for clarity i am showing “y” actual target and “output” prediction of model in first epoch only</p>
<pre><code class="lang-auto">def get_optimizer(model, lr):
    optim = torch_optim.Adam(model.parameters(), lr=lr, weight_decay=0.05) 
    return optim

batch_size = 2000 

def train_loop(model, epochs, lr):
    total    = 0
    sum_loss = 0
    output   = 0
    criterion = nn.BCEWithLogitsLoss()
    optim= get_optimizer(model)
    for epoch in range(epochs):
        for cat, y in train_dl:
            model.train()
            batch = y.shape[0]
            output = model(cat)
            if (epoch) ==1:
                print(f'y is {y.float}')
                print(f'y is {output[:,0]}')
            loss = criterion(output[:,0],y.float())
            optim.zero_grad()
            loss.backward()
            optim.step()
            total += batch
            sum_loss += batch*(loss.item())
        valid_ds = ClassifierDataset(X_val,y_val , features)
        batch_size = X_train.shape[0]
        valid_dl=DataLoader(valid_ds,batch_size=batch_size,shuffle=False)
        valid_dl=DeviceDataLoader(valid_dl, device)
        for cat, y in valid_dl:
            model.eval()
            output = model(cat)
            valid_loss = criterion(output[:,0],y.float())
        print(f'epoch:{epoch+1},training loss:{loss},valid loss:{valid_loss} ')

</code></pre>
<pre><code class="lang-auto">train_dl = DataLoader(train_ds, batch_size=batch_size,shuffle=True)
train_dl = DeviceDataLoader(train_dl, device)
model = multiNet(embedding_sizes)
to_device(model, device)
model.apply(init_weights) 
train_loop(model, epochs=120, lr=0.001)


epoch : 1,training loss : -127.1643,valid loss : -82.094856 
y:tensor([7., 3., 1.,  ..., 8., 7., 1.], device='cuda:0')
output:tensor([ 0.945,0.189,-1.194,...,-1.03,0.80,-1.05],device='cuda:0',grad_fn=&lt;SelectBackward&gt;)
epoch : 2,training loss : -298.340728,valid loss : -293.701477 
epoch : 3,training loss : -529.159423,valid loss : -535.595520 
epoch : 4,training loss : -882.299377,valid loss : -906.745788 
</code></pre>

Flock1: <p>Try with this:<br>
<code>torch.nn.functional.binary_cross_entropy</code></p>

ptrblck: <p><code>nn.BCEWithLogitsLoss</code> expects the targets to be in the range <code>[0, 1]</code> as described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss">docs</a>. Since your targets contain values outside of this range, the loss could be negative.</p>

TheOraware: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks much appreciated , it means i have target between 0 to 8. To get these target between range 1 to 0 , i need to one hot encode them as follows?</p>
<pre><code class="lang-auto">
Class_0    Class_1    Class_2    Class_3    Class_4    Class_5    Class_6    Class_7    Class_8    Class_9
-------    -------    -------    -------    -------    -------    -------    -------    -------    -------
      0          0          1          0          0          0          0          0          0          0
</code></pre>

ptrblck: <p>Based on the output it seems you are working on a multi-class classification (i.e. each sample has one target only), so you could directly use <code>nn.CrossEntropyLoss</code>.<br>
On the other hand, if some samples have zero, one, or more active targets you would be working on a multi-label classification, could use <code>nn.BCEWithLogitsLoss</code> and would then multi-hot encode the target.</p>

TheOraware: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks much appreciated</p>

TheOraware: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> why do we use softmax function during prediction of a model which is built on using nn.CrossEntropyLoss() loss function ? As i read somewhere CrossEntropyLoss has already softmax function</p>

ptrblck: <p>You shouldn’t use a <code>softmax</code> on the model outputs when you want to calculate the loss using <code>nn.CrossEntropyLoss</code>, since (as you’ve already said) <code>nn.CrossEntropyLoss</code> applies <code>F.log_softmax</code> and <code>nn.NLLLoss</code> internally, so pass the raw logits to this loss function instead.</p>
<p>On the other hand, you can apply a <code>softmax</code> on the model outputs (logits), if you want to “visualize” the probabilities or use them in any other way besides the input to <code>nn.CrossEntropyLoss</code>.</p>