Can we use cross entropy loss for binary classification

username: donekpr1
cooked: <p>Hi All,<br>
I want to confirm if we  can use cross entropy for binary class with single label clasisfication .I am using Roberta model and the data set is highly imbalanced(with more 1s and less 0 s).I tried to use weighted random sampler but my F1 score is very low (0.29) for 0 and (0.75) for 1s .I felt this is not working .<br>
I tried to use bcewithlogitloss along with pos_weight but loss is showing up as NA.</p>
<p>So,I thought to use cross entropy loss with class weight computed using sklearn computer class weight.Hope I am doing it right?</p>
<p>Appreciate if you can confirm these two things as asked<br>
1.Can I use cross entropy loss for binary classification in the above case?<br>
2.If so,can I use compute class weight of sklearn for calculating class weights?</p>
<p>Request to assist in this regard.</p>

username: ptrblck
cooked: <ol>
<li>Yes, you can use <code>nn.CrossEntropyLoss</code> for a binary classification use case and would treat it as a 2-class multi-class classification use case. In this case your model should output 2 logits instead of 1 as would be the case for a binary classification using <code>nn.BCEWithLogitsLoss</code>.</li>
<li>I don’t know which scikit-learn method you want to use, but guess these class weights might be a starter to check if your training would improve.</li>
</ol>

username: donekpr1
cooked: <p>Thanks for your response.<br>
I have used bcewithlogitloss() and pos weight calculated as below<br>
y_train=torch.Tensor(y_train)</p>
<p>num_positives = torch.sum(y_train, dim=0)</p>
<p>num_negatives = len(train_dataset) - num_positives</p>
<p>pos_weight = num_negatives / num_positives</p>
<p>pos_weight</p>
<p>For me no of 0s are very less than number of 1s .When I used this method or even with weighted random sampler for which I computed class weights as below<br>
y_train_indices = train_dataset.indices</p>
<p>y_train = [target[i] for i in y_train_indices]</p>
<p>class_sample_count = np.array(</p>
<p>[len(np.where(y_train == t)[0]) for t in np.unique(y_train)])</p>
<p>print(class_sample_count)</p>
<p>weight = 1. / class_sample_count</p>
<p>samples_weight = np.array([weight[t] for t in y_train])</p>
<p>samples_weight = torch.from_numpy(samples_weight)</p>
<p>print(samples_weight)</p>
<p>sampler = torch.utils.data.sampler.WeightedRandomSampler(samples_weight.type(‘torch.DoubleTensor’), len(samples_weight))</p>
<p>There is no improvement in the number of 0 s of F1 score and is resulted as below ,<br>
precision    recall  f1-score   support</p>
<pre><code>       0       0.23      0.75      0.35         4
       1       0.93      0.58      0.72        24

accuracy                           0.61        28
</code></pre>
<p>macro avg       0.58      0.67      0.54        28<br>
weighted avg       0.83      0.61      0.67        28</p>
<p>I ran for 3 epochs and both the methods were tried by substituting them in the loop of train data loader.<br>
The validation loss is not decreasing as you see above for both the methods .<br>
Hence I wanted to try now using cross entropy loss for which I am using the below ,<br>
np.unique(train_dataset.indices)</p>
<p>y_train = [target[i] for i in y_train_indices]</p>
<p>np.unique(y_train)</p>
<p>class_wts = compute_class_weight(“balanced”,classes= np.unique(y_train),y=y_train)</p>
<p>class_wts<br>
and substituting them in the cross entropy loss.</p>
<p>I have read many posts of this forum regarding the first two methods but in my case its overfitting .I am not sure as I have binary class with single label,the pos weight I have taken as a single value which is tensor 6.5 resulted from the above .<br>
Kindly correct me if there is any step that needs to be corrected for all the 3 methods.<br>
Thanks.</p>