How do I handle class imbalance for text data when using pretrained models like BERT?
Author: nikhil6041
Pull request: no

I have a skewed dataset consisting of samples of the form:

```
Category 1 10000
Category 2  2000
Category 3   400
Category 4   300 
Category 5   100

```
The dataset consists of text with data labeled into one of the five categories. I am trying to use the pretrained models like BERT for the classification task but the model fails to identify the categories 3-5 .I have tried to apply class weights in the loss criterion however it doesn't help much although it gives better performance as compared to simple fine tuning of the pretrained models. I have came to know about SMOTE and other methods in order to handle the class imbalance issues . But since most of the transformer models expect the inputs as text which are later tokenized by their respective tokenizers I am not able to do any kind of oversampling . If there is a workaround for this thing I would be interested to know about it.

Comments:
Hi!

You could try replacing the CrossEntropy loss with [this Dice Loss](https://github.com/fursovia/self-adj-dice), which may help you with the imbalance issues. In the paper linked in the repo they explain the design process. I have tried it with mixed results, although for my (skewed) dataset, weighting the CrossEntropy loss with the inverse frequency of each category has worked best.

Let me know if it works for you 👍🏻 

As a last resort, you could try undersampling category 1 to match the second, and maybe combine this with a weighted loss as well.

Thanks @viantirreau  for your suggestions I actually tried to use the dice loss as well as class weights with crossentropy loss and the results I got from the crossentropyloss was actually better than what I am getting with the dice loss , however both of them fails to detect the categories 3-5 . I will try to do undersampling  as the last resort however speaking of the class weights I have used  the sklearn's compute_class_weight for getting my class_weights as follows:
```
from sklearn.utils.class_weight import compute_class_weight

#compute the class weights
class_wts = compute_class_weight('balanced', np.unique(train_labels), train_labels)

```
Can you suggest any other workaround other than this strategy , I have came to know that neural networks tends to ignore class weights through an answer on one of the stackexchange sites . 


Hi!

I think it shouldn't make any difference, using my method returns exactly the same as Sklearn's `compute_class_weight`, but normalized so as to add up to 1. 

Using these class counts,

> Category 1 10000
> Category 2  2000
> Category 3   400
> Category 4   300 
> Category 5   100

 I get the following weights for the respective categories `array([0.00608519, 0.03042596, 0.15212982, 0.20283976, 0.60851927])`. 
Another nice (and extreme) experiment you could try is to over-emphasize the weights for the underrepresented classes, for example using something like `array([0.001, 0.001, 0.001, 0.001, 0.996])`, just as a sanity check to confirm the optimizer learns something about category 5.
I would also start testing the model's predictions on the training data first (it should overfit), and only then try to measure its generalization abilities on a held out development set. Maybe your gradients are not backpropagating to the first layers, your learning rate is way too big or you need some warmup steps.

Let me know if any of this works :)
Good luck!

Hi , @viantirreau Thanks for your suggestions. I did try it by reducing the class weights for majority classes and emphasizing the weights for minority class say category 5 , I found out that my neural network is still not able to learn anything about those classes , I have tried it with learning rates 1e-5,2e-5,5e-5 and warmup step of 1000 however no improvement is still being made on it . Any optimization strategies for the hyperparameters you can suggest?

You're welcome! 
Mmh interesting, what Transformers model are you using? Also, from what pretrained checkpoint are you initializing it? Are you sure there are no warnings like 'missing parameters, initializing from scratch'?

I have faced some vanishing gradient problems in the past that manifest as an unexplainable "preference" for a class, so I'd make sure that your gradients are alright. I find [this](https://gist.github.com/viantirreau/ec591a428a5c0112bd8fa84f70968574) code snippet pretty useful to diagnose the gradient flow by plotting its values across each layer. If you use Weights&Biases as a logging tool, you can `watch` the model and create even nicer plots in their dashboard. A warmup strategy was crucial in my experience to eliminate the gradient problems.

Also, if you are manually adjusting the attention masks or some of the model inputs, make sure to not pass ignore_index in some/all of the inputs. Some prints will help in making sure that the model inputs are as expected.
Another idea I'd test is to completely eliminate your categories 1 and 2 from the training examples, and see if the same phenomenon happens to the most common class by then (should be category 3). Try this alone and see if including the inverse frequency weights in the loss helps in any way.

Good luck!
Good luck!

Hi , @viantirreau  sorry for the delay in response I haven't received any warnings as such . I am using the bert transformer with bert-base-multilingual-cased as the checkpoint , I was trying to first build a custom model from the final output layer of the BERT model in order to accomodate the class imbalance issue . I haven't tried the weights and biases yet will surely check it out. I will try your other suggestions and will let you know about it. Thanks for your suggestions.


Hi , @viantirreau  sorry for the delay in response .I finally figured out the reason behind the performance degradation it was because I was freezing the base layers and only fine tuning one extra layer which I added to the base model. Since, the model had the data imbalance  issue already into it ,it was being biased towards the majority samples . It however performed much better when I unfreezed the base layers , however on the cost of additional gpu training time.

Hi, @nikhil6041. I'm glad you figured it out. Thanks for reaching back with your experience and solution! 🙌🏻 

I am actually having the same problem you experienced. I am building a multi-label multi-class classification Bert/distilbert model and encountered the same issue  with my 20 classes. Of course the data is imbalanced, and like you I thought I had locked down the base layers but I realized I hadn't and that model performed slight better with the imbalanced data than the locked down model. I could not figure out why other than knowing imbalanced data is a big deal. Unfortunately, the data set I have is extremely small so that is also probably playing a big role. @viantirreau and @nikhil6041,  one method I have seen used is a weighted cost function like adacost.  Has anyone had any success implementing this with Distilbert? I can provide more details or open a new ticket but this seemed very closely related.

Hi @johnrodriguez190380  have you tried KL divergence loss function ? Try to use it once. Also there have been certain instances where the usage of weighted cost function doesn't help much. I don't remember the paper which pointed out this thing but I read it somewhere in a stackoverflow answer. If in case the dataset you are using is really small you can try some data augmentation techniques , you can also use this  [](https://github.com/makcedward/nlpaug )repo it maybe helpful for u i guess. Let me know if anyone of this serves ur usecase.

Hi nikhil, how are you? maybe can you share your code/colab/repo to see how you solve the issue? 

Hi @nikhil6041 could you please share the script you wrote for changing the loss? I really appreciate it if you can share it with us!

Hey @sasu08  and @un-lock-me sorry for the late response I used the sadice loss for this one however it didnt solve my problem fully however there are certain other ways you can possibly try for this thing try to use text augmentations (there are various ways for it like using synonyms ,back translation to name a few , there is one library named nlpaug which might come in handy for both of you , have a look at it. For the sadice loss part you can have a look at my repo [](https://github.com/nikhil6041/OLI-and-Meme-Classification) , here is the link to the nlpaug library [](https://github.com/makcedward/nlpaug). Hope it helps!!

For the Sadie loss I could not find it in the repo could you please share the link here?

@un-lock-me  Here it is [sadice loss] https://github.com/fursovia/self-adj-dice   

Hi @nikhil6041 can you please share your notebook? 

Hi @pratikchhapolika you can find all my notebooks in this [repo](https://github.com/nikhil6041/OLI-and-Meme-Classification)

@nikhil6041 thanks for the helpful repo. I was running the code on Google Colab and I used the provided dataset.
However, I am getting this error in the Training Loop. 
TypeError: forward() got an unexpected keyword argument 'token_type_ids' 
What can be the issue? Thanks