Add class weight support for classification
Author: francisco-perez-sorrosal
Pull request: no

# 🚀 Feature request

Add support for the class re-weighting parameters provided in the current loss functions of PyTorch and TF.

## Motivation

In supervised ML classification problems, specially multiclass/multilabel, the classes in the datasets used to train a model may have very different distribution of discrete values, which is usually called class imbalance. The class imbalance may cause biases in the training of the models. DL frameworks like PyTorch or TF, allow in their loss functions (e.g. `CrossEntropyLoss`, `BCEWithLogitsLoss`...) certain parameters to adjust the possible imbalance of data in the classes. Currently, the HF framework does not allow to take advantage of these parameters, so it would be nice to have support of a class re-weighting mechanism at the HF level to improve the possible biases in the models when doing fine-tunings.

As I did in the past with the multi-label support, I’ve had to develop my own workarounds and specific headers for adding the class re-weight support to fine-tune my models. The class weight support basically requires a configuration parameter (e.g. `class_weights`) and some logic in the classification headers to basically:

1) Add the class weights only when training

```python
…
class_weights = None  # Class weights
if self.training and self.class_weights is not None:
    class_weights = self.class_weights.to(self.device)
…
```

…and…

2) Pass the weights in the loss function

```python
…
elif self.config.problem_type == "single_label_classification":
    loss_fct = CrossEntropyLoss(weight=class_weights)
    loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
elif self.config.problem_type == "multi_label_classification":
    loss_fct = BCEWithLogitsLoss(pos_weight=class_weights)
    loss = loss_fct(logits, labels)
…
```

In order to use it, it would only be required to pass in the model configuration the `class_weights` parameter, which is basically a list of floats (one for each class).

In this PR (#15449) I provide a tentative implementation (based on my workarounds) with some basic tests, adding the required scaffolding for the common configuration class and for the BertForSequenceClassification header.

Please let me know if you would be interested in this feature! I can continue working and refining the PR if you think this feature is important. In case you think it’s worth it and the approach is correct, I’ll probably need some advice on how to proceed next, as this feature -as it was the multilabel support- is header (classification mainly) and model dependent, so many model files would need to be modified potentially. Thanks in advance!

Comments:
Hey @francisco-perez-sorrosal! We encourage defining your own loss outside of the model for this use-case.

See https://github.com/huggingface/transformers/issues/9625#issuecomment-762167788 and https://github.com/huggingface/transformers/issues/7024#issuecomment-689625449 for more context.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hey @LysandreJik, thanks for your pointers! (and sorry for the late response) Yes, I already saw that but, the solution implying the override in `compute_loss()` in the Trainer implies that, if you are using the `AutoModel*` classes (e.g. AutoModelForSequenceClassification) the model's `forward()` method and the logic for deciding the problem_type (using `self.config.problem_type` and which includes the call to the loss function in most model heads) will be executed twice:

1. in the `forward()` method of each model head (e.g. `BertForSequenceClassification`), because it's predefined in the logic of each model's head.
2. in the override `compute_loss()` method, as you want to add the weights there; so you have to invoke  `outputs = model(**inputs)` and `logits = outputs.get("logits")` to get the logits, and then call the loss function again this time with the weights you want to consider for the labels.

Am I missing something?

Would be happy to see this PR accepted at some point in the future. From my point of view imbalanced datasets and class weights are a common problem and I dont think the current way to solve this problem is very intuitive.

Bumping this. I'm using `ViTForImageClassification` on a multi-label problem with many classes and few positive classes per example. I understand the sentiment in the comments linked above - that the goal is to provide simple and minimal interfaces; but in this particular regime, it's absolutely necessary to be able to set `pos_weight` for `BCEWithLogitsLoss`.

UPDATE: @francisco-perez-sorrosal for what it's worth, when I override `compute_loss` to incorporate weights, the compute time per iteration seems comparable. So maybe forward is called twice... But more likely it is not called twice. This is likely why you have the option to return `output` when you override `compute_loss` - precisely because it is not available anywhere else.

Final thought - I think having `pos_weight` exposed for the multilabel case is still worthwhile because it's such a common use case - and like I mentioned - without, there are many multilabel regimes that are not feasible.