Multilabel, multiclass models with >2 classes per label using BCELoss instead of CategoricalCrossentropy
Author: imatiach-msft
Pull request: no

### System Info

latest transformers

### Who can help?

@sgugger

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The PR here by @sgugger adds multilabel classification support with the BCEWithLogitsLoss:
https://github.com/huggingface/transformers/pull/14180/files

However, in the case where we have multiple labels, and the labels have multiple classes (or a mix of binary with multiple classes), it seems a bit strange to me to use BCEWithLogitsLoss and to force the user to one-hot-encode the labels with multiple classes during fine-tuning, which is the only way to make this work.  Instead, we should in that case just use CategoricalCrossentropy like in this tutorial for each of the labels:

https://towardsdatascience.com/multi-label-multi-class-text-classification-with-bert-transformer-and-keras-c6355eccb63a

Perhaps I missed something, but it seems better to me to allow the user to specify the labels without reformatting then and specify multiple CategoricalCrossentropy losses than to force them to reformat the labels into a binary one-hot-encoding for all classes across multiple labels.

For example, if the user had a dataset like:

Text                | Animal | Sound
Likes to fetch | Dog     | Woof
Likes to sleep | Cat       | Meow
Likes to fly      | Bird     |  Chirp

My understanding is that currently we have to one-hot-encode the labels into the following to make it work with BCEWithLogitsLoss which is used in the implementation:

Dog | Cat | Bird | Woof | Meow | Chirp
1      | 0      | 0    | 1       | 0         | 0   
0      | 1      | 0    | 0       | 1         | 0  
0     |  0      | 1    | 0       |  0         | 1

I think it might be useful instead to just allow the user to pass in the labels directly and use CategoricalCrossentropy.  Why is that currently not possible?

### Expected behavior

Allow the user to pass in the labels without creating a one-hot-encoded matrix for multiclass, multilabel scenario where there are >2 classes per label

Comments:
Also, just thinking about correctness, the model could possibly give both Dog and Cat as an answer when using BCEWithLogitsLoss since it treats them as independent labels during fine-tuning after one hot encoding.  This seems wrong to me.  Perhaps I am missing something here?  We should just use multiple CategoricalCrossentropy losses per label here both for correctness and to make it easier for the user to specify the labels to the model.

You are mistaking what we call a "multi-label" problem. A multi-label problem means one input can have zero, one or multiple labels. For instance the model could give both Dog and Cat as an answer because there might be both in the input.

For a model that predicts several categories of outputs, you will probably need to write your own head.

I guess I'm perhaps getting confused by the graph from scikit-learn here:

https://scikit-learn.org/stable/modules/multiclass.html?highlight=multilabel

multilabel classification is under sklearn.multioutput.  Perhaps I've heard the terms multilabel and multioutput used interchangeably too many times and I'm getting confused by that.  Also perhaps in tabular vs text contexts those terms may be used in slightly different ways.
I see now that the intention isn't to support this scenario in huggingface, so perhaps I can close this issue.  This would be a more advanced scenario/feature where there can be multiple outputs but some are grouped together.  There are also hierarchical models that output some top-level label as one class and then under that more specific labels as another class, thinking of the https://huggingface.co/datasets/DeveloperOats/DBPedia_Classes dataset here -- which in the description says "excellent benchmark for hierarchical multiclass/multilabel text classification", but perhaps it should call it multioutput since you would want the model to output all l1/l2/l3 labels with one of the multiple classes for each.  That is also similar to the hierarchical multilabel + multioutput + multiclass models some customers I have worked with recently used, although maybe if the model is supposed to output multiple labels like this where the labels are not independent "multilabel" is not the correct term then and only "multioutput" should be used.

I guess this issue is more of a feature request than a bug report to support "multioutput" instead of the current "multilabel" text classification scenario.  But this feature might be so niche and specific that it's not really worth it to implement in huggingface as another text classification parameter, so I'll just close it here.

Dog | Cat | Bird | Woof | Meow | Chirp
1 | 0 | 0 | 1 | 0 | 0
0 | 1 | 0 | 0 | 1 | 0
0 | 0 | 1 | 0 | 0 | 1

Is this format acceptable, or do we need to convert it to: One Hot? How do we convert it? / 这样的格式可以吗，还是需要转成 : One-Hot ，怎么转换？

@gg22mm allow to pass in the labels directly:
Text | Animal | Sound
Likes to fetch | Dog | Woof
Likes to sleep | Cat | Meow
Likes to fly | Bird | Chirp

would transform to:

Text | Animal | Sound
1      |           1|         1
2      |           2|         2
3      |           3|         3

or something similar - so the idea is to not have to one-hot-encode the data into binary labels.  In any case this was so long ago and I am no longer working on this right now, so I don't think this is really an important feature request, certainly not something that I need any time soon.