Open discussion for design decisions.
Author: pafi-code
Pull request: no

# 🚀 Feature request

Hi all,

I'd like to ask some questions about design decisions that were made generally for this library.

**First one:**
Why did you couple the forward directly with the loss calculation? I don't really see any benefits doing this, rather downsides for example when you want to extend the `Trainer` class with a specific loss that might require specific input you always need to make sure that the `__call__` method of the model can also calculate the loss before being able to calculate your own loss.
[Roberta Example](https://github.com/huggingface/transformers/blob/19e5ed736611227b004c6f55679ce3536db3c28d/src/transformers/models/roberta/modeling_roberta.py#L1185)
[Tutorial to extend Trainer](https://huggingface.co/docs/transformers/main_classes/trainer)

**Second one:**
Also somehow related to the first one. Why do you assume different input formats for `multi label classification` and `single label classification`. Why can't you just assume that input labels for classification tasks are always one hot encoded?
[Roberta Example](https://github.com/huggingface/transformers/blob/19e5ed736611227b004c6f55679ce3536db3c28d/src/transformers/models/roberta/modeling_roberta.py#L1236)

## Motivation

I'm trying to implement a small training framework for a specific use case around this package. However things like this above make it really hard for me to have a clean implementation that is not just a list of if and else statements.

@abhishekkrthakur @sgugger @LysandreJik 

Comments:
Hi @pafi-code, regarding your first question: the loss calculation is coupled with the forward but it is opt-in. The models only perform the loss calculation when you pass the `labels`, which is not a requirement. You're perfectly free to retrieve the logits and compute your loss yourself, outside of the model.

I'll let @sgugger answer regarding the `Trainer`, and @abhishekkrthakur regarding the multi-label classification vs single-label classification.

Excited to hear you're implementing a training framework around this package, don't hesitate to share with us when you're happy with it!

Hi @LysandreJik thanks for your reply! :+1: 

Regarding the first one:
Oh lol, I didn't see line `1221` to be honest. However by default the trainer always passes also the labels to the forward right? So if I would not like that to happen, I would always need to extend the trainer such that the model get's inputs without the labels right?

Regarding the second one:
Since you just told me that if I don't pass the labels through the forward, no loss is calculated. So I could simply implement it on my own assuming there are always one hot encoded labels, which should do the trick for me. However I'm still curious why the decision was made that way :smile: 

Unfortunately this will most likely not be public. :(

First of all, questions like this should be better asked on the [forums](https://discuss.huggingface.co/) as we keep issues for bugs and feature requests only.

Regarding your last questions, labels are not one-hot encoded because this is vastly memory inefficient and the GPU memory is limited. Instead of having a tensor of size ` batch_size`, you get a tensor of size `batch_size x num_labels`. It's fine if you have 2 labels, less fine when you have a thousand and a guarantee of OOM when you are training a language model with a vast vocabulary size.

Besides the cross-entropy loss in PyTorch expects labels that are not one-hot encoded, so we couldn't use the proper loss if we did one-hot encode labels.

Alright, I'll remember for the next time!

Okay that answers my question pretty good. Totally forgot about those aspects.
Thanks! :+1: