Why doesn't gpt2 model implement ignore_index in the loss?
Author: kirk86
Pull request: no

### Feature request

Going through the the [gpt2 model](https://github.com/huggingface/transformers/blob/v4.39.1/src/transformers/models/gpt2/modeling_gpt2.py) looks like it doesn't implement the `ignore_index` in it's loss besides the models for `ForQuestionAnswering`. 

Why is that? Is it this the intended behaviour?

### Motivation

I understand that some examples show that during preprocessing one should split the data into equal size chunks which would eliminate the need for `ignore_index` in the loss. But, at the same time when we do chunking we end up throwing data away that are not equal the to the `context_length`. 

Plus, this way we can't use the `DataCollatorForLanguageModeling` as it is, most examples resort into using the `default_data_collator`. 

Wouldn't it be easier to have `ignore_index` and then use which ever data collator makes more sense? As a matter in fact the `ignore_index` should either be passed by the user in the `config.ignore_index` or `model.ignore_index` or should be set automatically depending on the `data_collator` or the values of the data passed as input into the model forward call.

Is there a way to have a proper explanation on the docs with boldface letters whenever such decisions  are made so that we don't end up getting confused all the time and having to check the code definition of each model to understand what the heck we need to pass into this model and in what format does the model expect the data to be in.

### Your contribution

I don't know if this requires a PR but nonetheless happy to contribute.

Comments:
Hi,

There's no need to implement `ignore_index` as the [CrossEntropyLoss](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html) by PyTorch is [used by default](https://github.com/huggingface/transformers/blob/cbe58b4269457a6ca66a556224b23f9ef246f905/src/transformers/models/gpt2/modeling_gpt2.py#L1106), which supports `ignore_index` (with value -100) by default. You'll see various tutorials of Transformers which set the labels which do not need to incur a loss to -100.

Also note that a model like GPT-2 typically doesn't set labels to -100 during pre-training, as one just takes all the text from the corpora and concatenates them together. Those then get chunked into batches which the model is trained on. One only uses the ignore_index during supervised fine-tuning (SFT), as one typically doesn't want to incur a loss on padding tokens or on human instructions, but only on the completions.

Hi Niels, 

thanks for the reply I think I do get the point.
My question stems from the scenario where you have data and labels and want to train a model like GPT-2.
Do you concatenate data and labels in that case like below before chunking them into batches?

```
data = ["data_point_1", "data_point_2", "data_point_3"]
labels = ["label_1", "labels_2", "labels_3"]
combined = ["data_point_1 label_1", "data_point_2 label_2", "data_point_3 label_3"]
```

For instance, in the T5 model the labels are padded with zeros and then changed into `-100` to not calculate a loss on those padded tokens. I was under the impression that something similar might work for GPT-2 style models?
I know that GPT-2 doesn't require per se labels but if you happen to have them what's the best way to utilize them when training models for text generation/causal language modeling like GPT-2, Llama, Bart etc.


Can you point me to an example of doing pre-training and then fine-tuning as you described?

> One only uses the ignore_index during supervised fine-tuning (SFT), as one typically doesn't want to incur a loss on padding tokens or on human instructions, but only on the completions.



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.