Cross Entropy Loss Ignore Index in BARTForConditionalGeneration
Author: dipta007
Pull request: no

### System Info

This bug doesn't depend on the environment.

### Who can help?

@ArthurZucker @younesbelkada 

In the [code](https://github.com/huggingface/transformers/blob/6cbc1369a330860c128a1ba365f246751382c9e5/src/transformers/models/bart/modeling_bart.py#L1412), BARTForConditionalGeneration used the default cross-entropy loss, where the ignore_index is -100. To my understanding, the ignore index should be the pad_index, which is 1. I am not sure if I am missing something.

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
model_name = "facebook/bart-base"

model = BartForConditionalGeneration.from_pretrained(model_name)
tokenizer = BartTokenizer.from_pretrained(model_name)

input = [
    "Long live America.",
    "I love Huggingface as it has made the ML development so easy."
]

input_tok = tokenizer(input, return_tensors='pt', padding=True)

out = model(input_tok['input_ids'], labels=input_tok['input_ids'])



default_loss = out.loss


criterion = torch.nn.CrossEntropyLoss(ignore_index=model.config.pad_token_id)
should_be_loss = criterion(out.logits.view(-1, out.logits.size(-1)), input_tok['input_ids'].view(-1))


print(default_loss.item(), should_be_loss.item())
```

### Expected behavior

The loss should be `0.12098739296197891`
But the loss comes with the default loss output `4.7771992683410645`

Comments:
Hello, the default ignore index of the cross entropy loss is `-100` which is what we use. 
If you follow the docstring, the labels are supposed to be given by the user:

>         labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`):
>            Labels for computing the sequence classification/regression loss. Indices should be in :obj:`[-100, 0, ...,
>            config.vocab_size - 1]`. All labels set to ``-100`` are ignored (masked), the loss is only computed for
>            labels in ``[0, ..., config.vocab_size]``

The huggingface [datacollator](https://github.com/huggingface/transformers/blob/main/src/transformers/data/data_collator.py#L556) usually does this. 

This gives more freedom to the user. 

Thanks @ArthurZucker. So, I misunderstood. 
If I understood it correctly, if I only use BARTTokenizer, then I have to first replace all the `pad_tokens` with `-100` before passing it to the model. Is that correct?


Hi @dipta007 , yes that can also do the trick, you can also use the `shift_tokens_right` method: https://github.com/huggingface/transformers/blob/main/src/transformers/models/bart/modeling_bart.py#L74 that will properly shift your tokens to the right and correctly compute the loss

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.