shift_tokens_right in BART, FSMT incompatible with DataCollatorForLanguageModelling
Author: jethrokuan
Pull request: no

## Environment info
- `transformers` version: 4.2.0dev0
- Platform: Linux-4.14.81.bm.15-amd64-x86_64-with-debian-9.11
- Python version: 3.7.9
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten

## Information
I'm trying to train Bart from scratch on a masked language modelling task. I understand that this is currently not supported by HF, but I'm working on it and would like to bring up certain "blockers" that currently prevent this.

The Bart shift_tokens_right implementation looks like this: 

```
def shift_tokens_right(input_ids: torch.Tensor, pad_token_id: int):
    """
    Shift input ids one token to the right, and wrap the last non pad token (usually <eos>).
    """
    prev_output_tokens = input_ids.clone()

    assert pad_token_id is not None, "self.model.config.pad_token_id has to be defined."
    # replace possible -100 values in labels by `pad_token_id`
    prev_output_tokens.masked_fill_(prev_output_tokens == -100, pad_token_id)

    index_of_eos = (prev_output_tokens.ne(pad_token_id).sum(dim=1) - 1).unsqueeze(-1)
    decoder_start_tokens = prev_output_tokens.gather(1, index_of_eos).squeeze()
    prev_output_tokens[:, 1:] = prev_output_tokens[:, :-1].clone()
    prev_output_tokens[:, 0] = decoder_start_tokens

    return prev_output_tokens
```

The `shift_tokens_right` implementation assumes that anything that was filled as `-100` was a pad token, using that to try to find the index of the EOS token.

This is not always true. In `DataCollatorForLanguageModelling`, which is used in the example script, we see
https://github.com/huggingface/transformers/blob/748006c0b35d64cdee23a3cdc2107a1ce64044b5/src/transformers/data/data_collator.py#L303

This causes errors when trying to train a Bart model on language modelling.

```
Traceback (most recent call last):
  File "math_explain/masked_lm.py", line 281, in <module>
    main()
  File "math_explain/masked_lm.py", line 236, in main
    train_result = trainer.train()
  File "/home/tiger/.local/lib/python3.7/site-packages/transformers/trainer.py", line 815, in train
    tr_loss += self.training_step(model, inputs)
  File "/home/tiger/.local/lib/python3.7/site-packages/transformers/trainer.py", line 1157, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/tiger/.local/lib/python3.7/site-packages/transformers/trainer.py", line 1181, in compute_loss
    outputs = model(**inputs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/tiger/.local/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py", line 1233, in forward
    decoder_input_ids = shift_tokens_right(labels, self.config.pad_token_id)
  File "/home/tiger/.local/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py", line 75, in shift_tokens_right
    decoder_start_tokens = prev_output_tokens.gather(1, index_of_eos).squeeze()
RuntimeError: index -1 is out of bounds for dimension 1 with size 213
```

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)


Comments:
Hey @jethrokuan, we've merged a big BART PR yesterday just as a heads up, I think this might solve this problem for Bart -> could you check again?

@patrickvonplaten I think it does solve this problem: code looks good and runs fine, model results not great, but possibly a mistake of mine. Thanks!