BART with inputs_embeds crashes with shift_tokens_right
Author: pedrocolon93
Pull request: no

## Environment info
- `transformers` version: 4.11.0
- Platform: Linux-5.4.0-88-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.9.1+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


### Who can help
@patrickvonplaten, @patil-suraj


## Information

Model I am using: BART

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Get the input embeddings directly from the shared Embedding layer, build a dictionary without 'input_ids', pass it to the model 

```python
     model = BartModel.from_pretrained("facebook/bart-base")  
     tokenizer = BartTokenizer.from_pretrained("facebook/bart-base")
     bos_vec = model.shared(torch.tensor([tokenizer.bos_token_id])*model.encoder.embed_scale
     final_fake_batch = {
                         "inputs_embeds":[],
                     "attention_mask":[]
                 }
     vecs = [bos_vec,bos_vec]
     vecs = torch.cat(vecs,0)
     att = [1 for i in range(2)]
     final_fake_batch["inputs_embeds"].append(vecs)
     final_fake_batch["attention_mask"].append(torch.tensor(att))
     final_fake_batch["attention_mask"].append(torch.tensor(att))
     model(**final_fake_batch)  <-error happens here
```
I didn't check this ^^^ but it should give the same error.  Within the modeling_bart, you will get a NoneType error that happens in what is now:
```python
 def _prepare_bart_decoder_inputs(
     config, input_ids, decoder_input_ids=None, decoder_padding_mask=None, causal_mask_dtype=torch.float32
 ):
     """
     Prepare masks that ignore padding tokens in the decoder and a causal mask for the decoder if none are provided.
     This mimics the default behavior in fairseq. To override it pass in masks. Note: this is not called during
     generation
     """
     pad_token_id = config.pad_token_id
     if decoder_input_ids is None:
         **decoder_input_ids = shift_tokens_right(input_ids, pad_token_id) <<< HERE**
     bsz, tgt_len = decoder_input_ids.size()
     if decoder_padding_mask is None:
         decoder_padding_mask = make_padding_mask(decoder_input_ids, pad_token_id)
     else:
         decoder_padding_mask = invert_mask(decoder_padding_mask)
     if decoder_padding_mask is not None and decoder_padding_mask.shape[1] > 1:
         # never mask leading token, even if it is pad
         decoder_padding_mask[:, 0] = decoder_padding_mask[:, 1]
     tmp = fill_with_neg_inf(torch.zeros(tgt_len, tgt_len))
     mask = torch.arange(tmp.size(-1))
     tmp.masked_fill_(mask < (mask + 1).view(tmp.size(-1), 1), 0)
     causal_mask = tmp.to(dtype=causal_mask_dtype, device=decoder_input_ids.device)
     return decoder_input_ids, decoder_padding_mask, causal_mask
```
This function assumes input_ids is always going to be provided, which it might not.  A quick (albeit sloppy) fix is:
```python
 if decoder_input_ids is None and decoder_inputs_embeds is None:
     if input_ids is not None:
         decoder_input_ids = shift_tokens_right(
             input_ids, self.config.pad_token_id, self.config.decoder_start_token_id
         )
     elif inputs_embeds is not None:
         decoder_inputs_embeds = shift_embeddings_right(
             inputs_embeds, self.shared(
                 torch.tensor([self.config.decoder_start_token_id]).to(self.shared.weight.device)).unsqueeze(0)
         )
```
In which we essentially check if input ids are none, and if they are just check if we have provided inputs_embeds. Additionally, since it is no longer single tokens, a shift embedding function needs to be implemented, or the shift tokens right needs to be modified:
```python
 def shift_embeddings_right(inputs_embeds, decoder_start_token_vec):
     shifted_input_ids = inputs_embeds.new_zeros(inputs_embeds.shape)
     shifted_input_ids[:, 1:, :] = inputs_embeds[:, :-1,:]#.clone()
     shifted_input_ids[:, 0, :] = decoder_start_token_vec
     return shifted_input_ids
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Model should work with inputs_embeds rather than input_ids since it makes a check for one or the other. 



Comments:
If `inputs_embeds` are provided instead of `input_ids`, the `decoder_inputs_embeds` need to be provided as well. The problem is that when costum word embeddings are used we cannot automatically shift the input as we don't know what the "padding" token word embedding is as the user could have passed whatever word embedding. *E.g.*:

```python
decoder_inputs_embeds = shift_embeddings_right(
   inputs_embeds, self.shared(
       torch.tensor([self.config.decoder_start_token_id]).to(self.shared.weight.device)).unsqueeze(0)
)
```

makes the assumption the actualy word embedding is used, but we can't know that beforehand so the better solution is to just force the user to provide the `decoder_inputs_embeds` in this case. I'm happy about adding a better error message, but think that's all we can do here

Thats perfect! I didn't know the decoder inputs embeds were needed! just a tiny clarification would be good! 



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@pedrocolon93,

Would you be interested in open a PR to add such an error message maybe? :-)