Conditional Generation using input_embeds instead of input_ids
Author: chun-gan
Pull request: no

Hi @patrickvonplaten! 

When using input_embeds instead of input_ids as inputs to the BartForConditionalGeneration model, I am not able to generate the result. Could you please take a look? The same code works with GPT2 in place of Bart. Thanks! 

Here is the script
```
import torch
from transformers import BartForConditionalGeneration, BartTokenizer
  
model_path = "facebook/bart-large"
model = BartForConditionalGeneration.from_pretrained(model_path, output_hidden_states=True)
tokenizer = BartTokenizer.from_pretrained(model_path)
 
text = "I disapprove of what you <mask> , but"
input_ids = tokenizer.encode_plus(text, return_tensors='pt')['input_ids']

with torch.no_grad():
    x = model.get_input_embeddings()(input_ids).squeeze()

model(inputs_embeds = x)
```

Here is the Traceback I got
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-21-e17be5532b10> in <module>()
     18     x = model.get_input_embeddings()(input_ids).squeeze()
     19 
---> 20 model(inputs_embeds = x)
     21 

4 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/transformers/models/bart/modeling_bart.py in forward(self, input_ids, attention_mask, decoder_input_ids, decoder_attention_mask, encoder_outputs, past_key_values, inputs_embeds, decoder_inputs_embeds, labels, use_cache, output_attentions, output_hidden_states, return_dict)
   1244             output_attentions=output_attentions,
   1245             output_hidden_states=output_hidden_states,
-> 1246             return_dict=return_dict,
   1247         )
   1248         lm_logits = self.lm_head(outputs[0]) + self.final_logits_bias

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/transformers/models/bart/modeling_bart.py in forward(self, input_ids, attention_mask, decoder_input_ids, decoder_attention_mask, encoder_outputs, past_key_values, inputs_embeds, decoder_inputs_embeds, use_cache, output_attentions, output_hidden_states, return_dict)
   1079         # -> is this used for backward compatibility
   1080         if decoder_input_ids is None and decoder_inputs_embeds is None:
-> 1081             decoder_input_ids = shift_tokens_right(input_ids, self.config.pad_token_id)
   1082 
   1083         output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions

/usr/local/lib/python3.6/dist-packages/transformers/models/bart/modeling_bart.py in shift_tokens_right(input_ids, pad_token_id)
     67     Shift input ids one token to the right, and wrap the last non pad token (usually <eos>).
     68     """
---> 69     prev_output_tokens = input_ids.clone()
     70 
     71     assert pad_token_id is not None, "self.model.config.pad_token_id has to be defined."

AttributeError: 'NoneType' object has no attribute 'clone'
```

- `transformers` version: 4.1.1
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.4.0 (True)








Comments:
Hey @frankgandiao,

Note that Encoder-Decoder models usually require both `input_ids` and `decoder_input_ids`. Bart is special in a sense that it can automatically create the `decoder_input_ids` from the `input_ids` if you **don't** provide the `decoder_input_ids`. However, the model is not able to automatically create the `decoder_inputs_embeds` from the `inputs_embeds` if you provide only the `inputs_embeds` => to solve your problem you should provide the `decoder_inputs_embeds` as well. What you could do is the following: 

```python
from transformers.models.mbart.modeling_mbart.py import shift_tokens_right

input_ids = tokenizer(text, return_tensors='pt')['input_ids']
decoder_input_ids = shift_tokens_right(input_ids, tokenizer.pad_token_id)

inputs_embeds = model.get_input_embeddings()(input_ids).squeeze()
decoder_inputs_embeds = model.get_input_embeddings()(decoder_input_ids).squeeze()

model(inputs_embeds=inputs_embeds, decoder_inputs_embeds=decoder_inputs_embeds)
```

Hey @patrickvonplaten, 

Thanks for your reply! That makes sense and the problem is resolved!