Generate Function call throughs error when "inputs_embeds" argument passed
Author: abhikasd6523
Pull request: no

When using `inputs_embeds` as the argument instead of 'input_ids' while trying to generate text with GPT2 model, an error pops up about `input_ids`. 

```
from transformers import GPT2Tokenizer, GPT2LMHeadModel, GPT2Config
import transformers

import torch
import torch.nn as nn

model = GPT2LMHeadModel.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

some_random_texts = "This is a nice place to eat"
tokenized_text = tokenizer.encode(some_random_texts, return_tensors='pt')
tokenized_text_embeds = model.transformer.wte(tokenized_text)

output = model.generate(inputs_embeds=tokenized_text_embeds, max_length=50)
```


The error generated is:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-2-643fe4803ba4> in <module>
     12 tokenized_text_embeds = model.transformer.wte(tokenized_text)
     13 
---> 14 output = model.generate(inputs_embeds=tokenized_text_embeds, max_length=50)

C:\ProgramData\Anaconda3\envs\hugging_face\lib\site-packages\torch\autograd\grad_mode.py in decorate_context(*args, **kwargs)
     24         def decorate_context(*args, **kwargs):
     25             with self.__class__():
---> 26                 return func(*args, **kwargs)
     27         return cast(F, decorate_context)
     28 

C:\ProgramData\Anaconda3\envs\hugging_face\lib\site-packages\transformers\generation_utils.py in generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, encoder_no_repeat_ngram_size, num_return_sequences, max_time, decoder_start_token_id, use_cache, num_beam_groups, diversity_penalty, prefix_allowed_tokens_fn, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, forced_bos_token_id, forced_eos_token_id, remove_invalid_values, synced_gpus, **model_kwargs)
    891             # init `attention_mask` depending on `pad_token_id`
    892             model_kwargs["attention_mask"] = self._prepare_attention_mask_for_generation(
--> 893                 input_ids, pad_token_id, eos_token_id
    894             )
    895 

C:\ProgramData\Anaconda3\envs\hugging_face\lib\site-packages\transformers\generation_utils.py in _prepare_attention_mask_for_generation(self, input_ids, pad_token_id, eos_token_id)
    401         if is_pad_token_in_inputs_ids and is_pad_token_not_equal_to_eos_token_id:
    402             return input_ids.ne(pad_token_id).long()
--> 403         return input_ids.new_ones(input_ids.shape, dtype=torch.long)
    404 
    405     def _prepare_encoder_decoder_kwargs_for_generation(

AttributeError: 'NoneType' object has no attribute 'new_ones'
```


While solving this issue, I find that 'attention_mask' argument needs to be included too, to avoid this `self._prepare_attention_mask_for_generation` function being called altogether, another error pops ups.

The changes made are as follows:
```
from transformers import GPT2Tokenizer, GPT2LMHeadModel, GPT2Config
import transformers

import torch
import torch.nn as nn

model = GPT2LMHeadModel.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

some_random_texts = "This is a nice place to eat"
tokenized_text = tokenizer.encode(some_random_texts, return_tensors='pt')
tokenized_text_embeds = model.transformer.wte(tokenized_text)

att_mask = torch.ones(tokenized_text.shape[1])

output = model.generate(inputs_embeds=tokenized_text_embeds, attention_mask=att_mask, max_length=50)
```

and the error that pops up now is:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-7-17cdf18aa52b> in <module>
     14 att_mask = torch.ones(tokenized_text.shape[1])
     15 
---> 16 output = model.generate(inputs_embeds=tokenized_text_embeds, attention_mask=att_mask, max_length=50)

C:\ProgramData\Anaconda3\envs\hugging_face\lib\site-packages\torch\autograd\grad_mode.py in decorate_context(*args, **kwargs)
     24         def decorate_context(*args, **kwargs):
     25             with self.__class__():
---> 26                 return func(*args, **kwargs)
     27         return cast(F, decorate_context)
     28 

C:\ProgramData\Anaconda3\envs\hugging_face\lib\site-packages\transformers\generation_utils.py in generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, encoder_no_repeat_ngram_size, num_return_sequences, max_time, decoder_start_token_id, use_cache, num_beam_groups, diversity_penalty, prefix_allowed_tokens_fn, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, forced_bos_token_id, forced_eos_token_id, remove_invalid_values, synced_gpus, **model_kwargs)
    917                 raise ValueError("Make sure that `model_kwargs` include `encoder_outputs` of type `ModelOutput`.")
    918 
--> 919         if input_ids.shape[-1] >= max_length:
    920             input_ids_string = "decoder_input_ids" if self.config.is_encoder_decoder else "input_ids"
    921             logger.warning(

AttributeError: 'NoneType' object has no attribute 'shape'
```

Am I doing something wrong here or is there is a bug in the code of `generation_utils.GenerationMixin.generate()`?

Version control:
transformers: 4.6.1
torch: 1.7.1
python: 3.7.4

@patrickvonplaten 

Comments:
Hey @abhikasd6523,

I don't think that `generate()` currently supports `inputs_embeds` correctly. It would require quite some changes in `generate()` to make it work I'm afraid. Can you give me some more background on your use-case for passing `inputs_embeds` instead of `input_ids`. If it's a general enough use-case, I think we could try to make the required changes to `generate()`

Thanks a lot for replying.
I am trying to connect a custom encoder to the GPT2 model and would want to pass the vectored last layer values of the Encoder as input embeds. The goal is to generate a random sentence with this type of connection and architecture.

I see - did you try to directly use the `sample(...)` method? Think that this could work

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.