T5ForConditionalGeneration `prepare_inputs_for_generation` causes problems for `sample` function
Author: HamidSajjadi
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.10.2
- Platform: Linux Manjaro
- Python version: 3.9.7
- PyTorch version (GPU?): torch==1.9.1+cpu
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
The problem is in T5ForConditionalGeneration, so I think @patrickvonplaten, @patil-suraj,
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- T5: @patrickvonplaten, @patil-suraj

Library:

- Text generation: @patrickvonplaten
- Tokenizers: @LysandreJik
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): T5ForConditionalGeneration

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)
I'm trying to do a multinomial sampling using a pre-trained T5 model ([razent/SciFive-large-Pubmed_PMC](https://huggingface.co/razent/SciFive-large-Pubmed_PMC)).

## To reproduce

Steps to reproduce the behavior:

1. create a tokenizer and model using T5ForConditionalGeneration class (e.g. [razent/SciFive-large-Pubmed_PMC](https://huggingface.co/razent/SciFive-large-Pubmed_PMC)
2. call the `model.sample(input_ids=input_ids)` with any random input_ids
3. you will encounter the following error: `You have to specify either input_ids or inputs_embeds`

I believe that the problem cause is the following line and the T5ForConditionalGeneration implementation of the function `prepare_inputs_for_generation`: 

https://github.com/huggingface/transformers/blob/234cfefbb083d2614a55f6093b0badfb2efc3b45/src/transformers/generation_utils.py#L1528

where `decoder_input_ids` is set instead of `input_ids`
https://github.com/huggingface/transformers/blob/234cfefbb083d2614a55f6093b0badfb2efc3b45/src/transformers/models/t5/modeling_t5.py#L1678

while the `forward` function, which is used in the `generation_utils.py` expects `input_ids` parameter, not `decoder_input_ids`https://github.com/huggingface/transformers/blob/234cfefbb083d2614a55f6093b0badfb2efc3b45/src/transformers/generation_utils.py#L1531 



Comments:
Hi, 
The sample method is used by both decoder only (GPT2 like) models and seq2seq (encoder-decoder) models like T5. With encoder-decoder models, during generation, the encoder is called only once and the decoder is used in the generation loop which expects `decoder_input_ids`. Since T5 is encoder-decoder it accepts both `input_ids` and `decoder_input_ids`.

And the error is because, for generation with seq2seq models we first need to pass the `input_ids` through the encoder, get the `encoder_hidden_states` and which are then passed to the decoder. If we don't pass `encoder_hidden_states` the T5 model expects `input_ids` or `input_embeds` to be able to call the encoder and get the `encoder_hidden_states`, it raises an error if none of these are passed. Which is the issue here.

For seq2seq models like T5 I would recommend you to use `generate` method directly and pass `do_sample=True` for sampling instead of directly using the `sample` method. Otherwise, you'll need to manually call the encoder to get the `encoder_hidden_states`, prepare the initial `decoder_input_ids`, and then call the `sample` method.

Hope this helps!


Thank you for the detailed response. `generate` is working properly with `do_sample=True`