past_key_values tuple index out of range error when using text2text-generation pipeline with encoder-decoder model
Author: thominj
Pull request: no

## Environment info
- `transformers` version: 4.3.0
- Platform: Linux-5.4.0-65-generic-x86_64-with-Ubuntu-20.04-focal
- Python version: 3.7.9
- PyTorch version (GPU?): 1.7.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@LysandreJik

## Information

Model I am using (Bert, XLNet ...): I am using the encoder-decoder model with a Roberta encoder and RobertaForCausalLM decoder.

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

In my real code I am using custom pre-trained models and tokenizers, but the error and behavior is the same as that produced by the demo script below.


The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

I am trying to use a pipeline to generate results from an encoder-decoder model that was trained on a custom text2text dataset.

## To reproduce

Steps to reproduce the behavior:

You can just run the script below, or:

1. Load an encoder-decoder model with RoBERTa encoder and decoder
2. Create a text2text-generation pipeline with an appropriate tokenizer
3. Use the pipeline to generate a result

```python
import transformers

encoder = transformers.RobertaModel.from_pretrained(pretrained_model_name_or_path='roberta-base')
decoder = transformers.RobertaForCausalLM.from_pretrained(pretrained_model_name_or_path='roberta-base')

encoder_decoder_model = transformers.EncoderDecoderModel(encoder=encoder, decoder=decoder)

tokenizer = transformers.AutoTokenizer.from_pretrained('google/roberta2roberta_L-24_bbc')

text2text = transformers.pipeline('text2text-generation', model=encoder_decoder_model, tokenizer=tokenizer)

output = text2text('This is a test sentence.')

print(output)

```

Output:

```
If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`
normalizer.cc(51) LOG(INFO) precompiled_charsmap is empty. use identity normalization.
Traceback (most recent call last):
  File "demo.py", line 12, in <module>
    output = text2text('This is a test sentence.')
  File "/home/james/Code/demo/.env/lib/python3.7/site-packages/transformers/pipelines/text2text_generation.py", line 125, in __call__
    **generate_kwargs,
  File "/home/james/Code/demo/.env/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/home/james/Code/demo/.env/lib/python3.7/site-packages/transformers/generation_utils.py", line 913, in generate
    **model_kwargs,
  File "/home/james/Code/demo/.env/lib/python3.7/site-packages/transformers/generation_utils.py", line 1177, in greedy_search
    output_hidden_states=output_hidden_states,
  File "/home/james/Code/demo/.env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/james/Code/demo/.env/lib/python3.7/site-packages/transformers/models/encoder_decoder/modeling_encoder_decoder.py", line 430, in forward
    **kwargs_decoder,
  File "/home/james/Code/demo/.env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/james/Code/demo/.env/lib/python3.7/site-packages/transformers/models/roberta/modeling_roberta.py", line 937, in forward
    return_dict=return_dict,
  File "/home/james/Code/demo/.env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/james/Code/demo/.env/lib/python3.7/site-packages/transformers/models/roberta/modeling_roberta.py", line 771, in forward
    past_key_values_length = past_key_values[0][0].shape[2] if past_key_values is not None else 0
IndexError: tuple index out of range

```


## Expected behavior

I expect the pipeline to generate an output string.


Comments:
I have been digging into this a little bit more and found some information that might be helpful. It looks like the underlying problem is in the EncoderDecoderModel or one of its dependencies, not the pipeline. 

- When I replaced the pipeline call with a manual tokenization and call to the model's generate method, I got the same `tuple index out of range`  error for past_key_values.
- When I created the encoder_decoder_model using `transformers.EncoderDecoderModel.from_encoder_decoder_pretrained('roberta-base', 'roberta-base')`, the pipeline prediction worked. 
- If I use the AutoModel and AutoModelForCausulLM `from_pretrained` methods to create the encoder and decoder (mirroring the way that `from_encoder_decoder_pretrained` works) and then pass them to the EncoderDecoderModel constructor, I still get the `index out of range` error.
- If I use the `AutoModel.from_pretrained()` methods to create the encoder and decoder, then call `save_pretrained()` on them to save in a local directory, then load them using `EncoderDecoderModel.from_encoder_decoder_pretrained()`, the pipeline prediction works.

I believe there is some difference between the ways that EncoderDecoderModel's `init()` and `from_encoder_decoder_pretrained()` functions work that is leading to this error, but I haven't been able to figure out what the difference is, or why it is happening.

@thominj can you try with `decoder = transformers.RobertaForCausalLM.from_pretrained(pretrained_model_name_or_path='roberta-base', add_cross_attention=True, is_decoder=True, bos_token_id=<bos-id>, eos_token_id=<eos-id>)`?

> @thominj can you try with `decoder = transformers.RobertaForCausalLM.from_pretrained(pretrained_model_name_or_path='roberta-base', add_cross_attention=True, is_decoder=True, bos_token_id=<bos-id>, eos_token_id=<eos-id>)`?

That worked!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Is this considered to be expected behavior? If so, are add_cross_attention, is_decoder, bos_token_id, and eos_token_id all required for every decoder that can be used in EncoderDecoderModel?

HI @thominj 

Yes, `add_cross_attention` and `is_decoder` is required if you are initializing the model as a decoder yourself.

But if you do 
```python
model = EncoderDecoderModel. from_encoder_decoder_pretrained("roberta-base", "roberta-base")
```

then it'll happen automatically, the `from_encoder_decoder_pretrained` method takes care of this.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

The same exception may also be raised when model is in train mode, call model.eval() before may solve this problem. It happened when I use model `BartForConditionalGeneration`.