`generate` with `num_beam` > 1 does not work in EncoderDecoder models when `past` is supplied.
Author: seongminp
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.7.0.dev0
- Platform: Linux-5.4.0-72-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): 2.5.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: Both

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people. -->

- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj

## Information

Model I am using (Bert, XLNet ...): Bart, T5

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x ] my own task or dataset: (give details below)

## To reproduce

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
```python
import torch
from transformers import BartForConditionalGeneration, BartConfig

config = BartConfig.from_pretrained('facebook/bart-base')
bart = BartForConditionalGeneration.from_pretrained('facebook/bart-base', config=config)

batch_size = 4
input_ids = torch.zeros((batch_size, 1), dtype=torch.long)
attention_mask = torch.ones((batch_size, 1))

# past_key_value: tuple of length config.n_layers with each tuple having 2 tuples each,
# of which has 2 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)
embed_size_per_head = config.d_model // config.decoder_attention_heads
keys = torch.ones(config.decoder_layers, batch_size, config.decoder_attention_heads, 1, embed_size_per_head)
past = tuple((key, key) for key in keys)

# Works.
num_beams = 1
encoder_outputs = bart.get_encoder()(input_ids.repeat_interleave(num_beams, dim=0), return_dict=True)
bart.generate(input_ids=input_ids, attention_mask=attention_mask, encoder_outputs=encoder_outputs,past=past, use_cahce=True, num_beams=num_beams)

# Doesn't work.
num_beams = 3
encoder_outputs = bart.get_encoder()(input_ids.repeat_interleave(num_beams, dim=0), return_dict=True)
bart.generate(input_ids=input_ids, attention_mask=attention_mask, encoder_outputs=encoder_outputs, past=past, use_cahce=True, num_beams=num_beams)
```
## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

In the code snippet above, second call to `generate` crashes because `past_key_values` are not supplied to all beams. 

This happened when `past` argument is passed to `generate` in models where `is_encoder_decoder` is `True` (issue seen in Bart and T5).

To mitigate this issue, `past` should also be expanded in `_expand_inputs_for_generation` in `generation_utils.py`. (I've noticed that, at this point in the generation process, the script looks for `past` not `past_key_values` in `model_kwargs`.) 

I've submitted a pull request that applies the above mentioned patch.


Comments:
 Hey @seongminp,

Thanks for the issue report. It's a rather specific use-case to pass `past_key_values` to `generate()`. Could you give me some more detail when you need to do so? 

Hi @patrickvonplaten!

My use-case for passing `past_key_values` to `generate` is to manipulate the encoder hidden states before passing them to  decoder's cross attention. 

Specifically, I am using a encoder-decoder generative (as in modeling the latent space, like GAN or VAE) text model.  

Several existing works, like [Microsoft's Optimus](https://github.com/ChunyuanLI/Optimus) and [Fang et al.](https://arxiv.org/abs/2101.00828), adds custom manipulations for key/value of decoder's cross attention.

Official implementations of Optimus and Fang et al. are both implemented with this wonderful library, but uses a custom `generate` function because right now restrictions mentioned in this issue exists while passing `past` to `generate`.

Would love to hear your feedback!

Hey @seongminp,

Thanks for the feedback! The problem is that the `past` variable strongly varies from model to model. *E.g.* Bart uses a different `past` tuple structure then `gpt2` does and `xlnet` uses a completely different structure. We would have to add a specific `prepare_cache` method to each model which seems would add to much complexity to the `generate()` method for quite a specific case IMO. Do you think we could instead solve it by just forcing the user to preprocess `past` correctly before passing it to `generate()`? E.g., the following code:

```python
  past = tuple(                    
    (
         layer[0].index_select(0, expanded_return_idx).to(layer[0].device),                        
         layer[1].index_select(0, expanded_return_idx).to(layer[1].device),                    
     )                    
    for layer in past                
)
```

 could be executed by the user before calling `model.generate(input_ids, past=past)` no? 
We could make a nice forum post about it so that people interested in the work mentioned above would have access to the correct pre-processing of `past` :-) 

What do you think?



Hi again.

That makes more sense.

Trying to encompass all uses of `past` in `generate_utils` seems to be more trouble than it is worth. 

I'll close the pull request. Feel free to close this issue also!


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.