Shape mismatch with documentation for cross attentions tensor when performing sequence generation with encoder-decoder model
Author: fgbelidji
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.18.0
- Platform: Linux-4.19.0-20-cloud-amd64-x86_64-with-debian-10.12
- Python version: 3.7.12
- Huggingface_hub version: 0.4.0
- PyTorch version (GPU?): 1.11.0+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```


### Who can help?

@patrickvonplaten 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am trying to get the cross attentions weights of the decoder from Pegasus, but the tensor I get has a shape different from what the documentation states. I use the following code:

```
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_name = "google/pegasus-cnn_dailymail"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

prompt = "I use Hugging Face transformers everyday"

inputs = tokenizer([prompt, prompt], add_special_tokens=True, return_tensors="pt") 

batch_size = len(inputs.input_ids) # batch_size is 2
num_beams = 5
num_return_sequences = 3

outputs = model.generate(**inputs, 
                         num_beams=num_beams,
                         output_attentions=True, 
                         return_dict_in_generate=True, 
                         num_return_sequences=num_return_sequences,
                         use_cache=True
                        )

```
Size of inputs and outputs:
```
>>> inputs.input_ids.shape
torch.Size([2, 8]) # tensor of size (batch_size, input_sequence_length)

>>> outputs.sequences.shape
torch.Size([6, 47])   # tensor of size (batch_size*num_return_sequences, generated_length)
```

This is what the documentation says regarding `cross_attentions` tensor of a `BeamSearchEncoderDecoderOutput`:
https://github.com/huggingface/transformers/blob/60ad73448c7fc0149082b539ce8c223e42783a35/src/transformers/generation_utils.py#L351-L356

When inspecting the `cross_attentions`: 
```
>>>len(outputs.cross_attentions)
48 # it corresponds to the longest generated length of the 3 sequences -> OK with the doc
```

```
>>>len(outputs.cross_attentions[0])
16 # it corresponds to the number of layer of the decoder-> OK with the doc
```

```
>>>len(outputs.cross_attentions[0])
16 # it corresponds to the number of layer of the decoder-> OK with the doc
```
This is where it differs from the doc:

```
>>>cross_attentions[0][0].shape #cross attention for the 1st generated token and 1st decoder layer
torch.Size([10, 16, 1, 8]) #  tensor of size (batch_size*beam_size, num_heads, 1, sequence_length) but doc says it should be
                                         # tensor of size (batch_size, num_heads, generated_length, sequence_length)
```

```
>>>cross_attentions[1][0].shape #cross attention for the 2nd generated token and 1st decoder layer
torch.Size([10, 16, 1, 8]) #  tensor of size (batch_size*beam_size, num_heads, 1, sequence_length) but doc says it should be
                                         # tensor of size (batch_size, num_heads, generated_length, sequence_length)
```

Both the 1st and 3rd dimensions of the `cross_attentions` differ from the doc.
However when setting `use_cache=False` in `generate()` I obtain:

```
>>>cross_attentions[0][0].shape #cross attention for the 1st generated token and 1st decoder layer
torch.Size([10, 16, 1, 8]) #  tensor of size (batch_size*beam_size, num_heads, generated, sequence_length) 
```

```
>>>cross_attentions[1][0].shape #cross attention for the 2nd generated token and 1st decoder layer
torch.Size([10, 16, 2, 8]) #  tensor of size (batch_size*beam_size, num_heads, generated_length, sequence_length) 
```
Only the 1st dimension now differs from the doc, so it looks like using the cache enforces `generated_length` to 1

Is it possible to clarify this behavior? I assume the `cross_attentions` tensor and `decoder_attentions` should have the same specifications. Also, I think it would make sense to rename `sequence_length` to something like `encoder_input_sequence_length` for the `cross_attentions` spec and something like `decoder_input_sequence_length` for the `decoder_attentions` specs. At the moment, the doc implies that this is the same for both tensors.

### Expected behavior

```shell
Documentation should reflect the actual shape of cross_attentions tensor
```


Comments:
Thanks a lot @fgbelidji ! @patil-suraj do you want to give it a try? :-)

Yes, will look into it.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Have this issue been fixed?