Cannot restore `sequences_scores` from `scores` and `beam_indices` returned by `t5-base`
Author: namespace-Pt
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.18.0
- Platform: Linux-5.10.25-nvidia-gpu-x86_64-with-glibc2.31
- Python version: 3.9.12
- Huggingface_hub version: 0.5.1
- PyTorch version (GPU?): 1.10.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <No>
- Using distributed or parallel set-up in script?: <No>

### Who can help?

I was trying to restore the logit of each generated token in `t5-base` by `beam_search`. However, I found that the `sequences_scores` cannot be computed from the generated token indices, the `beam_indices`, and the `scores` returned by `model.generate()`. 

Here is my script with annotations:
```python
from transformers import T5ForConditionalGeneration, T5Tokenizer
# load model
model = T5ForConditionalGeneration.from_pretrained("t5-base")
tokenizer = T5Tokenizer.from_pretrained("t5-base")
inputs = tokenizer(["I love hugging face", "I love deep learning"], return_tensors="pt", truncation=True, padding="max_length", max_length=10)

max_length = 10
min_length = 10
num_beams = 10
num_return_seq = 2

outputs = model.generate(
    input_ids=inputs.input_ids,
    attention_mask=inputs.attention_mask,
    do_sample=False,
    max_length=max_length, 
    num_beams=num_beams, 
    num_return_sequences=num_return_seq,
    return_dict_in_generate=True,
    output_scores=True
)

sequences = outputs.sequences.transpose(0, 1)   # num_step + 1, batch_size * num_return_seq
beam_indices = torch.tensor(outputs.beam_indices).view(-1, max_length - 1) # batch_size * num_return_seq, num_step
scores = torch.stack(outputs.scores, dim=0)     # num_step, batch_size * num_beams, vocab_size
beam_indices = beam_indices.transpose(-1, -2)  # num_step, batch_size * num_return_seq

# get the associated logits over the vocabulary at each step
selected_distribution = scores.gather(dim=-2, index=beam_indices.unsqueeze(-1).expand(*beam_indices.shape, scores.shape[-1]))   # num_step, batch_size * num_return_seq, vocab_size
# get the associated logit of the selected token at each step
selected_score = selected_distribution.gather(dim=-1, index=sequences[1:].unsqueeze(-1))
# output cumulative scores and sequences_scores
>>> selected_score.squeeze().mean(0)
<<< tensor([-0.2241, -0.3926, -0.1859, -0.4169])
>>> outputs.sequences_scores
<<< tensor([-0.2017, -0.3534, -0.1674, -0.3752])
```

Why these two scores are unequal? Are there any specific notes for computing sequence scores? Also, I think returning the token score along with the `Generate Outputs` is **useful**. Thanks in advance for your reply: @patrickvonplaten, @Narsil, @gante.

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Just use the above code.

### Expected behavior

I prefer to know how to get the logit of each generated token in `T5`, from which I can properly restore `sequences_scores` returned by the model. Further, I think returning the logit of each generated token is useful for developing systems trying to use that score.

Comments:
Hi @namespace-Pt 👋 

Does [this function](https://github.com/huggingface/transformers/blob/dbd9641c8c0e146c078cbee11cdefcf556f6c817/src/transformers/generation_utils.py#L804) solve your issue?

I noticed that it is undocumented, so it is hard to find 😬 

@gante Thanks, the function worked. However, in the above example, how can I get the `sequences_scores` given the returned transition scores?

@namespace-Pt check these two threads:
1. https://github.com/huggingface/transformers/issues/16413
2. https://github.com/huggingface/transformers/issues/15869

TL;DR you can't directly atm unless you specify a length penalty of 0

Got that. Thank you.