Calling `generate` on a `T5ForConditionalGeneration` returns `n` tokens but `n-1` scores
Author: ClementRomac
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.20.1
- Platform: Linux-5.4.0-113-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.11.0+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no
```


### Who can help?

@patrickvonplaten, @Narsil

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
import torch

if __name__ == '__main__':
    torch.manual_seed(0)
    tokenizer = AutoTokenizer.from_pretrained('t5-small')
    model = AutoModelForSeq2SeqLM.from_pretrained('t5-small')

    input = tokenizer.encode("I enjoy walking with my cute dog", return_tensors='pt')
    result = model.generate(
        input,
        max_new_tokens=15,
        do_sample=True,
        return_dict_in_generate=True,
        output_scores=True,
    )

    print(len(result["scores"]))
    for sequence in result["sequences"]:
        print(len(sequence))
        print(tokenizer.decode(sequence))
```

Output:
```
15
16
<pad> Ich, liebe es, mes lustig beim laufen
```

### Expected behavior

I would have expected to have up to 15 tokens (as `max_new_tokens=15`) and `len(result["scores"]) == len(result["sequences"][0])`. However, the size of the returned sequence of tokens is always `len(result["scores"]) + 1`. In addition, if `max_new_tokens` is reached we have `len(result["sequences"][0]) == max_new_tokens + 1`.

When looking at the decoded sequence, there is always a pad token at the beginning.

I don't know if this is necessarily a bug but this behaviour is somewhat confusing, especially when trying to compute the probability of the sequence given scores.

Comments:
Hi, @ClementRomac 

If you look the [config.json](https://huggingface.co/t5-small/blob/main/config.json) file of the `t5-small` model, you will see it uses  `pad_token_id` as `decoder_start_token_id` (both are `0`).

The `scores` having length `len(sequence) - 1` is expected. Think it this way, 

```python
generated sequence = [decoder_start_token_id, token_1, token_2]
```

The scores is/are:

- score for generating `token_1` while we have `[decoder_start_token_id]`
- score for generating `token_2` while we have `[decoder_start_token_id, token_1]`

This is also documented in [generation_utils.py](https://github.com/huggingface/transformers/blob/main/src/transformers/generation_utils.py), for example

(`SampleEncoderDecoderOutput`)
https://github.com/huggingface/transformers/blob/afb71b672679e57449085e4955a321db8e5705b9/src/transformers/generation_utils.py#L172
or
(`GreedySearchEncoderDecoderOutput`)
https://github.com/huggingface/transformers/blob/afb71b672679e57449085e4955a321db8e5705b9/src/transformers/generation_utils.py#L101

etc.

Hey @ydshieh,

Thanks for your answer, it makes sense!

Could we consider documenting it a little bit more somewhere? I don't have any clear idea on where to put it but to be honest this behaviour can appear a bit confusing when looking at the documentation.

For instance, in [generation_utils.py](https://github.com/huggingface/transformers/blob/main/src/transformers/generation_utils.py), it is mentioned (both for `SampleEncoderDecoderOutput` and `GreedySearchEncoderDecoderOutput`):

1. that `sequence_length` should be up to `max_length` (however we get `max_length +1` in the above example)
2. that `scores` will have size `max_length-1` (however we get `max_length` scores in the above example) 

https://github.com/huggingface/transformers/blob/1dfa03f12b3748dc7e9c2b5ada40c3401ada23a5/src/transformers/generation_utils.py#L169-L175

@ClementRomac ,

I think it is because you use `max_new_tokens=15,` instead of the argument `max_length.`

See https://github.com/huggingface/transformers/blob/1dfa03f12b3748dc7e9c2b5ada40c3401ada23a5/src/transformers/generation_utils.py#L925-L929

I think it is quite well documented. It is possible to make it even more explicit to include `max_new_tokens` regarding the output format.

@patrickvonplaten  Do you think we should add this in `GreedySearchEncoderDecoderOutput` etc ..?

Always happy to make the generate docs more explicit!

Also gently pinging @gante here for feedback :-) 

Note: Some docstrings associated with `scores` have 

```
`(max_length-1,)`-shaped tuple of `torch.FloatTensor`
```

while others have 
```
`(max_length-input_ids.shape[-1],)`-shaped tuple of `torch.FloatTensor`
```

depending on whether the model is an encoder-decoder or a decoder-only (respectively)
______________________

I see two minor problems with the current docstrings:
1. Generation may stop before we generate `max_length` tokens (or `max_new_tokens` new tokens);
2. We are pushing away from `max_length` towards `max_new_tokens`. 

As such, it would be nice to improve the docs to address these two issues! Since the previous sentence in the docstring contains `(...) at each generation step`, perhaps something like this:

```
Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element per generation step),
```

The complete docstring would be:

```
scores (`tuple(torch.FloatTensor)` *optional*, returned when `output_scores=True` is passed or when `config.output_scores=True`):
            Processed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax)
            at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element per 
            generation step), with each tensor of shape `(batch_size, config.vocab_size)`).
```

WDYT?

@gante Looks good to me, as long as we keep `batch_size*num_return_sequences` instead of `batch_size` wherever it applies.

Very much agree with @gante here!

Assigned to me to update the docstring for all three frameworks 

@ClementRomac Hi! I have been trying to calculate the probability of a sequence but am not sure how to do it. As you mentioned calculating the probability, can you please tell me how to do it?

I have the scores for each step of the generate method, but not sure how to use them.

What I am doing is, given a premise and a hypothesis, I am trying to identify whether they are entailment, contradiction, or, neutral. I am getting the classification correctly, I just don't know how to calculate the probability of the sequence being **entailment**

```python
def is_entailment(premise, hypothesis):
    entailment_premise = premise
    entailment_hypothesis = hypothesis

    token_output = tokenizer("mnli premise: " + entailment_premise + " hypothesis: " + entailment_hypothesis,
                             return_tensors="pt", return_length=True)
    input_ids = token_output.input_ids

    output = model.generate(input_ids, output_scores=True, return_dict_in_generate=True, max_length=50)
    entailment_ids = output["sequences"]

    entailment = tokenizer.decode(entailment_ids[0], skip_special_tokens=True)
    return entailment
```