Unexpected generated token probabilities derived from scores
Author: hacobe
Pull request: no

## Environment info

- `transformers` version: 4.11.3
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyTorch version (GPU?): 1.9.0+cu111 (False)
- Tensorflow version (GPU?): 2.6.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

@patrickvonplaten

## Information

I'm generating tokens from the GPT2 model. I'm trying to get the the log probability of each generated token conditional on the previous generated tokens, i.e., the same numbers generated by [logprobs](https://beta.openai.com/docs/api-reference/completions/create#completions/create-logprobs) in the OpenAI API for the sampled tokens.

Here is the code I'm using to get the "scores" for the generation:

```python
from transformers import AutoModelForCausalLM
from transformers import AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("gpt2", return_dict_in_generate=True)
tokenizer = AutoTokenizer.from_pretrained("gpt2")

input_ids = tokenizer("Today is a nice day", return_tensors="pt").input_ids

generated_outputs = model.generate(input_ids, num_beams=2, max_length=8, output_scores=True)
```

The documentation for scores for BeamSearchDecoderOnlyOutput is the following:
"scores (tuple(torch.FloatTensor) optional, returned when output_scores=True is passed or when config.output_scores=True) – Processed beam scores for each vocabulary token at each generation step. Beam scores consisting of log softmax scores for each vocabulary token and sum of log softmax of previously generated tokens in this beam . (max_length-input_ids.shape[-1],)-shaped tuple of torch.FloatTensor with each tensor of shape (batch_size * num_beams * num_return_sequences, config.vocab_size))." (https://huggingface.co/transformers/internal/generation_utils.html#beamsearchoutput)

And there is a discussion post that says the following about scores:
"So in the case of beam_search, the scores correspond to the log probability of all words + the log probability all previous scores in your beam. So regarding the image this means that scores[0][0] will correspond to the log probabilities of all possible words in the vocabulary, so assuming your vocab would only consist of dog, nice, car and the probs are the same as in the diagram, the values would correspond to log(0.4), log(0.5), log(0.1). Scores[1][0] then corresponds to the chosen word of time step one (e.g. dog) and all possible values again, se: log(0.4) + log(0.05), log(0.4) + log(0.05), log(0.4) + log(0.9) using the diagram above again." (https://discuss.huggingface.co/t/showing-individual-token-and-corresponding-score-during-beam-search/3735/3)

Using that documentation, I try to get the probabilities, but end up with probabilities outside of [0, 1]:

```python
import numpy as np

gen_ids = generated_outputs["sequences"][0, input_ids.shape[-1]:]

logprob_beam0_gen_token0 = generated_outputs["scores"][0][0, gen_ids[0]].tolist()
logprob_beam0_gen_token1 = generated_outputs["scores"][1][0, gen_ids[1]].tolist() - logprob_beam0_gen_token0
logprob_beam0_gen_token2 = generated_outputs["scores"][2][0, gen_ids[2]].tolist() - logprob_beam0_gen_token0 - logprob_beam0_gen_token1

probs = np.exp([logprob_beam0_gen_token0, logprob_beam0_gen_token1, logprob_beam0_gen_token2])

# (0.15271044542780696, 2.6683572850297056e-05, 824.6099669421758)
probs[0], probs[1], probs[2]
```

## To reproduce

See this colab:
https://colab.research.google.com/drive/11rRAFuNycLLDiDDwU02mBgXjpBOXCe4P#scrollTo=WwIzz1CQYfgU

## Expected behavior

I would expect the probabilities to be between 0 and 1.

Comments:
I use a code snippet to explain this, feel free to tell me if I'm wrong:

```
import torch

from transformers import AutoModelForCausalLM
from transformers import AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("gpt2", return_dict_in_generate=True)
tokenizer = AutoTokenizer.from_pretrained("gpt2")

input_ids = tokenizer("Today is a nice day", return_tensors="pt").input_ids

generated_outputs = model.generate(input_ids, num_beams=2, max_length=8, output_scores=True)
gen_ids = generated_outputs["sequences"][0, input_ids.shape[-1]:]
vocab_size = generated_outputs["scores"][0].shape[-1]


print(gen_ids) # tensor([ 11, 475, 314])

# Here we find out at each time-step, which beam each generated id belongs to.
values, indices = torch.topk(generated_outputs["scores"][0].view(-1), k=2)
print(values, (indices % vocab_size), (indices / vocab_size).long()) # tensor([-1.5148, -1.8792]) tensor([329,  11]) tensor([0, 0])
values, indices = torch.topk(generated_outputs["scores"][1].view(-1), k=2)
print(values, (indices % vocab_size), (indices / vocab_size).long()) # tensor([-3.6481, -3.9508]) tensor([475, 262]) tensor([1, 0])
values, indices = torch.topk(generated_outputs["scores"][2].view(-1), k=2)
print(values, (indices % vocab_size), (indices / vocab_size).long()) # tensor([-5.6957, -5.8748]) tensor([314, 340]) tensor([0, 0])

# So we know at time-step 1, the token id `475` belongs to beam 1, others belong to beam 0.

logprob_beam0_gen_token0 = generated_outputs["scores"][0][0, gen_ids[0]]
logprob_beam0_gen_token1 = generated_outputs["scores"][1][1, gen_ids[1]] - logprob_beam0_gen_token0
logprob_beam0_gen_token2 = generated_outputs["scores"][2][0, gen_ids[2]] - logprob_beam0_gen_token0 - logprob_beam0_gen_token1

print(logprob_beam0_gen_token0.exp(), logprob_beam0_gen_token1.exp(), logprob_beam0_gen_token2.exp()) 

# Outputs:
# tensor(0.1527) tensor(0.1705) tensor(0.1290)
```

Got it. Thanks!

See: https://discuss.huggingface.co/t/generation-probabilities-how-to-compute-probabilities-of-output-scores-for-gpt2/3175/15?u=patrickvonplaten

I'll improve the behavior for beam search

https://github.com/huggingface/transformers/pull/14654