Bug on compute_transition_scores, inconsistency between two ways of evaluating probabilities
Author: hongzhoulin89
Pull request: no

### System Info


Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.30.2
- Platform: Linux-5.15.109+-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.11 (gpu)
- Jax version: 0.4.13
- JaxLib version: 0.4.13
- Using GPU in script?: y
- Using distributed or parallel set-up in script?: n

### Who can help?

@gante 


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Colab Link: https://colab.research.google.com/drive/1ldLuHr2h4nSlTv5TGO06iORByJJNjCJ6?usp=sharing

```python
import torch
import sentencepiece
import accelerate
import transformers
from transformers import GenerationConfig, LlamaForCausalLM, LlamaTokenizer

if torch.cuda.is_available():
    num_gpus = torch.cuda.device_count()
    device = "cuda"
else:
    device = "cpu"
print(device)

model_path ="openlm-research/open_llama_3b"
tokenizer = LlamaTokenizer.from_pretrained(model_path)
model = LlamaForCausalLM.from_pretrained(model_path, device_map='auto')

input_text = 'Hello, I am frustrated'

n_seq = 1
max_new_tokens = 5
model.eval()

input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to(device)
print(input_ids)
output_sample = model.generate(input_ids=input_ids,   
                max_new_tokens=max_new_tokens, 
                do_sample=True, 
                top_p = 0.9, 
                top_k = 15,
                num_return_sequences=n_seq, 
                return_dict_in_generate=True, 
                output_scores=True)

# Evaluate transition scores using official method
transition_scores = model.compute_transition_scores(output_sample.sequences, output_sample.scores, normalize_logits=True)
print(transition_scores)
print('generated sequence: ', tokenizer.batch_decode(output_sample.sequences, skip_special_tokens= True)[0])

# Take the text generated and re-evaluate the probability 
text_generated = tokenizer.batch_decode(output_sample.sequences, skip_special_tokens= True)[0]
generated_input_ids = tokenizer(text_generated, return_tensors="pt").input_ids.to(device)
print(generated_input_ids)

with torch.no_grad():
  model_output = model(generated_input_ids)

# collect the probability of the generated token -- probability at index 0 corresponds to the token at index 1
probs = torch.log_softmax(model_output.logits, dim=-1).detach()
probs = probs[:, :-1, :]
generated_input_ids_shifted = generated_input_ids[:, 1:]
gen_probs = torch.gather(probs, 2, generated_input_ids_shifted[:, :, None]).squeeze(-1)
print(gen_probs[:,-max_new_tokens:])
```





### Expected behavior

I am comparing the transition matrix by using 

1.  The official implementation 
`model.compute_transition_scores(output_sample.sequences, output_sample.scores, normalize_logits=True)`

2. Another official suggestion by @gante explained in the Announcement of the probability generation 
https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075/17?u=redpig-at-imo

As they are both suggesting by Joao, I am expecting the two ways to return the exact same probability, however this is not the case, which seems to be weird to me. Am I missing anything or this is expected? 

<img width="1106" alt="Screen Shot 2023-07-12 at 8 28 09 PM" src="https://github.com/huggingface/transformers/assets/29802555/9f8d4b6f-d62a-4fb3-abe9-510c9f28b11e">

<img width="1069" alt="Screen Shot 2023-07-12 at 8 28 02 PM" src="https://github.com/huggingface/transformers/assets/29802555/ff089ce5-da5e-452a-b375-a19d3003a3d6">

Thanks in advance! 


Comments:
Usually, `model.generate` does some post-process on the logits. As you can see in your code snippet, there are arguments you passed
```python
  do_sample=True, 
  top_p = 0.9, 
  top_k = 15,
```
If we just go through the model outputs (your second way) and compute `log_softmax` on the model raw logits, this value doesn't  go through the postprocess. So the value won't be the same.

Still tag @gante to see if he has further comments.

> Usually, `model.generate` does some post-process on the logits. As you can see in your code snippet, there are arguments you passed
> 
> ```python
>   do_sample=True, 
>   top_p = 0.9, 
>   top_k = 15,
> ```
> 
> If we just go through the model outputs (your second way) and compute `log_softmax` on the model raw logits, this value doesn't go through the postprocess. So the value won't be the same.
> 
> Still tag @gante to see if he has further comments.

Thanks for the prompt response! 

It sounds like there are some postprocess steps affecting the logits/probability, I am curious what are those postprocess mechanism. Would you mind provide some more context on what the postprocess is trying to achieve, or maybe point me to the source code. 

From my understanding, the part on` do_sample, top_p, top_k ` is only affecting the sampling strategy, not the underlying probability, or maybe it re-normalize the conditional probability? Thanks!  

You might be right regarding @hongzhoulin89 . It has been sometime I haven't deal with those arguments. Let me take a look, unless our generation super expert @gante faster than me for a comment! 

@hongzhoulin89 

One place is 

https://github.com/huggingface/transformers/blob/91d7df58b6537d385e90578dac40204cb550f706/src/transformers/generation/utils.py#L2372-L2375

or 

https://github.com/huggingface/transformers/blob/91d7df58b6537d385e90578dac40204cb550f706/src/transformers/generation/utils.py#L2652-L2656

And you can check inside `generate` what `logits_warper = self._get_logits_warper(generation_config)` gives in your case.

https://github.com/huggingface/transformers/blob/91d7df58b6537d385e90578dac40204cb550f706/src/transformers/generation/utils.py#L1576-L1588

@hongzhoulin89 👋 

@ydshieh said it all -- we often (almost ways, actually) manipulate the logits after the forward pass while generating. There are many reasons to do so, and each reason may add an additional post-processing step. Here are a few examples:
- Whisper has special sequences at the beginning of the generation, to select its mode
- We might want to block certain words from being generated
- We might want to adjust the distribution to be more/less biased towards the most likely tokens

They are applied in the places @ydshieh pointed out, and you can check these further docs:
- List of possible manipulations, triggered through the config file: https://huggingface.co/docs/transformers/v4.30.0/en/main_classes/text_generation#transformers.GenerationConfig
- Implementation of the logit manipulations: https://github.com/huggingface/transformers/blob/main/src/transformers/generation/logits_process.py

Thanks a lot, this is really helpful!