Larger Logits != Larger Probability
Author: Hannibal046
Pull request: no
Labels: name: bug

### System Info

transformers version: 4.22.0.dev0
Platform: Linux-5.8.0-51-generic-x86_64-with-glibc2.10
Python version: 3.8.13
Huggingface_hub version: 0.8.1
PyTorch version (GPU?): 1.12.1+cu113 (True)
Tensorflow version (GPU?): not installed (NA)
Flax version (CPU?/GPU?/TPU?): not installed (NA)
Jax version: not installed
JaxLib version: not installed
Using GPU in script?:
Using distributed or parallel set-up in script?:

### Who can help?

@gante @patrickvonplaten 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import BartTokenizer,BartForConditionalGeneration
model_path = "/data/pretrained_model/bart_base"
toker = BartTokenizer.from_pretrained(model_path)
model = BartForConditionalGeneration.from_pretrained(model_path)

input_tokens = ["what do you think it ? huggingface is a great library. And I enjoy it very much",
                "transformers is so good"]
batch_size = 2
num_beams = 10
max_length = 5
num_return_sequences = 5

input_ids = toker(input_tokens,return_tensors='pt',padding=True).input_ids
output = model.generate(input_ids,max_length=max_length,num_beams=num_beams,num_return_sequences=num_return_sequences,
                        return_dict_in_generate=True,output_scores=True)


def get_logits_and_probs(output,num_return_sequence,batch_size,eos_token_id):
    """
    using for-loop to get positional-wise logits and probability
    """
    import torch
    total = num_return_sequence * batch_size
    token_logits = [[] for _ in range(total)]
    token_probs = [[] for _ in range(total)]
    continue_or_not = [True for _ in range(total)]
    for time_step in range(len(output.scores)):
        cur_scores = output.scores[time_step] ## num_beam,vocab_size
        for idx in range(total):
            cur_beam = output.beam_indices[idx][time_step]
            cur_token = output.sequences[idx][time_step+1] ## decoder_start_token_id
            if continue_or_not[idx]:
                token_probs[idx].append(torch.softmax(cur_scores[cur_beam],dim=-1)[cur_token].item())
                token_logits[idx].append(cur_scores[cur_beam][cur_token].item())
            if cur_token==eos_token_id:
                continue_or_not[idx]=False
    return token_logits,token_probs
token_logits,token_probs = get_logits_and_probs(output,num_return_sequences,batch_size,toker.eos_token_id)


def avg(ls):
    return sum(ls)/len(ls)

## check if my get_logits_and_probs function is correct by compare it with output.sequences_scores
for idx in range(num_return_sequences*batch_size):
    if idx == num_return_sequences:
        print("*"*20)
    print(avg(token_logits[idx]),output.sequences_scores[idx].item())
print("probability")
for idx in range(num_return_sequences*batch_size):
    if idx == num_return_sequences:
        print("*"*20)
    print(avg(token_probs[idx]))
```


![](https://global.discourse-cdn.com/standard14/uploads/hellohellohello/original/2X/9/9676daaecb6ba71cee7f47807b9d786014f03100.jpeg)

![](https://global.discourse-cdn.com/standard14/uploads/hellohellohello/original/2X/0/0cd3664135fdfa06393cf35123c01e17acee0e83.jpeg)


### Expected behavior

I find that the order given by beam search is determined by `sum(logits)` rather than `sum(probability)`. I am not sure if it is correct, intuitively, the probability is a relative value that is comparable between tokens generated from different time step,  but logits are not. 

the example above shows that the 5th sequence given by beam search actually has higher probability than 2nd sentence. 

Comments:
Hi @Hannibal046 👋 

The sentences are sorted by the sum of the scores (i.e. logits with potential modifiers on top), as you wrote. However, if you want to map it back to probabilities, the operator to apply is the product, not the sum :) The sum of logarithms corresponds to the logarithm of the product. Applying the product instead, you'll see the examples are correctly sorted.

For more info, see [our blog post](https://huggingface.co/blog/how-to-generate)

___________________________

Since the original question is solved, I'm closing the issue. Feel free to reopen if you find related bugs!