Consistently getting lower log probabilities for more probable sequences
Author: hxiaoyang
Pull request: no

### System Info

- `transformers` version: 4.24.0
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.15
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.12.1+cu113 (True)
- Tensorflow version (GPU?): 2.9.2 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@patrickvonplaten @gante @nars

### Information

- [X] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I have been using OPT models (125m~66b) to score sequences (see #20008) from a set of multiple choice problems (8 choices). I found that, on average, the wrong sequences are ~10% (slightly below 1/8) likely to have the maximum probability (among the 8 choices), but the correct sequence is ~1% likely to have the maximum probability. GPT-3 displays the exact opposite (expected) behaviour. My current hypotheses are
1. I calculated log probabilities incorrectly; see `score` function below. Although I think we figured it out here #20008, with the help of @gante.
2. `model(input).logits` might have returned the wrong thing.

To reproduce the error (not on my specific task but in general):
1. Define
```python
model = AutoModelForCausalLM.from_pretrained("facebook/opt-6.7b",
                                             device_map='auto')
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-6.7b", use_fast=False)

def score(prompt):
    input_ids = tokenizer(prompt, return_tensors="pt").input_ids
    input_tokens = [tokenizer.decode(id) for id in input_ids[0]]
    input_logprobs = []
    logits = model(input_ids).logits
    all_tokens_logprobs = log_softmax(logits.double(), dim=2)
    for k in range(input_ids.shape[1]):
        input_logprobs.append(all_tokens_logprobs[:,k,input_ids[0,k]])
    input_logprobs = [input_logprobs[k].detach().numpy()[0] for k in range(len(input_logprobs))]
    return input_tokens, input_logprobs

def display(prompt):
    input_tokens, input_logprobs = score(prompt)
    out_str = ""
    for i in range(len(input_logprobs)):
        out_str = out_str + str(input_tokens[i]) + ": " + str(input_logprobs[i]) + " "
    print(out_str)
```

2. Run `display` on pairs of prompts and compare probabilities. For example:

- Run `display("he works at the university as a professor")`
  - Output: `

: -9.622528113218403 he: -3.4190597141055896  works: -11.342809676330889  at: -7.323995369804066  the: -7.797857269329157  university: -8.355341028220346  as: -8.410616662546053  a: -7.980906483021017  professor: -10.740861657554035 `
- Run `display("he works at the university as a clown")`
  - Output: `
: -9.584896765616625 he: -3.4396850232969243  works: -11.341423123411294  at: -7.338279654205086  the: -7.808466620856404  university: -8.343803691603435  as: -8.401746292912188  a: -7.94869412338276  clown: -6.500715861445045`
- Note that the `LogProb("professor" | "he works at the university as a") < LogProb("clown" | "he works at the university as a")`.

3. Additional examples:
- `display("he could not play due to injury")`
  - `
: -9.609577798790037 he: -3.4115336573055703  could: -9.250618499206704  not: -6.846035151696857  play: -9.919994452618948  due: -9.421663548632559  to: -8.411191754931467  injury: -9.714442198832101 `
- `display("he could not play due to apple")`
  - `
: -9.595368031203268 he: -3.402465835779653  could: -9.260463627711584  not: -6.82763851395037  play: -9.899493418563107  due: -9.375417731887582  to: -8.429231628522128  apple: -7.371283146690385 
`
- `display("Q: who makes the iphone? A: Apple")`
  -  `
: -9.735390750532165 Q: -4.982232076966033 :: -11.777887545688749  who: -9.614314712501576  makes: -13.181075048715304  the: -7.544189630785205  : -4.4103913220035835 iph: -14.036189109433794 one: -12.96041481602171 ?: -12.274218276071288  A: -10.156031264618123 :: -12.576397941389803  Apple: -9.243583670812342 `
- `display("Q: who makes the iphone? A: Banana")`
  - `
: -9.749204391127897 Q: -4.9924473604492725 :: -11.771810712752153  who: -9.594715884838685  makes: -13.212738897899317  the: -7.47622315378366  : -4.443921992316447 iph: -14.110338198583378 one: -12.932230614752198 ?: -12.295626501904344  A: -10.15586013628168 :: -12.57128175779327  Banana: -7.862753006007329 `

### Expected behavior

Should be getting higher log probabilities for more probable sequences. For example, it should be the case that `LogProb("professor" | "he works at the university as a") > LogProb("clown" | "he works at the university as a")`.

There also appears to be very few tokens with a high (close to 0) probability. It is also weird to me that the first token (after `
`) seems to have a very high probability.

Basically the model displays the exact opposite of the expected behaviors. This has truly been puzzling. Any input is appreciated!

Comments:
Hey @xiaoyangnickhu 👋 

I believe your snippet has an issue with indexing when appending to `input_logprobs`. `all_tokens_logprobs` contains the log probs for the NEXT token, not for the current token. i.e. accessing `all_tokens_logprobs[batch_idx, 0, token]` has the log probs for the token at position `1`, not for the token at position `0`.

Factoring that in, you get the outcomes that you are expecting. For instance, in the first example, `professor` is the most likely token for the last slot 🤗 

_______________________________________________
Snippet with the indexing correction:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from torch.nn.functional import log_softmax

model = AutoModelForCausalLM.from_pretrained("facebook/opt-6.7b",
                                             device_map='auto')
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-6.7b", use_fast=False)

def score(prompt):
    input_ids = tokenizer(prompt, return_tensors="pt").input_ids
    input_tokens = [tokenizer.decode(id) for id in input_ids[0]]
    input_logprobs = []
    logits = model(input_ids).logits
    all_tokens_logprobs = log_softmax(logits.double(), dim=2)
    for k in range(1, input_ids.shape[1]):
        input_logprobs.append(all_tokens_logprobs[:, k-1, input_ids[0,k]])
    input_logprobs = [input_logprobs[k].detach().numpy()[0] for k in range(len(input_logprobs))]
    return input_tokens, input_logprobs

def display(prompt):
    input_tokens, input_logprobs = score(prompt)
    out_str = ""
    for i in range(len(input_logprobs)):
        out_str = out_str + str(input_tokens[i+1]) + ": " + str(input_logprobs[i]) + " "
    print(out_str)
```

The fix works. Thanks!