Different PTB perplexity in transformers 4.31 vs 4.32
Author: tsengalb99
Pull request: no

### System Info

python: 3.10
transformers 4.31 and 4.32
datasets: 2.14.6
pytorch 2.1.0 with cuda 11.8

OS: ubuntu 20.04
GPU: Various NVIDIA GPUs
CPU: Various Intel CPUs 

I can reproduce this issue on multiple machines with different hardware configs.


### Who can help?

@ArthurZucker 
@younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am trying to run inference on llama2-70b on the PTB dataset using GPTQ's evaluation sampling method. Using transfomers version 4.31 and before gives a perplexity of 19.55. Using transformers 4.32 and later gives a perplexity of 22.67. The two perplexities should be the same, i.e. the transformers version used should not affect results. I tried multiple datasets versions on both 4.31 and 4.32 and the datasets version used does not appear to affect perplexity. Below is sample code you can use to reproduce this issue. This issue does not happen on Wikitext2 and C4, two other datasets that I tested.

```python
import torch
import datasets
import transformers
from transformers import LlamaTokenizer, LlamaForCausalLM
import random

torch.set_grad_enabled(False)
torch.manual_seed(0)
random.seed(0)
seqlen = 4096

def get_ptb(model):
    from datasets import load_dataset
    testdata = load_dataset('ptb_text_only', 'penn_treebank', split='test')
    from transformers import AutoTokenizer
    tokenizer = AutoTokenizer.from_pretrained(model, use_fast=False)
    testenc = tokenizer(" ".join(testdata['sentence']), return_tensors='pt')
    return testenc

model_str = 'meta-llama/Llama-2-70b-hf'
model = transformers.LlamaForCausalLM.from_pretrained(model_str,
                                                      torch_dtype='auto',
                                                      low_cpu_mem_usage=True,
                                                      device_map='auto').half()
input_tok = get_ptb(model_str)['input_ids']
nsamples = input_tok.numel() // seqlen
input_tok = input_tok[0, :(seqlen * nsamples)].view(nsamples, seqlen)
print(input_tok.shape)

loss_fct = torch.nn.CrossEntropyLoss().cuda()
acc_loss = 0.0
for ii in range(nsamples):
    input = input_tok[ii, :].cuda().view(1, -1)
    output = model(input,
                   use_cache=False,
                   output_hidden_states=False,
                   output_attentions=False)[0]
    shift_logits = output[:, :-1, :].contiguous()
    shift_labels = input[:, 1:]
    loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
    acc_loss += loss.item()

avg_loss = acc_loss / nsamples

ppl = torch.exp(torch.tensor(avg_loss)).item()
print(f'perplexity: {ppl}')
```

### Expected behavior

The PTB perplexities should be the same between transformers versions.

Comments:
Interesting issue. 
Do you see differences in padding and masks before the forward? Or it only shows in the logits?
Is the issue still present with `seqlen < 4096`? like 1024?


Hey 🤗 ! Would recommend you to upgrade to the latest version, even if there is a bug and we fix it, we can't really do anything about the previous version so it's not really relevant (only to know where it started diverging). Could be as simple as a tokenizer issue that was later on fixed 😉 

I am not seeing differences in the input tokens. I am not feeding in any masks, just raw tokens. I have not tried smaller sequences. I do not have time right now to debug this issue but the attached example code should reproduce the issue. You can use smaller models since 70b fp16 may be hard to run.

> Would recommend you to upgrade to the latest version, even if there is a bug and we fix it, we can't really do anything about the previous version so it's not really relevant (only to know where it started diverging).

At least two important papers (QuIP, GPTQ) use PTB with the transformers library in their evaluations of quantized models. It may be that the old version was correct and some change in the new version broke something. People should know which version is correct, and perhaps that version is the old version. This is especially important since quantization is an active research area and someone may write a paper comparing their numbers on the old transformers to a baseline on the new transformers where their method would do better from the "bug," or vice versa and mistakenly believe their method does worse.

Understood. 🤗 I don't have the bandwidth to dive deep into this, from the releases differences, there does not seems to be any changes done to the code, but rather to the slow tokenizers, which should not affect the fast one. 
Other changes could come from configuration updates made directly on the hub like: https://huggingface.co/meta-llama/Llama-2-70b-hf/commit/6aa89cf376ffeaf8b7cf6fa5d744a87e340445a8 or the max length argument or the padding token id. The `self.config.pretraining_tp` argument also affects the logits and was set to `8` in the config at a certain point. 


You mentioned that this does not happen for all datasets, and only for two of them so it's a bit hard to guess what can be wrong here! The community might be able to help so leaving this open 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.