Perplexity number of wikitext-103 on gpt-2 don't match the paper
Author: Akhila-Yerukola
Pull request: no
Labels: name: Discussion

name: wontfix

Hi,
The reported perplexity number of gpt-2 (117M) on wikitext-103 is 37.5.

However when I use the pre-trained tokenizer for gpt-2 `GPT2Tokenizer` using:
`tokenizer = GPT2Tokenizer.from_pretrained('gpt2')` to tokenize wikitext-103, and then evaluate it using the pre-trained 117M gpt-2 model, I get a ppl of 48.4

Note: I have added newlines instead of EOS tags at the end of each line read. I've also normalized the loss by the number of tokens originally in wikitext-103 as mentioned by Alec Radford at https://github.com/openai/gpt-2/issues/78 

Could you please let me know whats wrong here?

Comments:
Are you using context length 256 by chance?

On WikiText-2 I see

I see 29.17378 using context length 1024 and 43.3393707 using context length 256
This is close to OpenAI report result of  29.41
(although I don't get why it doesn't match it exactly)


@yaroslavvb 
No, I'm using a context length of 1024. 

Although, when I evaluate it on wikitext-2, my numbers dont match with yours. Would it be possible for you to share what you've done? I believe that you might have normalized the loss by the number of tokens after the gpt-2 tokenization, and not by the number of tokens originally in wikitext-2. 

Here's our loss [computation](https://github.com/cybertronai/bflm/blob/b6ba6d97c9ccdf2b12e104fbdcd0bed25ada7b68/train_gpt2.py#L163)

We don't normalize the loss here, just take the average output of `model(batch, lm_labels=batch)` where model is `GPT2LMHeadModel.from_pretrained(args.model_name_or_path)`

Note wikitext-103 and wikitext-2 have the same test set, so you should get `29.17378` perplexity on either one

@Akhila-Yerukola when we tried normalizing by the original token count, our ppl number got much further from the paper.

I ran `wikitext-2-raw/wiki.train.raw` through the encoder and got 2417793 tokens. Then I split the original on spaces and got 2088678. 
```py
math.exp(math.log(29.41) * 2088678 / 2417793) # 18.56
```
We got 30.4 ppl unnormalized and 28 after [detokenizing](https://github.com/cybertronai/bflm/blob/master/detokenizer.py), so we're still quite far from 18.56.
Another possibility is that for evaluation, instead of chunking the dataset and shoving it through, they pass it in 1 token at a time the same way they do for sample generation. This would significantly reduce the loss because the model wouldn't "forget" what it was doing at the beginning of each sequence. I'll try this later today.

As a quick test, I calculated loss based on only the last 10 tokens of the sequence without changing the chunking. I got test loss of 23.9 and train 24.9.

@yaroslavvb @8enmann 
I still think the numbers don't match the paper.
The Table 3 in the paper which contains Zero-shot results on many datasets has a "SOTA" row which correspond to the reported perplexity numbers on the test sets of the data sets. (I verified this for wikitext-2 and wikitext-103)
When I run [this](https://github.com/cybertronai/bflm/blob/b6ba6d97c9ccdf2b12e104fbdcd0bed25ada7b68/train_gpt2.py) evaluation script, I get ppl of 23.85 on wikitext-2 (not raw), 28.186 on wikitext-103 (not raw),  29.01 on wikitext-103-raw (and wikitext-2-raw). None of these match the reported 117M gpt-2 model (which is the model available) numbers from the paper (29.41 for wikitext-2, 37.5 for wikitext-103)
From the looks of it wikitext-103 is way off.

You're right, our results are very different from the paper. Perhaps they used the detokenized non-raw version and also for the labels passed -1 for <UNK> tokens, which causes them not to be evaluated for loss. This, combined with the 1 at a time trick I mentioned 2 days ago, could bring PPL down to ~18. However that still leaves a big puzzle as to why the loss they reported on wikitext-103 is so much worse than wikitext-2.

Related issue, Lambada numbers also don't match the paper (31% instead of reported 46%) https://github.com/huggingface/pytorch-pretrained-BERT/issues/491

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> @yaroslavvb @8enmann
> When I run [this](https://github.com/cybertronai/bflm/blob/b6ba6d97c9ccdf2b12e104fbdcd0bed25ada7b68/train_gpt2.py) evaluation script, I get ppl of 23.85 on wikitext-2 (not raw), 28.186 on wikitext-103 (not raw), 29.01 on wikitext-103-raw (and wikitext-2-raw). None of these match the reported 117M gpt-2 model (which is the model available) numbers from the paper (29.41 for wikitext-2, 37.5 for wikitext-103)

Is it possible to know why 0.7 and 0.3? --> 0.7 * exp_average_loss + 0.3 * loss.item()


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This discussion is old, so this may not be applicable anymore. But I'd like to offer a data point if it's still the case. I tried gpt2 and gpt2-medium on OpenWebText (tokenized with HuggingFace's corresponding tokenizer settings), and I got the ppl about 24 and 18, respectively, whereas the openai version of them is 17 and 13, respectively. This is good enough to say that I probably didn't make any catastrophic mistake, but there still is some gap, which may or may not explain the performance gap on other datasets.