GPT as a Language Model
Author: mdasadul
Pull request: no
Labels: name: wontfix

I am interested to use GPT as Language Model to assign Language modeling score (Perplexity score) of a sentence.  Here is what I am using 
```import torch
import math
from pytorch_pretrained_bert import OpenAIGPTTokenizer, OpenAIGPTModel, OpenAIGPTLMHeadModel
# Load pre-trained model (weights)
model = OpenAIGPTLMHeadModel.from_pretrained('openai-gpt')
model.eval()
# Load pre-trained model tokenizer (vocabulary)
tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt')

def score(sentence):
    tokenize_input = tokenizer.tokenize(sentence)
    tensor_input = torch.tensor([tokenizer.convert_tokens_to_ids(tokenize_input)])
    loss=model(tensor_input, lm_labels=tensor_input)
    return math.exp(loss)


a=['there is a book on the desk',
                'there is a plane on the desk',
                        'there is a book in the desk']
print([score(i) for i in a])
21.31652459381952, 61.45907380241148, 26.24923942649312
```
Is it the right way to score a sentence ?


Comments:
Looks good to me. It's perplexity so lower is better.
You can do a `math.exp(loss.item())` and call you model in a `with torch.no_grad()` context to be a little cleaner.

Oh no wait, you need to compare to the shifted inputs:
`loss=model(tensor_input[:-1], lm_labels=tensor_input[1:])`
It's a causal model, it predicts the next token given the previous ones.

I can see inside the ```class OpenAIGPTLMHeadModel(OpenAIGPTPreTrainedModel)```  this shifting is happening 
```        if lm_labels is not None:
            # Shift so that tokens < n predict n
            shift_logits = lm_logits[:, :-1].contiguous()
            shift_labels = lm_labels[:, 1:].contiguous()
```

Do I still need to use 
 ```loss=model(tensor_input[:-1], lm_labels=tensor_input[1:])```

Oh you are right, this has been added now with #404.
So the way you are doing looks fine to me.
Shifting the logics inside the model can a bit dangerous for the people who are used to train a causal model the usual way, I'll add a mention in the README.

Thanks for your quick response. I can see there is a minor bug when I am trying to predict with a sentence which has one word. You can re create the error by using my above code.
```
sentence='Learn'
score(sentence)
```

Unfortunately, given the way the model is trained (without using a token indicating the beginning of a sentence), I would say it does not make sense to try to get a score for a sentence with only one word.

How can we use this to get the probability of a particular token? So, for instance, let's say we have the following sentence. 
"He was going home"
and we want to get the probability of "home" given the context "he was going"
like in GLTR tool by harvard nlp @thomwolf 

@thomwolf If the shifting of the lm_labels matrix isn't necessary (before passing into the model's forward method) because of the internal logit shifting, should the preprocess code for finetuning GPT1 in RocStories be changed? At [https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_openai_gpt.py#L86](url), I believe the continuations are shifted over in lm_labels one relative to input_ids.

Oh yes, of course! Do you want to submit a PR on that? Otherwise I'll take of it later.

For sure! Will look at it soon.

On Thu, Apr 25, 2019 at 11:33 PM Thomas Wolf <notifications@github.com>
wrote:

> Oh yes, of course! Do you want to submit a PR on that? Otherwise I'll take
> of it later.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/pytorch-pretrained-BERT/issues/473#issuecomment-486942939>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AC6UQICJ3ROXNOJXROIKYN3PSKO4LANCNFSM4HFJZIVQ>
> .
>


@thomwolf  Hey how can I give my own checkpoint files to the model while loading


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Is this score normalized on sentence lenght? And if not, what do I need to change to normalize it? 


I'm confused whether the right way to calculate the perplexity for GPT2 is what the OP has done or as per the documentation https://huggingface.co/transformers/perplexity.html? Or  both are equivalent for some value of the stride?