Inconsistant output between pytorch-transformers and pytorch-pretrained-bert
Author: Nealcly
Pull request: no
Labels: name: wontfix

## 📚 Migration

<!-- Important information -->

Model I am using (GPT, GPT2, XLNet):

Language I am using the model on (English):

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)
```
def xlnet_score(text, model, tokenizer):
    # Tokenized input
    tokenized_text = tokenizer.tokenize(text)
    # text = "[CLS] Stir the mixture until it is done [SEP]"
    sentence_prob = 0
    #Sprint(len(tokenized_text))
    for masked_index in range(0,len(tokenized_text)):
        # Mask a token that we will try to predict back with `BertForMaskedLM`
        masked_word = tokenized_text[masked_index]
        if masked_word!= "<sep>":
            masked_word = tokenized_text[masked_index]
            tokenized_text[masked_index] = '<mask>'
            # assert tokenized_text == ['[CLS]', 'who', 'was', 'jim', 'henson', '?', '[SEP]', 'jim', '[MASK]', 'was', 'a', 'puppet', '##eer', '[SEP]']
            # print (tokenized_text)
            input_ids = torch.tensor(tokenizer.convert_tokens_to_ids(tokenized_text)).unsqueeze(0)
            index = torch.tensor(tokenizer.convert_tokens_to_ids(masked_word))

            perm_mask = torch.zeros((1, input_ids.shape[1], input_ids.shape[1]), dtype=torch.float)
            perm_mask[:, :, masked_index] = 1.0  # Previous tokens don't see last token
            target_mapping = torch.zeros((1, 1, input_ids.shape[1]), dtype=torch.float)  # Shape [1, 1, seq_length] => let's predict one token
            target_mapping[0, 0, masked_index] = 1.0  # Our first (and only) prediction will be the last token of the sequence (the masked token)

            input_ids = input_ids.to('cuda')
            perm_mask = perm_mask.to('cuda')
            target_mapping = target_mapping.to('cuda')
            index = index.to('cuda')

            with torch.no_grad():
                outputs = model(input_ids, perm_mask=perm_mask, target_mapping=target_mapping, labels = index)
            next_token_logits = outputs[0]
            length = len(tokenized_text)
            # predict_list = predictions[0, masked_index]
            sentence_prob -= next_token_logits.item()
            tokenized_text[masked_index] = masked_word
        #tokenized_text = tokenized_text.split()
    #return math.pow(sentence_prob, 1/(len(tokenized_text)-3))
    return sentence_prob/(length-1)

def gpt_score(text, model, tokenizer):
    # Tokenized input
    # text = "[CLS] I got restricted because Tom reported my reply [SEP]"
    input_ids = torch.tensor(tokenizer.encode(text)).unsqueeze(0)  # Batch size 1
    input_ids = input_ids.to('cuda')
    with torch.no_grad():
        outputs = model(input_ids, labels=input_ids)
    loss, logits = outputs[:2]
    # text = "[CLS] Stir the mixture until it is done [SEP]"
    sentence_prob = -loss.item()
    #return math.pow(sentence_prob, 1/(len(tokenized_text)-3))
    return sentence_prob

def gpt2_score(text, model, tokenizer):
    # Tokenized input
    # text = "[CLS] I got restricted because Tom reported my reply [SEP]"
    input_ids = torch.tensor(tokenizer.encode(text)).unsqueeze(0)  # Batch size 1
    input_ids = input_ids.to('cuda')
    with torch.no_grad():
        outputs = model(input_ids, labels=input_ids)
    loss, logits = outputs[:2]
    # text = "[CLS] Stir the mixture until it is done [SEP]"
    sentence_prob = -loss.item()
    #return math.pow(sentence_prob, 1/(len(tokenized_text)-3))
    return sentence_prob

def score(sentence):
    tokenize_input = tokenizer.tokenize(sentence)
    tensor_input = torch.tensor([tokenizer.convert_tokens_to_ids(tokenize_input)])
    tensor_input = tensor_input.to('cuda')
    loss=model(tensor_input, labels=tensor_input)[0]
    return math.exp(loss)


config = XLNetConfig.from_pretrained('xlnet-base-cased')
tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')
model = XLNetLMHeadModel(config)
model.to('cuda')
model.eval()
a=['there is a book on the desk',
                'there is a plane on the desk',
                        'there is a book under the desk']
print([xlnet_score(i,model,tokenizer) for i in a])

config = OpenAIGPTConfig.from_pretrained('openai-gpt')
tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt')
model = OpenAIGPTLMHeadModel(config)
model.to('cuda')
model.eval()

print([gpt_score(i,model,tokenizer) for i in a])
print([score(i) for i in a])

config = GPT2Config.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel(config)
model.to('cuda')
model.eval()

print([gpt_score(i,model,tokenizer) for i in a])`
```
The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details)

Details of the issue:

<!-- A clear and concise description of the migration issue. If you have code snippets, please provide it here as well. -->
So the issue here is that I try to calculate the perplexity (or loss) of the sentence to determine which sentence makes more sense. However, as #473 shows that we could just retrieve the loss. The scores I get with PyTorch-transformer is different from the scores in that post. For the `def score` function, I literally copy the code in post #473 for comparison.

```
a=['there is a book on the desk',
                'there is a plane on the desk',
                        'there is a book under the desk']
print([model_score(i,model,tokenizer) for i in a])

negative of loss get from XLnet
[-11.915737946828207, -11.859564940134684, -11.996480623881022]
negative of loss get from GPT
[-10.969852447509766, -11.002564430236816, -10.877273559570312]
perplexity get from GPT
[58096.0205576014, 60027.88181824669, 52959.01330928259]
negative of loss get from GPT-2
[-11.469226837158203, -11.445046424865723, -11.510353088378906]
```
Furthermore, as you can see, none of these results above make much sense. 
## Environment

* OS: Linux
* Python version: 3.6
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch):
* Using GPU ? Yes
* Distributed of parallel setup ? No
* Any other relevant information:

## Checklist

- [x] I have read the migration guide in the readme.
- [x] I checked if a related official extension example runs on my machine.

## Additional context

<!-- Add any other context about the problem here. -->

Comments:
See  #954.  I got bitten by the same documentation _bug_.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.