Wrong Mask LM prediction with BertForMaskedLM
Author: AtmaHou
Pull request: no

# 📚 Migration

## Information

<!-- Important information -->

Model I am using (Bert, XLNet ...):
Bert, Electra

Language I am using the model on (English, Chinese ...):
English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## Details

### When I transfer to transformers=2.7.0, I find that the LM model failed to predict correct masked tokens.

I test the **transformer** model on the old LM example of **pytorch_pretrained_bert**:
"Who was Jim Henson ? Jim Henson was a puppeteer"

My test code goes like following:

```python
# coding: utf-8
import torch
from transformers import BertTokenizer, BertModel, BertForMaskedLM, AutoModel, AutoTokenizer, AutoModelWithLMHead, ElectraModel, ElectraForMaskedLM



MODEL_PATH = 'Resources/bert-base-uncased/uncased_L-12_H-768_A-12/'

VOCAB = MODEL_PATH

print('== tokenizing ===')
tokenizer = BertTokenizer.from_pretrained(VOCAB)

# Tokenized input
text = "Who was Jim Henson ? Jim Henson was a puppeteer"
tokenized_text = tokenizer.tokenize(text)

masked_index = 6
tokenized_text[masked_index] = '[MASK]'


# Convert token to vocabulary indices
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
segments_ids = [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]
input_mask = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]

# ======== Convert inputs to PyTorch tensors
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])

# ======== predict tokens ========
print('== LM predicting ===')
# Load pre-trained model (weights)
model = BertForMaskedLM.from_pretrained(MODEL_PATH)
model.eval()

# Predict all tokens
predictions = model(tokens_tensor, segments_tensors)[0]

# confirm we were able to predict 'henson'
predicted_index = torch.argmax(predictions[0, masked_index]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])
print('predicted_token', predicted_token)

```



## Other Details
(1) Such testing code works fine with the previous version of **pytorch_pretrained_bert**. 
But now it seems that model predicts a random token. 

(2) Random predicting also happened when I load electra model with ElectraForMaskedLM.



<!-- A clear and concise description of the migration issue.
    If you have code snippets, please provide it here as well.
    Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
    Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.
    -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
 
- `transformers` version: 2.7.0
- Platform: linux
- Python version: 3.7
- PyTorch version (GPU?): 1.2-gpu
- Tensorflow version (GPU?): no
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

<!-- IMPORTANT: which version of the former library do you use? -->
* `pytorch-transformers` or `pytorch-pretrained-bert` version (or branch):
pytorch-pretrained-bert

## Checklist

- [x] I have read the migration guide in the readme.
 ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers);
  [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [x] I checked if a related official extension example runs on my machine.


Comments:
This is probably because you're not using special tokens at all. When using BERT you should add the `[CLS]` and `[SEP]` tokens at the appropriate places. Modifying your code to include these generates the correct answer:

```py
# Tokenized input
text = "Who was Jim Henson ? Jim Henson was a puppeteer"
tokenized_text = tokenizer.tokenize(text)

masked_index = 7  # <-- the masked index needs to be offset by 1 because a [CLS] token will be added at the beginning
tokenized_text[masked_index] = '[MASK]'

# Convert token to vocabulary indices
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
indexed_tokens = tokenizer.build_inputs_with_special_tokens(indexed_tokens)  # <-- should add special tokens, this method does it


segments_ids = [0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]  # <-- modify this to include special tokens
input_mask = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]  # <-- modify this to include special tokens

# ======== Convert inputs to PyTorch tensors
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])

# ======== predict tokens ========
print('== LM predicting ===')
# Load pre-trained model (weights)
model = BertForMaskedLM.from_pretrained(MODEL_PATH)
model.eval()

# Predict all tokens
predictions = model(tokens_tensor, segments_tensors)[0]

# confirm we were able to predict 'henson'
predicted_index = torch.argmax(predictions[0, masked_index]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])
print('predicted_token', predicted_token)
```

Result:

```
== tokenizing ===
== LM predicting ===
predicted_token ['henson']
```


Please note that there is a much simpler way of doing what you did, by using the `encode` method which automatically manage the special tokens. The `encode_plus` method manages the attention mask and segment IDs as well. Here's the full code using the `encode_plus` method:

```py
import torch
from transformers import BertTokenizer, BertModel, BertForMaskedLM, AutoModel, AutoTokenizer, AutoModelWithLMHead, ElectraModel, ElectraForMaskedLM

MODEL_PATH = 'bert-base-uncased'

VOCAB = MODEL_PATH

print('== tokenizing ===')
tokenizer = BertTokenizer.from_pretrained(VOCAB)

# Tokenized input
text = "Who was Jim Henson ? Jim [MASK] was a puppeteer"
inputs = tokenizer.encode_plus(text, return_tensors="pt")

masked_index = 7

model = BertForMaskedLM.from_pretrained(MODEL_PATH)
model.eval()

print('== LM predicting ===')
# Predict all tokens
predictions = model(**inputs)[0]

# confirm we were able to predict 'henson'
predicted_index = torch.argmax(predictions[0, masked_index]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])
print('predicted_token', predicted_token)

```

Thanks a lot for the answer! I can move on my projects. 
It is still kind of weird that the old code works correctly without '[CLS]' and '[SEP]'. 
Has some underlying code logic changed?

It's weird, I agree! This is the correct way to do it, and that's the way it should have been done in the previous versions as well, though. Glad you could get your code working!