Behavior of Masked-LM BERT, dependence on masked token
Author: IngoMarquart
Pull request: no

I am experimenting with the masked-LM for BERT to understand how the masking affects predictions of the other tokens.

Of course using no [MASK] is not the intended usage, nor is it to predict each token in the sentence. But my understanding is that the LM head is a separate softmax classifier, taking the final embeddings of BERT for the whole sequence as an input. Therefore the model outputs predictions for all tokens, including the masked token.

I would have expected, that when no token is masked, the prediction should be pretty much perfect. The embedding of any word is then a function of both context and its own position-adjusted initial encoding.
If a token is masked, however, BERT essentially needs to predict it from context and position.

Interestingly, I have run across a sentence 
(Donald Trump is the president of the United States), where the first word is not predicted whenever no [MASK] token is set, but is predicted correctly if a later token is masked.

Consider the sequence without any masking
`['[CLS]', 'donald', 'trump', 'is', 'the', 'president', 'of', 'the', 'united', 'states', '[SEP]']`

The output of the masked LM model is
`['.', '.', 'trump', 'is', 'the', 'president', 'of', 'the', 'united', 'states', '.']`
where the first token is missing

If we input the sequence
`['[CLS]', 'donald', 'trump', 'is', 'the', '[MASK]', 'of', 'the', 'united', 'states', '[SEP]']`

Then the output is correctly
`['.', 'donald', 'trump', 'is', 'the', 'president', 'of', 'the', 'united', 'states', '.']`

But this behavior also occurs with masking: further experimentation shows that the position of the [MASK] token determines whether the sentence is correctly predicted. If the [MASK] is early in the sequence (position 2-4 in this case), the first word is mispredicted. If it is later, after position 4, then the first token is predicted correctly.
There are other sentences, where this "problem" does not occur (for example if the sentence starts with "the").


I am trying to understand this behavior. Why does BERT fail prediction of the first non-masked token in some cases, in particular, when no token is masked and the model should have "full information"?
Am I misunderstanding the model or the implementation?


Attached is a small example based on the github readme that replicates this behavior


[lm_test.py.txt](https://github.com/huggingface/transformers/files/3735740/lm_test.py.txt)


Edit: In case you are wondering why the heck I would want to do this. I am working with a model that uses (part of the) logits from the LM head repeatedly for different positions. The corpus is fixed. So the correct way would be to run the LM each time, but if I could run BERT instead once for every sequence in the corpus and save the relevant predictions, it would save a lot of time. 


Comments:
Yes, this is nice illustration of the discrepancy between Bert's training (in which masked tokens are provided) and Bert's testing (in which no masked token is provided).

I've noticed that this also frequently occurs when the last token in the sentence is masked. 

For example,
`['[CLS]', 'donald', 'trump', 'is', 'the', 'president', 'of', 'the', 'united', '[MASK]', '[SEP]']`
is predicted as
`['.', 'donald', 'trump', 'is', 'the', 'president', 'of', 'the', 'united', '.', '.']`

But if we mask a middle token, anything besides the last, then it works well:
`['[CLS]', 'donald', 'trump', 'is', 'the', 'president', 'of', 'the', '[MASK]', 'states', '[SEP]']`
is predicted as
`['.', 'donald', 'trump', 'is', 'the', 'president', 'of', 'the', 'united', 'states', '.']`

@thomwolf Any insight on why this is the case?


We are also interested to work with the prediction of each token (not just the masked ones) and are wondering what is happening there. 

As you closed this issue, @IngoMarquart , have you found an explanation, or some link, which sheds more light on this?

Hey guys, 
I would like to investigate how good the top k predictions are. But I don't know how you can generate more than one prediction. 

Can anyone help with this?