Logical error in the run_lm_finetuning?
Author: snakers4
Pull request: no

Hi,

@thomwolf @nhatchan 
@tholor @deepset-ai

Many thanks for amazing work with this repository =)

I maybe grossly wrong or just missed some line of the code somewhere, but it seems to me that there is a glaring issue in the overall logic of `examples/run_lm_finetuning.py` - I guess you never pre-trained the model till convergence from scratch, right?

_________________________________________

**Context**

I have already been able to fit the model to the Russian version of the SQUAD dataset from scratch (so-called **SberSQUAD** from sdsj 2017), and I was able to obtain **~40% EM w/o any pre-training**. Afaik, ~60% EM is about the top result on this dataset, achieved using BiDAF, so the model worksm which is good =). 

Anyway this was a sanity check for me to see that the model is sound, obviously to **achieve good results you need to pre-train first** (afaik the authors of the BERT paper did not even post any results w/o pre-training, right?).

So now I am planning to pre-train BERT for the Russian language with various pre-processing ideas:
- BPE (like in the original);
- Embedding bag (works well for "difficult" languages) + ;

_________________________________________

**The Problem**

First of all let's quote the paper
```
In order to train a deep bidirectional representation, we take a straightforward approach of masking 
some percentage of the input tokens at random, and then predicting only those masked tokens. 

We refer to this procedure as a “masked LM” (MLM), although it is often referred to as a Cloze task in 
the literature (Taylor, 1953). In this case, the fi- nal hidden vectors corresponding to the mask tokens are
 fed into an output softmax over the vo- cabulary, as in a standard LM. In all of our exper- iments, we 
mask 15% of all WordPiece tokens in each sequence at random. In contrast to denoising auto-encoders 
(Vincent et al., 2008), we only pre- dict the masked words rather than reconstructing the entire input. 

```

So as far as I can see:
- We mask / alter some of the input (afaik the masking scheme [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_lm_finetuning.py#L276) is correct) and make the model correct our "mistakes". It only makes sense - we break the input, and the model corrects it;


- But if you look [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_lm_finetuning.py#L142), [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_lm_finetuning.py#L331-L334) and [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_lm_finetuning.py#L371) - it seems to me that in the code:
  - Just padded / processed tokens are passed as input;
  - The lm targets are the "messed up" tokens;

So, the training is kind of reversed.
The correct sequence is passed, but the incorrect sequence is the target.

Anyway - I may just have missed some line of code, that changes everything.
I am just trying to understand the model properly, because I need to do a total rewrite of the pre-processing, because in my domain usage of embedding bags proved to be more beneficial than BPE.

Many thanks!






Comments:
Hi, @snakers4,

I think this part is correct. 
The input comes from `tokens` and `input_ids` in line 371, some of which are _already_ masked/altered, and the LM targets are `lm_label_ids`, which contain the original tokens. 
Note that `random_word`, called in line 331 and 332, masks the words in `tokens_a` and `tokens_b` _in-place_; `t1_random` and `tokens_a` refer the same object actually. 

If you are trying to pre-train a model from scratch and having slow convergence issue, see discussions in #202. 

> The input comes from tokens and input_ids in line 371, some of which are already masked/altered, and the LM targets are lm_label_ids, which contain the original tokens.

Ah, you are right, I see it [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_lm_finetuning.py#L354-L371), sorry. I totally missed the in-place part.

This [bit](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_lm_finetuning.py#L301) explains why  `lm_label_ids` are the original tokens.

This was a bit  counter-intuitive.
Anyway, thanks for the explanations, now everything is clear.