seems there is a bug in fine tuning language model
Author: imhuim982
Pull request: no

For masked language model, the input should be tokens masked. But in examples/run_lm_finetuning.py, input is not masked.
In method convert_example_to_features, is it supposed to use masked output as token input?

    tokens = []
    segment_ids = []
    tokens.append("[CLS]")
    segment_ids.append(0)
    for token in tokens_a:# is tokens_a supposed to be t1_random?
        tokens.append(token)
        segment_ids.append(0)
    tokens.append("[SEP]")
    segment_ids.append(0)

    assert len(tokens_b) > 0
    for token in tokens_b:
        tokens.append(token)
        segment_ids.append(1)
    tokens.append("[SEP]")
    segment_ids.append(1)

    input_ids = tokenizer.convert_tokens_to_ids(tokens)

Comments:
I believe this is not a bug you are referring to, but indeed some confusing part of the code that we should probably change to avoid future confusion. `tokens_a` get masked **inplace** by the method `random_word`. `tokens_a` and `t1_random` refer indeed to the same objects. You can see that the input got masked also by checking the logs of the first examples:
```
 Iteration:   0%|          | 0/196 [00:00<?, ?it/s]01/31/2019 11:31:35 - INFO - __main__ -   *** Example ***
01/31/2019 11:31:35 - INFO - __main__ -   guid: 0
01/31/2019 11:31:35 - INFO - __main__ -   tokens: [CLS] [MASK] to 95 % of the [MASK] ##y [MASK] ' s [MASK] [SEP] le ##que ##ux ( 2005 : [MASK] ) . [SEP]
01/31/2019 11:31:35 - INFO - __main__ -   input_ids: 101 103 1106 4573 110 1104 1103 103 1183 103 112 188 103 102 5837 3530 5025 113 1478 131 103 114 119 102
01/31/2019 11:31:35 - INFO - __main__ -   input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
01/31/2019 11:31:35 - INFO - __main__ -   segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1
01/31/2019 11:31:35 - INFO - __main__ -   LM label: [-1, 1146, -1, -1, -1, -1, -1, 6831, -1, 1236, -1, -1, 14296, -1, -1, -1, -1, -1, -1, -1, 125, -1, -1, -1] 
01/31/2019 11:31:35 - INFO - __main__ -   Is next sentence label: 0 
01/31/2019 11:31:37 - INFO - __main__ -   *** Example ***
01/31/2019 11:31:37 - INFO - __main__ -   guid: 1
01/31/2019 11:31:37 - INFO - __main__ -   tokens: [CLS] a car [MASK] [MASK] 200 mill ##ig ##rams . [SEP] the rain had only ceased with the gray [MASK] of morning [MASK] [SEP]
01/31/2019 11:31:37 - INFO - __main__ -   input_ids: 101 170 1610 103 103 2363 6159 6512 24818 119 102 1103 4458 1125 1178 6445 1114 1103 5021 103 1104 2106 103 102
01/31/2019 11:31:37 - INFO - __main__ -   input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
01/31/2019 11:31:37 - INFO - __main__ -   segment_ids: 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1
01/31/2019 11:31:37 - INFO - __main__ -   LM label: [-1, -1, -1, 2980, 1110, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 24177, -1, -1, 1120, -1] 
01/31/2019 11:31:37 - INFO - __main__ -   Is next sentence label: 1 
```

 Please see discussion in https://github.com/huggingface/pytorch-pretrained-BERT/issues/226

ah, ok, i see, thx~

Added a PR to simplify this.

Btw, would inplace mask pollute training data in 'on_memory' way?

You mean if the original training sentence stored in `train_dataset.all_docs` get somehow modified (= masked)?!
=> No, this object is not touched by the LM masking / padding / cutting