Add line by line option to mlm/plm scripts
Author: sgugger
Pull request: yes

# What does this PR do?

The old `run_language_modeling` script was supporting the option to choose `line_by_line` or not for the datasets in MLM/PLM. This PR adds that option to `run_mlm` and `run_plm`. It also updates the README to present those options and adds a flag to disable dynamic batching on TPU: TPUs need all batches to always have the same size to avoid recompiling the code at each training/evaluation step.

All scripts are tested on ditributed GPU/TPU env with or without the new flags and train to expected ppl on wikitext-2


Comments:
Hi, I am currently executing the **run_mlm.py** file, because I do not know the entire code structure very well, and I am a little confused about the 262 lines of code in the **run_mlm.py** file. It is line 262 in the figure below. 
![2020-11-03 19-21-40屏幕截图](https://user-images.githubusercontent.com/39059793/97980374-a1070f00-1e0b-11eb-9ba7-11041d1049cc.png)

I think whether False in **padding = "max_length" if data_args.pad_to_max_length else False** should be True. Still say that True or False has no effect on the result. Thank you. Please ignore if I understand it wrong.  :)

In addition, the following figure shows the usage of **tokenizer** in the Transformers documentation.
![2020-11-03 19-37-35屏幕截图](https://user-images.githubusercontent.com/39059793/97980647-15da4900-1e0c-11eb-8c86-891d37dcf901.png)



Hi there, the test is as intended: the behavior is the following:
- if `data_args.pad_to_max_length` is True, then we will pad to the maximum length of the model.
- otherwise we don't pas (yet). Padding will done by the data collator so that we pad to the maximum length in the batch (dynamic padding).

I got it, thanks!

Hello, I still have a question to consult you. I want to train the **Translation Language Modeling (TLM)** in **XLM** (Paper: Cross-lingual Language Model Pretraining). The translation language modeling (**TLM**) is very similar to the  **Masked Language Modeling (MLM)**, which only shows the difference in the form of input data. If I want to use the **run_mlm.py** file to achieve the effect of training the translation language modeling (**TLM**), can I just modify the composition of training data without modifying the source code of the **run_mlm.py** file? Is this feasible?

For example, for the masked language modeling (**MLM**), one row of my training data is a language, as shown below:

( **Row 1** ) polonium 's isotopes tend to decay with alpha or beta decay ( **en** ) .
( **Row 2** ) 231 and penetrated the armour of the Panzer IV behind it ( **en** ) .
( **Row 3** ) die Isotope von Polonium neigen dazu , mit dem Alpha- oder Beta-Zerfall zu zerfallen ( **de** ) .
( **Row 4** ) 231 und durchbrach die Rüstung des Panzers IV hinter ihm ( **de** ) .
**...**

For the translation language modeling (**TLM**), my training data is a combination of two parallel corpora (It is to splice the above data in pairs. The separator is **[/s]**.), as shown below:

( **Row 1** ) polonium 's isotopes tend to decay with alpha or beta decay ( **en** ) . **[/s]** die Isotope von Polonium neigen dazu , mit dem Alpha- oder Beta-Zerfall zu zerfallen ( **de** ) .
( **Row 2** ) 231 and penetrated the armour of the Panzer IV behind it ( **en** ) . **[/s]** 231 und durchbrach die Rüstung des Panzers IV hinter ihm ( **de** ) .
**...**


If I only modify the training data into a combination of two parallel corpora before executing the **run_mlm.py** file, can I achieve the effect of training the translation language modeling (**TLM**)?

Looking forward to your answer, thank you very much!

Hi @i-wanna-to this last question is something you should post on the forum for discussion at https://discuss.huggingface.co