Add `examples/language_modeling/run_mlm_no_trainer.py`
Author: hemildesai
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR adds an example of finetuning a Masked Language Model (without using `Trainer`) to show the functionalities of the new accelerate library.



## Who can review?

@sgugger 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Thanks again!

How to distributed training?~~~ In no trainer mlm

The same way as any other scripts: `python -m torch.distributed.launch --nproc_per_node xxx run_mlm_no_trainer.py --script_args`.

hello@sgugger , when I used multi-gpu, I got this error message:
(basic_dl) root@PM00011093:/data/zhaoyichen/workplace/transformers-master/examples# python -m torch.distributed.launch \
>     --nproc_per_node 2 pytorch/language-modeling/run_mlm_no_trainer.py \
>     --dataset_name wikitext \
>     --dataset_config_name wikitext-2-raw-v1 \
>     --model_name_or_path roberta-base \
>     --output_dir /tmp/test-mlm
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
usage: run_mlm_no_trainer.py [-h] [--dataset_name DATASET_NAME]
                             [--dataset_config_name DATASET_CONFIG_NAME]
                             [--train_file TRAIN_FILE]
                             [--validation_file VALIDATION_FILE]
                             [--validation_split_percentage VALIDATION_SPLIT_PERCENTAGE]
                             [--pad_to_max_length] --model_name_or_path
                             MODEL_NAME_OR_PATH [--config_name CONFIG_NAME]
                             [--tokenizer_name TOKENIZER_NAME]
                             [--use_slow_tokenizer]
                             [--per_device_train_batch_size PER_DEVICE_TRAIN_BATCH_SIZE]
                             [--per_device_eval_batch_size PER_DEVICE_EVAL_BATCH_SIZE]
                             [--learning_rate LEARNING_RATE]
                             [--weight_decay WEIGHT_DECAY]
                             [--num_train_epochs NUM_TRAIN_EPOCHS]
                             [--max_train_steps MAX_TRAIN_STEPS]
                             [--gradient_accumulation_steps GRADIENT_ACCUMULATION_STEPS]
                             [--lr_scheduler_type {linear,cosine,cosine_with_restarts,polynomial,constant,constant_with_warmup}]
                             [--num_warmup_steps NUM_WARMUP_STEPS]
                             [--output_dir OUTPUT_DIR] [--seed SEED]
                             [--model_type {clip,bigbird_pegasus,deit,luke,gpt_neo,big_bird,speech_to_text,vit,wav2vec2,m2m_100,convbert,led,blenderbot-small,retribert,mt5,t5,pegasus,marian,mbart,blenderbot,distilbert,albert,camembert,xlm-roberta,bart,longformer,roberta,layoutlm,squeezebert,bert,openai-gpt,gpt2,megatron-bert,mobilebert,transfo-xl,xlnet,flaubert,fsmt,xlm,ctrl,electra,reformer,funnel,lxmert,bert-generation,deberta,deberta-v2,dpr,xlm-prophetnet,prophetnet,mpnet,tapas,ibert}]
                             [--max_seq_length MAX_SEQ_LENGTH]
                             [--line_by_line LINE_BY_LINE]
                             [--preprocessing_num_workers PREPROCESSING_NUM_WORKERS]
                             [--overwrite_cache OVERWRITE_CACHE]
                             [--mlm_probability MLM_PROBABILITY]
run_mlm_no_trainer.py: error: unrecognized arguments: --local_rank=0
usage: run_mlm_no_trainer.py [-h] [--dataset_name DATASET_NAME]
                             [--dataset_config_name DATASET_CONFIG_NAME]
                             [--train_file TRAIN_FILE]
                             [--validation_file VALIDATION_FILE]
                             [--validation_split_percentage VALIDATION_SPLIT_PERCENTAGE]
                             [--pad_to_max_length] --model_name_or_path
                             MODEL_NAME_OR_PATH [--config_name CONFIG_NAME]
                             [--tokenizer_name TOKENIZER_NAME]
                             [--use_slow_tokenizer]
                             [--per_device_train_batch_size PER_DEVICE_TRAIN_BATCH_SIZE]
                             [--per_device_eval_batch_size PER_DEVICE_EVAL_BATCH_SIZE]
                             [--learning_rate LEARNING_RATE]
                             [--weight_decay WEIGHT_DECAY]
                             [--num_train_epochs NUM_TRAIN_EPOCHS]
                             [--max_train_steps MAX_TRAIN_STEPS]
                             [--gradient_accumulation_steps GRADIENT_ACCUMULATION_STEPS]
                             [--lr_scheduler_type {linear,cosine,cosine_with_restarts,polynomial,constant,constant_with_warmup}]
                             [--num_warmup_steps NUM_WARMUP_STEPS]
                             [--output_dir OUTPUT_DIR] [--seed SEED]
                             [--model_type {clip,bigbird_pegasus,deit,luke,gpt_neo,big_bird,speech_to_text,vit,wav2vec2,m2m_100,convbert,led,blenderbot-small,retribert,mt5,t5,pegasus,marian,mbart,blenderbot,distilbert,albert,camembert,xlm-roberta,bart,longformer,roberta,layoutlm,squeezebert,bert,openai-gpt,gpt2,megatron-bert,mobilebert,transfo-xl,xlnet,flaubert,fsmt,xlm,ctrl,electra,reformer,funnel,lxmert,bert-generation,deberta,deberta-v2,dpr,xlm-prophetnet,prophetnet,mpnet,tapas,ibert}]
                             [--max_seq_length MAX_SEQ_LENGTH]
                             [--line_by_line LINE_BY_LINE]
                             [--preprocessing_num_workers PREPROCESSING_NUM_WORKERS]
                             [--overwrite_cache OVERWRITE_CACHE]
                             [--mlm_probability MLM_PROBABILITY]
run_mlm_no_trainer.py: error: unrecognized arguments: --local_rank=1
Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/root/anaconda3/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/data/zhaoyichen/pyvenv/basic_dl/lib/python3.6/site-packages/torch/distributed/launch.py", line 261, in <module>
    main()
  File "/data/zhaoyichen/pyvenv/basic_dl/lib/python3.6/site-packages/torch/distributed/launch.py", line 257, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/data/zhaoyichen/pyvenv/basic_dl/bin/python', '-u', 'pytorch/language-modeling/run_mlm_no_trainer.py', '--local_rank=1', '--dataset_name', 'wikitext', '--dataset_config_name', 'wikitext-2-raw-v1', '--model_name_or_path', 'roberta-base', '--output_dir', '/tmp/test-mlm']' returned non-zero exit status 2.



You need to launch it with `--use_env` when using the PyTorch launcher (or use `accelerate launch`).