Pre training problem, please help me out
Author: mosh98
Pull request: no

## Environment info

- `transformers` version: 4.8.2
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.11
- PyTorch version (GPU?): 1.9.0+cu102 (False)
- Tensorflow version (GPU?): 2.5.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
@sgugger
@LysandreJik
@patrickvonplaten 
@patil-suraj


## Information

Model I am using: CodeBert

The problem arises when using:
* [X ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Using the run_mlm.py script from : https://github.com/huggingface/transformers/tree/master/examples/pytorch/language-modeling
2.Inserting the code on terminal:
`python run_mlm.py --model_name_or_path microsoft/codebert-base --train_file /content/TrainingFiles.txt --do_train --do_eval --output_dir /tmp/test-mlm`
3. Getting this error
`python run_mlm.py --model_name_or_path microsoft/codebert-base --train_file /content/TrainingFiles.txt --do_train --do_eval --output_dir /tmp/test-mlm  File "run_mlm.py", line 67
    model_name_or_path: Optional[str] = field(
                      ^
SyntaxError: invalid syntax`


## Expected behavior
I just want to further pre train my model, any help would be much appreciated


Comments:
Are you using the `run_mlm.py` script from master?

yes sir i am

Can you create a colab to reproduce?

I just ran the script on google colab terminal.

`python run_mlm.py --model_name_or_path microsoft/codebert-base --train_file /content/TrainingFiles.txt --do_train --do_eval --output_dir /tmp/test-mlm`

i have attached the training text file here.
[TrainingFiles.txt](https://github.com/huggingface/transformers/files/6841465/TrainingFiles.txt)





its definitely a syntax error in the run_mlm.py file, i tried just running it by itself and this is what it throws out

`/content# python /content/run_mlm.py 
  File "/content/run_mlm.py", line 68
    model_name_or_path: Optional[str] = field(
                      ^
SyntaxError: invalid syntax`


Ok let me try this, I get back to you in a minute.

I ran it in a Google Colab notebook, it worked fine for me. 
https://colab.research.google.com/drive/1Hqih55xGyVCvNOQKsYOw1mMeULGCR6oQ?usp=sharing

Note that the `run_mlm.py` script checks the minimum version required, which is currently set to 4.9.0 (which will be released soon). I removed that check since it hasn't been released yet.

Thats interesting, weird that it didn't work for me.
Huge thanks Niels have a wonderful day 

Hi,

I just pre trained my model and it got saved in a directory, 

My question is how do i load that model for fine tuning.

I feel like i am not doing something correct here.

These are the files generated after pre training the bert model: 
`all_results.json  
 pytorch_model.bin	 
 tokenizer.json
vocab.json
config.json	   
runs			    
trainer_state.json
eval_results.json 
 special_tokens_map.json  
training_args.bin
merges.txt	   
tokenizer_config.json   
 train_results.json`

and then i tried to load the model using this,
`model = torch.load('/content/transformers/examples/pytorch/language-modeling/content/tmp/test-mlm/pytorch_model.bin',map_location='cpu')`

Is this correct?
It seems to load it but do i have to load any kind of saved dict or any other sort of files?

A bit confused

Appreciate any help

Hi,

It's easiest to just load it using the `.from_pretrained(directory_path)` method. Let's say you have trained a BertForMaskedLM model, then you can load it as follows:

```
from transformers import BertForMaskedLM

model = BertForMaskedLM.from_pretrained("path_to_the_directory_with_pytorch_model.bin_and_config.json")
```


okay that makes sense, Huge thanks again Niels! :D

Have a wonderful day!