num_samples=0 when using pretrained model
Author: Rajashan
Pull request: no

When training the "run_language_modelling.py" example with a pretrained bert model, I get the error:

"""
Traceback (most recent call last):                                                                                                                                                                                                             File "run_language_modeling.py", line 284, in <module>                                                                                                                                                                                         main()                                                                                                                                                                                                                                     File "run_language_modeling.py", line 254, in main                                                                                                                                                                                             trainer.train(model_path=model_path)                                                                                                                                                                                                       File "C:\Users\AppData\Roaming\Python\Python38\site-packages\transformers\trainer.py", line 243, in train                                                                                                                             train_dataloader = self.get_train_dataloader()                                                                                                                                                                                             File "C:\Users\AppData\Roaming\Python\Python38\site-packages\transformers\trainer.py", line 179, in get_train_dataloader                                                                                                              RandomSampler(self.train_dataset) if self.args.local_rank == -1 else DistributedSampler(self.train_dataset)                                                                                                                                File "C:\Users\Anaconda3\envs\kompetence\lib\site-packages\torch\utils\data\sampler.py", line 93, in __init__                                                                                                                         raise ValueError("num_samples should be a positive integer "                                                                                                                                                                             ValueError: num_samples should be a positive integer value, but got num_samples=0 
"""
The cached_lm_berttokenizer that gets saved to the data folder looks like this:

"""
€]”.
"""
In the folder for my bert model, I have a config.json, pytorch_model.bin, special_tokens_map.json, vocab.txt, tokenizer_config.json. I run the following command


python run_language_modeling.py --output_dir=. --model_type=bert --model_name_or_path="H:\danish_bert_uncased_v2\\" --mlm --train_data_file="H:\\data\train.txt" --eval_data_file="H:\data\test.txt" --do_eval --do_train --overwrite_output_dir -block_size=1000  

Is anything missing here? The data is composed of sentences seperated by newlines and an empty line between documents. 



Comments:
Hi, did you find out the reason for this error? I get the same error even on Wikitext-2 corpus 

Yeah, it is an issue with how your data is read. Try the --line_by_line argument.