Understanding feature creation 
Author: AVSuni
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hi, 

I am trying to understand feature creation when using run_lm_finetuning.py. If I let the script download the tokenizer files, everything works fine. If I manually assign a folder for the tokenizer with vocab.json and merges.txt downloaded from Hugging Face,  I get an empty dataframe during feature creation. 

`11/19/2019 16:36:20 - INFO - __main__ -   Creating features from dataset file at /data/test
11/19/2019 16:41:53 - INFO - __main__ -   Saving features into cached file /data/test/roberta-base_cached_lm_999999999998_moleculenet_roberta_train.csv
Traceback (most recent call last):
  File "run_lm_finetuning.py", line 558, in <module>
    main()
  File "run_lm_finetuning.py", line 510, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 175, in train
    train_sampler = RandomSampler(train_dataset) if args.local_rank == -1 else DistributedSampler(train_dataset)
  File "/usr/local/conda3/lib/python3.7/site-packages/torch/utils/data/sampler.py", line 94, in __init__
    "value, but got num_samples={}".format(self.num_samples))
ValueError: num_samples should be a positive integer value, but got num_samples=0`


Comments:
Just to clarify, I am asking this because I originally tried to create a custom vocab/merges files by giving a new path for tokenizer.from_pretrained for my new files. Failure to do so led me to notice that giving a new path for the original files downloaded from S3 also fail in the new path. If I put my custom files to the path used for the S3 cache (with identical filenames used by the cache, everything works fine.  In other words, if I replace the contents of the following files with my custom content, everything works:

`11/19/2019 18:31:15 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-vocab.json from cache at /root/.cache/torch/transformers/d0c5776499adc1ded22493fae699da0971c1ee4c2587111707a4d177d20257a2.ef00af9e673c7160b4d41cfda1f48c5f4cba57d5142754525572a846a1ab1b9b
11/19/2019 18:31:15 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-merges.txt from cache at /root/.cache/torch/transformers/b35e7cd126cd4229a746b5d5c29a749e8e84438b14bcdb575950584fe33207e8.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda`  

Have you learned anything about this? I'm currently running to the same issue trying to specify an existing model. I noticed that the cache_lm_* file that it creates is only 5 bytes, and gedit says the files is corrupt.

Yes, so bizarrely even if you have the original hugggingface files in a new location, an empty data frame (the cache_lm file) will be created. Try replacing the original hugging face files downloaded from S3 with your custom vocab/merges files. They should be in /root/.cache/torch/transformers

If you still run into the same problem, then either your data is too small or the vocab/merges files are not in the correct format.

@AVSuni I get the same issue when using a checkpoint generated by huggingface/transformers. Even when using cached versions of the merge/vocab files from `/root/.cache/torch/transformers/`. Even when I delete the `cache_lm_*` file, specifying the model rather than giving a model name to be used causes this issue.

Maybe I'm being superstitious, but setting a block size seems to alleviate the issue. I didn't bother tracing the code, but I did notice that when I set `block_size`, the cache file is set with the number of the block size. When I do not set the block size, the temp file has some absurdly long number in the cache file where the block size would be. I don't really see that documented anywhere, but here we are.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.