Failing to run pregenerate_training_data.py & finetune_on_pregenerated.py
Author: SamuelLarkin
Pull request: no
Labels: name: wontfix

Hi,
  I would like to fine tune BERT using my own data.

```
   readonly model=bert-base-multilingual-cased
   export PYTORCH_PRETRAINED_BERT_CACHE=.
   #[https://github.com/huggingface/pytorch-pretrained-BERT/tree/master/examples/lm_finetuning](BERT Model Finetuning using Masked Language Modeling objective)

   # pytorch-pretrained-bert convert_tf_checkpoint_to_pytorch $bert_model_home/multi_cased_L-12_H-768_A-12/{bert_model.ckpt,bert_config.json} bert-base-multilingual-cased

   #zcat --force corpora/*.{en,fr} > my_corpus.txt
   #zcat --force corpora/unannotated_seq.{en,fr} > my_corpus.txt

   mkdir -p training

   #Pregenerating training data
   python3 pytorch-pretrained-BERT/examples/lm_finetuning/pregenerate_training_data.py \
      --train_corpus my_corpus.txt \
      --bert_model $model \
      --output_dir training/ \
      --epochs_to_generate 3 \
      --max_seq_len 256

   mkdir -p finetuned_lm

   #Training on pregenerated data
   python3 pytorch-pretrained-BERT/examples/lm_finetuning/finetune_on_pregenerated.py \
      --pregenerated_data training/ \
      --bert_model $model \
      --output_dir finetuned_lm/ \
      --epochs 3
```

Then I get
```
Model name 'bert-base-multilingual-cased' was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese). We assumed 'https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-vocab.txt' was a path or url but couldn't find any file associated to this path or url.
Traceback (most recent call last):
  File "pytorch-pretrained-BERT/examples/lm_finetuning/pregenerate_training_data.py", line 338, in <module>
    main()
  File "pytorch-pretrained-BERT/examples/lm_finetuning/pregenerate_training_data.py", line 293, in main
    vocab_list = list(tokenizer.vocab.keys())
AttributeError: 'NoneType' object has no attribute 'vocab'
No training data was found!
```

```
wc my_corpus.txt
390400 my_corpus.txt
```

Why is `bert-base-multilingual-cased` not found in `(bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese)` when it is clearly there?

Why am I getting a NoteType for the vocab, I thought it was supposed to be autodownloaded if missing.

What does `bert-base-multilingual-cased` represent?  Should I have converted beforehand a tf model to a pytorch model and named it `bert-base-multilingual-cased` represent?

Comments:
Your server probably can't reach AWS to download the models.
I need to make these error messages more clear, they currently gather several failure cases.
Will do that in the coming release of next week.

What should I do if I have downloaded the package manually?

If you download them manually, you will have to figure out what is their file name in the cache.  My problem was that on the cluster I'm using, worker nodes don't have access to the internet but the interactive nodes do.  It's when I was tracing the code on an interactive node that I downloaded the models.  Then, to run on a worker node, I define `export PYTORCH_PRETRAINED_BERT_CACHE=$BERT_MODEL_HOME/pytorch_pretrained_bert` which obviously is pointing to my cache.

Back to downloading manually, you will have to "properly" name your model.  Looking at my cache, I see
`
a803ce83ca27fecf74c355673c434e51c265fb8a3e0e57ac62a80e38ba98d384.681017f415dfb33ec8d0e04fe51a619f3f01532ecea04edbfd48c5d160550d9c
` which is actually `bert-base-cased.tar.gz`.  The name of the model's file is base on some web id (sorry not familiar with this).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.