How to load the finetuned model for retraining from checkpoints in run_squad.py?
Author: Tahsin-Mayeesha
Pull request: no
Labels: name: wontfix

Because of bad internet connection and computational issues its hard for us to train a large number of epochs. We're trying to use the run_squad.py script for bangla QA system training. We have trained the model before and have the checkpoints. 

```
!python run_squad.py \
  --model_type distilbert \
  --model_name_or_path ../data/tmp/attempt_with_dataset_v3/checkpoint-5000/ \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file ../data/dataset_v_3/train_bangla_samples.json \
  --predict_file ../data/dataset_v_3/valid_bangla_samples.json \
  --version_2_with_negative \
  --per_gpu_train_batch_size 12 \
  --learning_rate 5e-5 \
  --num_train_epochs 1.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --logging_steps 100 \
  --save_steps 100 \
  --fp16 \
  --evaluate_during_training \
  --output_dir ../data/mytrial
```
command produces this error : 

```
12/20/2019 14:06:45 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 1, distributed training: False, 16-bits training: True
12/20/2019 14:06:47 - INFO - transformers.configuration_utils -   loading configuration file ../data/tmp/attempt_with_dataset_v3/checkpoint-5000/config.json
12/20/2019 14:06:47 - INFO - transformers.configuration_utils -   Model config {
  "activation": "gelu",
  "attention_dropout": 0.1,
  "dim": 768,
  "dropout": 0.1,
  "finetuning_task": null,
  "hidden_dim": 3072,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "max_position_embeddings": 512,
  "n_heads": 12,
  "n_layers": 6,
  "num_labels": 2,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pruned_heads": {},
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "tie_weights_": true,
  "torchscript": false,
  "use_bfloat16": false,
  "vocab_size": 30522
}

12/20/2019 14:06:47 - INFO - transformers.tokenization_utils -   Model name '../data/tmp/attempt_with_dataset_v3/checkpoint-5000/' not found in model shortcut name list (distilbert-base-uncased, distilbert-base-uncased-distilled-squad, distilbert-base-german-cased, distilbert-base-multilingual-cased). Assuming '../data/tmp/attempt_with_dataset_v3/checkpoint-5000/' is a path or url to a directory containing tokenizer files.
12/20/2019 14:06:47 - INFO - transformers.tokenization_utils -   Didn't find file ../data/tmp/attempt_with_dataset_v3/checkpoint-5000/vocab.txt. We won't load it.
12/20/2019 14:06:47 - INFO - transformers.tokenization_utils -   Didn't find file ../data/tmp/attempt_with_dataset_v3/checkpoint-5000/added_tokens.json. We won't load it.
12/20/2019 14:06:47 - INFO - transformers.tokenization_utils -   Didn't find file ../data/tmp/attempt_with_dataset_v3/checkpoint-5000/special_tokens_map.json. We won't load it.
12/20/2019 14:06:47 - INFO - transformers.tokenization_utils -   Didn't find file ../data/tmp/attempt_with_dataset_v3/checkpoint-5000/tokenizer_config.json. We won't load it.
Traceback (most recent call last):
  File "run_squad.py", line 614, in <module>
    main()
  File "run_squad.py", line 528, in main
    cache_dir=args.cache_dir if args.cache_dir else None)
  File "/content/gdrive/My Drive/huggingfaceattempt/transformers/transformers/tokenization_utils.py", line 302, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/content/gdrive/My Drive/huggingfaceattempt/transformers/transformers/tokenization_utils.py", line 370, in _from_pretrained
    list(cls.vocab_files_names.values())))
OSError: Model name '../data/tmp/attempt_with_dataset_v3/checkpoint-5000/' was not found in tokenizers model name list (distilbert-base-uncased, distilbert-base-uncased-distilled-squad, distilbert-base-german-cased, distilbert-base-multilingual-cased). We assumed '../data/tmp/attempt_with_dataset_v3/checkpoint-5000/' was a path or url to a directory containing vocabulary files named ['vocab.txt'] but couldn't find such vocabulary files at this path or url.
```
but none of the checkpoint folders have any vocabulary stored. Are we supposed to pass the checkpoint folder path to the model_name_or_path for training again from that checkpoint?

Comments:
Ah, turns out to run from a pretrained model we have to specify the output_dir as the previous checkpoint. I feel like its quite unintuitive. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.