can't resume training from a saved checkpoint in run_glue
Author: ohadrozen
Pull request: no
Labels: name: wontfix

# 🐛 Bug
Hi,
I'm using run_glue.py to train Roberta model. I ran the training for a few hours, but after 2 epochs it crashed due to low disk space. I now want to resume the training, and for that, I replaced the --model_name_or_path from roberta-base to [my checkpoint dir]. But then I get the following error:

"OSError: Model name 'models/tmp/roberta512/checkpoint-27000' was not found in tokenizers model name list (roberta-base, roberta-large, roberta-large-mnli, distilroberta-base, roberta-base-openai-detector, roberta-large-openai-detector). We assumed 'models/tmp/roberta512/checkpoint-27000' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'] but couldn't find such vocabulary files at this path or url."

in the checkpoint dir there are only two files:
pytorch_model.bin and config.json

## Information
This is the command line I used to run initially:
python run_glue.py    --model_name_or_path roberta-base   --task_name MNLI   --do_train   --do_eval   --data_dir /home/nlp/ohadr/PycharmProjects/BERT_classification/glue_data/MNLI   --max_seq_length 512   --per_device_train_batch_size 8   --learning_rate 5e-5   --num_train_epochs 3.0   --output_dir /tmp/MNLI/roberta512

This is the command line I tried to resume training with:
python run_glue.py    --model_name_or_path /tmp/MNLI/roberta512/checkpoint-27000   --task_name MNLI   --do_train   --do_eval   --data_dir /home/nlp/ohadr/PycharmProjects/BERT_classification/glue_data/MNLI   --max_seq_length 512   --per_device_train_batch_size 8   --learning_rate 5e-5   --num_train_epochs 3.0   --output_dir models/tmp/roberta512_cont

Model I am using (Bert, XLNet ...): Roberta

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [v] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [v] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. train the model using the following:
python run_glue.py    --model_name_or_path roberta-base   --task_name MNLI   --do_train   --do_eval   --data_dir /home/nlp/ohadr/PycharmProjects/BERT_classification/glue_data/MNLI   --max_seq_length 512   --per_device_train_batch_size 8   --learning_rate 5e-5   --num_train_epochs 3.0   --output_dir /tmp/MNLI/roberta512

2. 
use the following line to resume training from the last saved checkpoint - for that, change the marked directory below with your own directory:
python run_glue.py    --model_name_or_path [/tmp/MNLI/roberta512/checkpoint-27000<<change it!>>]   --task_name MNLI   --do_train   --do_eval   --data_dir /home/nlp/ohadr/PycharmProjects/BERT_classification/glue_data/MNLI   --max_seq_length 512   --per_device_train_batch_size 8   --learning_rate 5e-5   --num_train_epochs 3.0   --output_dir models/tmp/roberta512_cont

Error message:
"OSError: Model name 'models/tmp/roberta512/checkpoint-27000' was not found in tokenizers model name list (roberta-base, roberta-large, roberta-large-mnli, distilroberta-base, roberta-base-openai-detector, roberta-large-openai-detector). We assumed 'models/tmp/roberta512/checkpoint-27000' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'] but couldn't find such vocabulary files at this path or url."

## Expected behavior
I would expect the model to continue training from that checkpoint

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.1
- Platform: Linux-3.10.0-1127.13.1.el7.x86_64-x86_64-with-centos-7.8.2003-Core
- Python version: 3.7.7
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in> YES
- Using distributed or parallel set-up in script?: <fill in> PARALLEL




Comments:
Thanks!

sorry, in the checkpoint directory I have 5 files (and not 2 as I wrote above):
pytorch_model.bin
training_args.bin
config.json
optimizer.pt
scheduler.pt


the vocabulary file is missing in the checkpoint folder. You can use the base model vocabulary by adding more parameter `--tokenizer_name roberta-base`.

try this:
```
python run_glue.py --model_name_or_path [/tmp/MNLI/roberta512/checkpoint-27000<<change it!>>] --tokenizer_name roberta-base --task_name MNLI --do_train --do_eval --data_dir /home/nlp/ohadr/PycharmProjects/BERT_classification/glue_data/MNLI --max_seq_length 512 --per_device_train_batch_size 8 --learning_rate 5e-5 --num_train_epochs 3.0 --output_dir models/tmp/roberta512_cont
```

I am having a similar issue when trying to evalutate checkpoints on a test set. If I copy the `vocab.txt` from the final model to the checkpoint folder and evaluate it, the accuracy is significantly lower. The final model had 0.54 and the checkpoints are all in the range from 0.31 to 0.38. That confuses me. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


how to train user custom data ?