continuing fine-tuning from the last checkpoint 
Author: naturecreator
Pull request: no

Hello,

While fine-tuning BERT on the custom data using "run_language_modeling.py" script, due to memory issue the fine-tuning stopped in the middle. However, I tried to resume the fine-tuning from the last checkpoint. But, I came across with the following error:

```
python run_language_modeling.py     --output_dir=/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/new_results     --model_type=bert     --model_name_or_path=/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/new_results/result_dir/checkpoint_37000/     --do_train     --train_data_file=$TRAIN_FILE     --do_eval     --eval_data_file=$TEST_FILE     --mlm  --per_gpu_train_batch_size=4
/home/ai-students/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/ai-students/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/ai-students/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/ai-students/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/ai-students/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/ai-students/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/ai-students/anaconda3/envs/env_nesara/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/ai-students/anaconda3/envs/env_nesara/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/ai-students/anaconda3/envs/env_nesara/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/ai-students/anaconda3/envs/env_nesara/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/ai-students/anaconda3/envs/env_nesara/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/ai-students/anaconda3/envs/env_nesara/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
11/08/2020 22:40:18 - WARNING - __main__ -   Process rank: -1, device: cuda:0, n_gpu: 2, distributed training: False, 16-bits training: False
11/08/2020 22:40:18 - INFO - __main__ -   Training/evaluation parameters TrainingArguments(output_dir='/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/new_results', overwrite_output_dir=False, do_train=True, do_eval=True, do_predict=False, evaluate_during_training=False, evaluation_strategy=<EvaluationStrategy.NO: 'no'>, prediction_loss_only=False, per_device_train_batch_size=8, per_device_eval_batch_size=8, per_gpu_train_batch_size=4, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=5e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=3.0, max_steps=-1, warmup_steps=0, logging_dir='runs/Nov08_22-40-18_aistudents-msi', logging_first_step=False, logging_steps=500, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=500, dataloader_num_workers=0, past_index=-1, run_name='/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/new_results', disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=None)
Traceback (most recent call last):
  File "/home/ai-students/anaconda3/envs/env_nesara/lib/python3.6/site-packages/transformers/configuration_utils.py", line 387, in get_config_dict
    raise EnvironmentError
OSError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run_language_modeling.py", line 355, in <module>
    main()
  File "run_language_modeling.py", line 236, in main
    config = AutoConfig.from_pretrained(model_args.model_name_or_path, cache_dir=model_args.cache_dir)
  File "/home/ai-students/anaconda3/envs/env_nesara/lib/python3.6/site-packages/transformers/configuration_auto.py", line 329, in from_pretrained
    config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/ai-students/anaconda3/envs/env_nesara/lib/python3.6/site-packages/transformers/configuration_utils.py", line 396, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load config for '/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/new_results/result_dir/checkpoint_37000/'. Make sure that:

- '/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/new_results/result_dir/checkpoint_37000/' is a correct model identifier listed on 'https://huggingface.co/models'

- or '/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/new_results/result_dir/checkpoint_37000/' is the correct path to a directory containing a config.json file

```

Command used to fine-tune from the last checkpoint is as follows:

```
python run_language_modeling.py     --output_dir=/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/new_results     --model_type=bert     --model_name_or_path=/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/new_results/result_dir/checkpoint_37000/     --do_train     --train_data_file=$TRAIN_FILE     --do_eval     --eval_data_file=$TEST_FILE     --mlm  --per_gpu_train_batch_size=4

```

Here is the command used to fine-tune BERT earlier:

```

python run_language_modeling.py     --output_dir=/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/result_dir     --model_type=bert     --model_name_or_path=bert-base-cased     --do_train     --train_data_file=$TRAIN_FILE     --do_eval     --eval_data_file=$TEST_FILE     --mlm  --per_gpu_train_batch_size=4

```

@sgugger Could anyone please let me know on how to resume fine-tuning from the last checkpoint? 


Thanks in advance :)

Comments:
It looks like the config was not saved in the checkpoint folder you are passing. Double-check its contents, but apparently, the model was not properly saved inside it.

> It looks like the config was not saved in the checkpoint folder you are passing. Double-check its contents, but apparently, the model was not properly saved inside it.

Thanks for replying @sgugger 

I have the following files in my checkpoint folder:

```
config.json  optimizer.pt  pytorch_model.bin  scheduler.pt  trainer_state.json  training_args.bin

```
and inside the "config.json" it looks like this:

```
{
  "_name_or_path": "bert-base-cased",
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "type_vocab_size": 2,
  "vocab_size": 28996
}

```

Do you have any idea where exactly I am going wrong?

Oh, I think you just have a typo in your path:
```
/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/new_results/result_dir/checkpoint_37000/
```
should be
```
/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/new_results/result_dir/checkpoint-37000/
```
(a dash instead of the underscore).

Ahhhh my bad!
Thanks a lot @sgugger 
I had encountered one more problem (vocab.txt is missing):
```
Traceback (most recent call last):
  File "run_language_modeling.py", line 355, in <module>
    main()
  File "run_language_modeling.py", line 244, in main
    tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path, cache_dir=model_args.cache_dir)
  File "/home/ai-students/anaconda3/envs/env_nesara/lib/python3.6/site-packages/transformers/tokenization_auto.py", line 336, in from_pretrained
    return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/ai-students/anaconda3/envs/env_nesara/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1649, in from_pretrained
    list(cls.vocab_files_names.values()),
OSError: Model name '/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/result_dir/checkpoint-37000' was not found in tokenizers model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased, TurkuNLP/bert-base-finnish-cased-v1, TurkuNLP/bert-base-finnish-uncased-v1, wietsedv/bert-base-dutch-cased). We assumed '/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/result_dir/checkpoint-37000' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.txt'] but couldn't find such vocabulary files at this path or url.

```

After adding the parameter --tokenizer_name I could resolve the issue and now the fine-tuning resumes as expected

Below is the command:

```
python run_language_modeling.py     --output_dir=/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/new_results     --model_type=bert     --model_name_or_path=/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/result_dir/checkpoint-37000    --do_train     --train_data_file=$TRAIN_FILE     --do_eval     --eval_data_file=$TEST_FILE     --mlm  --per_gpu_train_batch_size=4 --tokenizer_name=bert-base-cased
```