How to read a checkpoint and continue training?
Author: a-maci
Pull request: no
Labels: name: Need more information

name: wontfix

I wanted to experiment with longer training schedules. How do I re-start a run from it’s fine-tuned checkpoint?

Comments:
Hi, what fine-tuning script and model are you referring to?

I would like to know how to restart / continue runs as well.
I would like to fine tune on half data first, checkpoint it. Then restart and continue on the other half of the data.

Like the `main` function in this finetuning script:
https://github.com/huggingface/pytorch-pretrained-BERT/blob/3fc63f126ddf883ba9659f13ec046c3639db7b7e/examples/lm_finetuning/simple_lm_finetuning.py

@thomwolf  Hi. I was experimenting with run_squad.py on colab. I was able to train and checkpoint the model after every 50 steps. However, for some reason, the notebook crashed and did not resume training. Is there a way to load that checkpoint and resume training from that point onwards? 

I am fine-tuning using run_glue.py on bert. Have a checkpoint that I would like to continue from since my run crashed. Also, what happens to the tensorboard event file? For example, if my checkpoint is at iteration 250 (and my checkpoint crashed at 290), will the Tensorboard event file be appended correctly???

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I think the solution is to change the model name to the checkpoint directory. When using the `run_glue.py` example script I changed the parameter from `--model_name_or_path bert-base-uncased` to `--model_name_or_path ../my-output-dir/checkpoint-1600`

> I think the solution is to change the model name to the checkpoint directory. When using the `run_glue.py` example script I changed the parameter from `--model_name_or_path bert-base-uncased` to `--model_name_or_path ../my-output-dir/checkpoint-1600`

Hi, this works but may I know what did you do the OURPUT-DIR? Keeping the same one while "overwriting" or starting a new one? Thanks!

> I think the solution is to change the model name to the checkpoint directory. When using the `run_glue.py` example script I changed the parameter from `--model_name_or_path bert-base-uncased` to `--model_name_or_path ../my-output-dir/checkpoint-1600`

Hi, I tried this. The following error message shows: "We assumed '/cluster/home/xiazhi/finetune_results_republican/checkpoint-1500' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'] but couldn't find such vocabulary files at this path or url." But only after all epochs are done will the vocal.json and merges.txt be generated.


@anniezhi I have the same problem. This makes training very difficult; anyone have any ideas re: how to save the tokenizer whenever the checkpoints are saved?

@anniezhi I figured it out - if loading from a checkpoint, use the additional argument --tokenizer_name and provide the name of your tokenizer. Here's my helper bash script for reference :

```
#!/bin/bash
conda activate transformers

cd "${HOME}/Desktop"
rm -rf "./${1}"

TRAIN_FILE="/media/b/F:/debiased_archive_200.h5"

#Matt login key
wandb login MY_API_KEY

python bao-ai/training_flows/run_language_custom_modeling.py \
    --output_dir="./${1}" \
    --tokenizer_name=gpt2 \
    --model_name_or_path="${2}" \
    --block_size "${3}" \
    --per_device_train_batch_size "${4}" \
    --do_train \
    --train_data_file=$TRAIN_FILE\
```

If you're using the latest release (v3.1.0), the tokenizer should be saved as well, so there's no need to use the `--tokenizer_name` anymore.

For any version <3.1.0, @apteryxlabs's solution is the way to go!

> 

Browse parameters, resume_from_checkpoint=./
![image](https://user-images.githubusercontent.com/82740038/189427930-06ad046b-2c3b-4dfa-b531-bc9a96b706c9.png)
Now, the code runs from checkpoint
![image](https://user-images.githubusercontent.com/82740038/189428009-6946e6c0-aee6-4079-a388-21abad592630.png)


> 

Browse parameters, resume_from_checkpoint=./
![image](https://user-images.githubusercontent.com/82740038/189427930-06ad046b-2c3b-4dfa-b531-bc9a96b706c9.png)
Now, the code runs from checkpoint
![image](https://user-images.githubusercontent.com/82740038/189428009-6946e6c0-aee6-4079-a388-21abad592630.png)