how to continue training from a checkpoint with Trainer?
Author: oalej-dv
Pull request: no

# ❓ Questions & Help

## Details

I am trying to continue training my model (gpt-2) from a checkpoint, using Trainer. However when I try to do it the model starts training from 0, not from the checkpoint. I share my code because I don't know where I'm making the mistake.


```
import torch
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

from transformers import TextDataset,DataCollatorForLanguageModeling, AutoTokenizer, GPT2LMHeadModel, Trainer, TrainingArguments

tokenizer = AutoTokenizer.from_pretrained("gpt2-large")

train_dataset = TextDataset(
          tokenizer=tokenizer,
          file_path='textfile (1).txt',
          block_size=128)

data_collator = DataCollatorForLanguageModeling(
        tokenizer=tokenizer, mlm=False,
    )

model = GPT2LMHeadModel.from_pretrained("checkpoint-9500").to(device) ##HERE I LOAD FROM CHECKPOINT

training_args = TrainingArguments(
    output_dir='./results',         # output directory
    num_train_epochs=4,              # total # of training epochs
    per_device_train_batch_size=1,  # batch size per device during training
    per_device_eval_batch_size=64,   # batch size for evaluation
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./logs',            # directory for storing logs
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_dataset,
    #eval_dataset=validation_dataset,
    prediction_loss_only=True,
)

trainer.train()



```
Thanks a lot for the help.


Comments:
Hi there, you have to pass the checkpoint path to the method `Trainer.train` to resume training:
```
trainer.train("checkpoint-9500")
```
If you set your logging verbosity to the INFO level (`transformers.logging.set_verbosity_info()`) you should then see information about the training resuming and the number of steps skipped.

Great, thanks a lot for your help Sylvain.

Works perfect.


Hey, @fumpe which version of transformers were you using here? Please do let me know.

> Hi there, you have to pass the checkpoint path to the method `Trainer.train` to resume training:
> 
> ```
> trainer.train("checkpoint-9500")
> ```
> 
> If you set your logging verbosity to the INFO level (`transformers.logging.set_verbosity_info()`) you should then see information about the training resuming and the number of steps skipped.

@sgugger if training is over, `num_train_epochs`, is reached, how do you load the checkpoint and train say for the next n epochs? when I load the checkpoint, it only trains until that epoch is over 

```
trainer.num_train_epochs = trainer.num_train_epochs + 5
trainer.train("checkpoint-9500")
```


`Trainer` does not have a `num_train_epochs` attribute that is used, you need to set `trainer.args.num_train_epochs`. To be sure everything is updated, you probably need to re-instantiate the `Trainer` with the new `TrainingArguments` though.

@sgugger: I wanted to fine tune a language model using ```--resume_from_checkpoint``` since I had sharded the text file into multiple pieces. I noticed that the ```_save()``` in Trainer doesn't save the optimizer & the scheduler state dicts and so I added a couple of lines to save the state dicts. 
And I printed the learning rate from scheduler using ```lr_scheduler.get_last_lr()``` in ```_load_optimizer_and_scheduler()``` right after this [line](https://github.com/huggingface/transformers/blob/ac12a5ae47b352458694194ae7c8b971310015ee/src/transformers/trainer.py#L1678). It always prints 0 (as the last learning rate) although I initialize the learning rate with a non-zero value during the first run and I am also leaving the warmup args as default (which is 0). Here's the command I am using to run the script: 

```
CUDA_VISIBLE_DEVICES=2 python lm_scripts/lm_wwm.py --model_name_or_path "./output_dir/demo1/" --train_file \
"input_data_part2.csv" --validation_file "validation.csv"  --logging_dir "./output_dir/demo_log2" --logging_steps 1000 \
--save_steps 1000 --save_total_limit 2 --seed 42 --output_dir "./output_dir/demo2" --evaluation_strategy "epoch" \
--per_device_train_batch_size 5 --per_device_eval_batch_size 10 --do_train --do_eval --resume_from_checkpoint True \
--ignore_data_skip --num_train_epochs 2
``` 

Here's the [gist](https://gist.github.com/adithya8/7823d111efec9f3ff5313f2438068c2e) of the changes I made to the trainer code (Lines 904, 905, 1602, 1603). It would be super helpful if you could point me in the right direction to fix this?

@adithya8 The optimizer, learning rate scheduler, state of the scalers and all the RNGs are saved when doing a checkpoint, you should check the list of files you have in your `checkpoint-xxx` folder.

Gotcha! Thanks

When I resume training from a checkpoint, I use a new batch size different from the previous training and it seems that the number of the skipped epoch is wrong.

For example, I trained a model for 10 epochs with `per_device_train_batch_size=10` and generate a checkpoint. Then I  resume training from this checkpoint with `per_device_train_batch_size=10`, the trainer will skip 12 epochs.

What should I do if I want to resume training with a different batch size?

That feature is not supported, you should resume training with the exact same hyperparameters or start a new training if you want to change them.

I have trained a model and saved the checkpoint，Now I want to increase the number of epochs to continue training, but I have a problem.My training loss is not able to match the previous training, there is a very big difference（First up, then down, but not down to the original level）.I increased the number of epochs from 20 to 30.
`training_args = TrainingArguments( 
            output_dir= models_path + '/' + chain + '_' + modeltype + 'bert/two/log',
            save_total_limit=5,
            do_train=True,
            do_eval=True,
            evaluation_strategy='epoch',
            #eval_steps=1,
            overwrite_output_dir=True,
            num_train_epochs=30,
            per_device_train_batch_size=128,
            learning_rate=9.282980634054497e-05,
            weight_decay=0.1531103573951345,
            warmup_ratio=0.05549995455206759,
            logging_first_step=True,
            ignore_data_skip=True,
            save_steps=100,
            logging_dir = models_path + '/' + chain + '_' + modeltype + 'bert/two/runs/' + datetime.datetime.strftime(datetime.datetime.now(),'%Y-%m-%d_%H:%M:%S'),
            logging_steps=10,
            seed = 19930606
        )`
`trainer = Trainer(
            model=model,
            args=training_args,
            data_collator=data_collator,
            train_dataset=train_dataset,
            eval_dataset = eva_dataset
        )`
` trainer.train(resume_from_checkpoint=True)`
Thanks a lot for the help.

> have trained a model and saved the checkpoint，Now I want to increase the number of epochs to continue training, but I have a problem.My training loss is not able to match the previous training, there is a very big difference（First up, then down, but not down to the original level）.I increased the number of epochs



> That feature is not supported, you should resume training with the exact same hyperparameters or start a new training if you want to change them.

For example i want to freeze some layers in the checkpointed model. Is it possible?

I have two bin files like pytorch_model-00001-of-00002.bin and pytorch_model-00002-of-00002.bin in the checkpoint-1000 folder but it won't load! It still says that "ValueError: Can't find a valid checkpoint at checkpoint-1000."

Make sure you are using a source install of Transformers. This is a bug that was recently fixed @oscar-eaglewatch 

How do I differentiate between,

1. continue pre-training from a checkpoint with the previous optimizer/scheduler state
2. continue pre-training from a checkpoint but reset the optimizer and the scheduler 



@sgugger I have been changing my hyperparameters and got `Can't find a valid checkpoint`. Now it seems I cannot find the original hyperparameter settings (esp. `for` `per_device_train_batch_size`, `gradient_accumulation_steps`, and `max_steps`). I did find the `max_steps` in some json file in the checkpoint, but not the other hyperparameter settings. How can I find out the settings I should pick?

Hi, there, I got same error with deepspeed. training noramlly and resume got `Can't find a valid checkpoint`

first I try resume_from_checkpoint with `out/lora-Vicuna-chat` (output_path) got  `Can't find a valid checkpoint`
then I send `out/lora-Vicuna-chat/checkpoint-6000` I can not load the lora weights........

```
"base_model.model.model.layers.31.self_attn.k_proj.lora_B.default.weight", "base_model.model.model.layers.31.self_attn.v_proj.weight", "base_model.model.model.layers.31.self_attn.v_proj.lora_A.default.weight", 
"base_model.model.model.layers.31.self_attn.v_proj.lora_B.default.weight", "base_model.model.model.layers.31.self_attn.o_proj.weight", "base_model.model.model.layers.31.self_attn.o_proj.lora_A.default.weight", 
"base_model.model.model.layers.31.self_attn.o_proj.lora_B.default.weight", "base_model.model.model.layers.31.self_attn.rotary_emb.inv_freq", "base_model.model.model.layers.31.mlp.gate_proj.weight", 
"base_model.model.model.layers.31.mlp.gate_proj.lora_A.default.weight", "base_model.model.model.layers.31.mlp.gate_proj.lora_B.default.weight", "base_model.model.model.layers.31.mlp.down_proj.weight", 
"base_model.model.model.layers.31.mlp.down_proj.lora_A.default.weight", "base_model.model.model.layers.31.mlp.down_proj.lora_B.default.weight", "base_model.model.model.layers.31.mlp.up_proj.weight", 
"base_model.model.model.layers.31.mlp.up_proj.lora_A.default.weight", "base_model.model.model.layers.31.mlp.up_proj.lora_B.default.weight", "base_model.model.model.layers.31.input_layernorm.weight", 
  Unexpected key(s) in state_dict:"base_model.model.model.layers.31.self_attn.q_proj.lora_A.weight", "base_model.model.model.layers.31.self_attn.q_proj.lora_B.weight", "base_model.model.model.layers.31.self_attn.k_proj.lora_A.weight", 
"base_model.model.model.layers.31.self_attn.k_proj.lora_B.weight", "base_model.model.model.layers.31.self_attn.v_proj.lora_A.weight", "base_model.model.model.layers.31.self_attn.v_proj.lora_B.weight", 
"base_model.model.model.layers.31.self_attn.o_proj.lora_A.weight", "base_model.model.model.layers.31.self_attn.o_proj.lora_B.weight", "base_model.model.model.layers.31.mlp.gate_proj.lora_A.weight", 
"base_model.model.model.layers.31.mlp.gate_proj.lora_B.weight", "base_model.model.model.layers.31.mlp.down_proj.lora_A.weight", "base_model.model.model.layers.31.mlp.down_proj.lora_B.weight", 
```

the model with some suffix `default`, but samed model didn't have......

I am confused so much

Transformers have been a few years old, I thought that it should be mature by now, atleast 1 core version that works flawlessly all the way. 

But man, basic and simple issues like saving checkpoints, weight reload, resume, skip dataset etc... with pure transformers or deepspeed...still bumping around, real pain in the b** :( ... I must say that I admired Yolov5 most in the way they described things, annoucing news, bugs fixed. My 2 cents for this great repo, it deserves much more than this.

Falling back to 4.28.1 :((

I am having the same problem with Deepspeed, it seems that a different format of checkpoint is expected for Deepspeed. Did you manage to resolve the issue? @lucasjinreal lucasjinreal 

> Hi there, you have to pass the checkpoint path to the method `Trainer.train` to resume training:
> 
> ```
> trainer.train("checkpoint-9500")
> ```
> 
> If you set your logging verbosity to the INFO level (`transformers.logging.set_verbosity_info()`) you should then see information about the training resuming and the number of steps skipped.

@sgugger how would I start at the same step I left off on? (Or before my training run crashed, it stopped at step 1333 and I have ckpt there, and I used a consistent seed, set on the script.

(Also un related-ish there is a hugging face wiki, available as a raw file? I would love to train a deepseek model on hugging face code, that way can limit the lower the amount of people asking questions as the model could possibly answer it for them.)