How to save the model after using checkpoint to continue training
Author: yung1231
Pull request: no

I am trying to continue training my model (gpt2) from a checkpoint. However, the error occurred in the trainer area when I finished training.

I save a checkpoint for every 5000 steps, but because there is not enough space, the previous checkpoint will be deleted.

The last checkpoint I used was `checkpoint-70000`, but when I saved it, I had to find `checkpoint-35000`, but I had already deleted `checkpoint-35000`

Then how can I save the final trained model?

```
training_args = TrainingArguments(
    output_dir=model_checkpoints_dir,              # The directory where the model checkpoints and other output files will be saved.
    num_train_epochs=5,               # The total number of training epochs to run.
    per_device_train_batch_size=64,    # batch size per device during training
    per_device_eval_batch_size=32,     # batch size for evaluation
    warmup_steps=200,                  # number of warmup steps for learning rate scheduler
    weight_decay=0.01,                 # strength of weight decay
    logging_dir=model_log_dir,   # directory for storing logs
    prediction_loss_only=True,  
    save_steps=5000,
    logging_steps=5000,
    evaluation_strategy="steps",
    save_strategy="steps",
    load_best_model_at_end=True
  )
  
trainer = Trainer(
    model=model,                         # the instantiated 🤗 Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    data_collator=data_collator,
    train_dataset=tokenized_train_dataset,         # training dataset
    eval_dataset=tokenized_eval_dataset
  )

trainer.train(resume_from_checkpoint = True)
```

error message

```
/usr/local/lib/python3.9/dist-packages/transformers/trainer.py in _sorted_checkpoints(self, output_dir, checkpoint_prefix, use_mtime)
   2758         # Make sure we don't delete the best model.
   2759         if self.state.best_model_checkpoint is not None:
-> 2760             best_model_index = checkpoints_sorted.index(str(Path(self.state.best_model_checkpoint)))
   2761             for i in range(best_model_index, len(checkpoints_sorted) - 2):
   2762                 checkpoints_sorted[i], checkpoints_sorted[i + 1] = checkpoints_sorted[i + 1], checkpoints_sorted[i]

ValueError: '/content/drive/MyDrive/exp/model_checkpoints/checkpoint-35000' is not in list
```

Thanks a lot for the help.

Comments:
It looks like you accidentally deleted the best checkpoint. To fix this and be able to resume training, I'd advise to manually modify the `training_state` (which should be stored in a file named `trainer_state.json` in the checkpoint-70000 folder) and remove the key for `best_model_checkpoint`.

I'm having the same problem.

For me it was because I cloned the repository to a different path (of a different machine).

Since `best_model_checkpoint` contains an absolute path it cannot find the checkpoint at that path.

I fixed it by manually editing `best_model_checkpoint` from `trainer_state.json`.

Is there a way to store a relative path instead of an absolute one?