Naming convention for (pytorch) checkpoints broken?
Author: ioana-blue
Pull request: no

# 🚀 Feature request

In previous versions of the library and sample files, checkpoints were saved with some naming convention that had `checkpoint` in the name file. Subsequent jobs could look in the output directory and check if any checkpoint is available first; if found, it would load the checkpoint and the corresponding config and continue training from where it left off; if not found, it would check for the model_path_or_name. 

I'm under the impression that this convention broke, from what I can tell. When using the utilities from the library, for pytorch models, the model is saved with the name `pytorch_model.bin` (WEIGHTS_NAME in file_utils.py) and when looking to load a checkpoint PREFIX_CHECKPOINT_DIR = "checkpoint" from trainer_utils.py is used. So it doesn't match and it starts training from scratch.

One (local) way to fix this is to rewrite searching for a checkpoint instead of using the one in the library. 

Is there any other option that allows a pipeline of jobs without using different scripts (e.g., one script that loads the original pretrained bert model, for example, and all subsequent runs use a different script that point the model_path to the local path where the pytorch_model.bin is saved). 

I guess the feature request is to bring this feature back. One way to do it is to use command line args for checkpoint names instead of using hardcoded naming in the files.

## Motivation

Cascading/pipelined training jobs: one job starts, takes a checkpoint, the next one picks up from the last checkpoint. The same script is used for either first or intermediate job in the pipeline.



Comments:
I think I might have come up with a bandaid solution for now. If the output dir exists and overwrite output dir flag is not set, load the configuration from the output dir and resume training from `pytorch_model.bin`. I'm going to give this a try, I think it's going to work. 

Yes, as I expected, it worked. I modified the sample scripts as follows:

```
    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:
        # last_checkpoint = get_last_checkpoint(training_args.output_dir)
        # check there is a WEIGHTS_NAME model in the output directory and use that as the last_checkpoint
        if os.path.isfile(os.path.join(training_args.output_dir, WEIGHTS_NAME)):
            # use it as checkpoint
            last_checkpoint = training_args.output_dir
```

The ramification of this change is that the checkpoint is going to be replaced upon training completion (which so happens is what I needed). 

I still think the checkpoint naming conventions should be reconciled (re: WEIGHTS_NAME and PREFIX_CHECKPOINT_DIR) so I'll leave this feature request open. 

Nothing has changed in the way checkpoints are named since version 2 at least, e.g. the checkpoints are saved in `args.output_dir/checkpoin-xxx` where xxx is the number of training steps elapsed since the beginning of training.

The change you are suggesting would remove the ability for people to resume training from the last checkpoint saved which is not something we want. If you want to start your training from a specific saved model, you can pass along 
`--model_name_or_path path_to_folder_with_saved_model`.

@sgugger Thanks for the reply. I think I wanted the behavior that you're talking about (not what I ended up doing, i.e., looking for a saved model directly in the output dir). 

Based on what you just said, I looked again at the code and there are two ways to save a checkpoint: `save_model` and `_save_checkpoint`. It so happens that the sample text classification scripts use `save_model` directly which does not create the `checkpoint-xxx` directory. Underneath, `_save_checkpoint` calls `save_model` but with the dir `checkpoint-xxx` which was the behavior that I wanted. 

BTW, what's the "right" way of saving models/checkpoints? the `_` in `_save_checkpoint` makes me believe it's supposed to be a utility function and there is another API function. 

So I guess what's happening is the sample script calls `save_model` when the training ends (saving the `pytorch_model.bin` directly in the output dir) and that confused me. Bottom line: the current sample script for text classification can't resume training from the last checkpoint saved (because it doesn't "save a checkpoint" it "saves a model" https://github.com/huggingface/transformers/blob/master/examples/text-classification/run_glue.py#L466).

I completely agree with you that that is a behavior everybody wants, i.e., resuming training from last checkpoint. If you agree, I'll change this issue to read save checkpoint in sample text classification script.

I think you are confusing:
- saving checkpoints during training, done automatically by the `Trainer` every `save_steps` (unless you are using a different strategy)
and
- saving the final model, which is done at the end of training.

Got it, thanks for clarifying the terminology! 

PS: It so happens that I needed a checkpoint to be saved at the end of training, now I understand how that's done.