Streamlined UX in saving, loading, continue training.
Currently each time I ran the command with the same experiment name, a new version is created and trained from scratch.
If I wanted to load back the model to continue training, I have to create a different script, input the path for the previous checkpoints and meta_tags.
Proposal:

Start training, input only 1 log_path and 1 experiment_name. Test_tube data and saved models will be included in the same folder.
In each experiment, default is to create a new model and trained from scratch (current behavior)
If user passed in --continue-training, model will load the latest model (not necessarily the best model) from the latest version, then create a new experiment version and continue training from there.
If user passed in --continue-training--best, model will load the the best model from the latest version, then create a new experiment version and continue training from there.
If user passed in --continue-training(--best) --version=X, auto load the model from version X and start training.

In the end, all the path and dir are determined from only the log_path and name of experiment, and default behavior is the same as of now:
python train.py --exp_name=exp1 --....
If user want to change some hyperparams, or start finetuning, they will only need to add:
python train.py --exp_name=exp1 --....  --continue-trainig--best --version=x
What do you think?
Happy to discuss more, as I am not sure this is suitable for cluster training or not.
I can make a PR if you're interest.