Preemptive Training for Cloud Computing
🚀 Feature
By passing some parameters to the trainer, the trainer will start a preemptive learning process. So if the program will be interrupted, e.g. because of more important jobs, and continued later the trainer automatically recognizes that there was a training process before and continues training.
Motivation
Cloud computing is getting cheaper and often for researchers or hobby ml users it is more economical to use cloud computing for training. Also, there are often free offers for new users which getting up to 300$. Often preemptive cloud computing machines are much cheaper than normal non-preemptive ones because the machine could be interrupted and will continue later.
Pitch
I know that there is a 'resume_from_checkpoint' parameter, but it needs some code around to implement this feature. Wouldn't it be better, let's say, to have a 'preemptive_save_dir' and 'preemptive_backup_step' as trainer parameters. The first one sets the directory, where the checkpoints are saved and the second how often (e.g. after every 100 batches or every 5 epochs). Also maybe there is some work to do when it comes to distributed or multi GPU learning?
Alternatives
Every user who wants to run preemptive cloud computing solutions to train his model has to take care to continuously save checkpoints, and check on program start if there was a training process before and continue training itself.
It is just an idea. I thought we could discuss it. :)