Pause and resume support
🚀 Feature
It would be really helpful if we could have a pause and resume support for training networks. I am not talking about restarting from checkpoint training, but more in the lines of stopping in between randomly, the saving the optimizer state_dicts, and if resumed, then starting from the same place where it left off during training which would be a slightly more complicated and precise version of checkpointing.
Motivation
During one of my recent experiments, I am currently running very very long epochs, epochs usually last for 12 hours over 10000 samples and a batch size of 1 is causing constraints to train.
I sometimes want to add new images to the data loader without losing the state_dict and it was possible in Pytorch with some hacky methods, but I wanted to know if we already/could support this feature.
From my experience, stopping with a keyboard interrupts usually calls a graceful shutdown as mentioned here which does save optimizer state_dicts, etc and call an on_train_end, but is it possible to resume from the same point of the data loader?
Also, It would be great to know if all interrupts are supported with this feature which feels a little unclear reading this
Could you shed some light on this @williamFalcon if possible?