Checkpointing for Callbacks.
Is your feature request related to a problem? Please describe.
It would be nice to enable checkpointing of the callbacks. The best case scenario is if checkpointing would allow you to resume training completely as if it hadn't stopped, and I think this feature would bring us one step closer.
Describe the solution you'd like
If you look at the code for LR schedulers in PyTorch it should be simple to implement:
  def state_dict(self):
        """Returns the state of the scheduler as a :class:`dict`.

        It contains an entry for every variable in self.__dict__ which
        is not the optimizer.
        """
        return {key: value for key, value in self.__dict__.items() if key != 'optimizer'}

    def load_state_dict(self, state_dict):
        """Loads the schedulers state.

        Arguments:
            state_dict (dict): scheduler state. Should be an object returned
                from a call to :meth:`state_dict`.
        """
        self.__dict__.update(state_dict)

They don't include the optimizer, because that would be stored/restored separately.
I would like to have a similar interface in the Callbacks, such that for example if training stops between each epoch and is restored, this will not make the Early Stopping callback void.
Additional Context
Reference docs: https://williamfalcon.github.io/pytorch-lightning/Trainer/Checkpointing/
Discussion/Unrelated
This is hardly related, but if you have time to read and consider the idea I would appreciate it. :)
It is hard to know all the things that must be done to make checkpointing completely save and restore the program state. (Short of using formal verification, which of course can't be done on the users code.)
One coding style that could make it more manageable is to adopt a somewhat more "functional" style in some of code. Currently we have self.restore simply return nothing and mutate the Trainer state. But realize that this function is always called when training is started
trainer.fit -> ... -> self.__run_pretrain_routine -> ... -> self.restore -> ... -> self.train.
The state we are using is always initialized/restored and then mutated later down the function call.
If the state is read from a checkpoint and only used in the training loop, is there any reason to keep this state in Trainer after trainer.fit returns?
The reason for the checkpoints is to be able to restore a training session, so what if we instead had a signature state = self.restore, then call self.train(state), where state contains all the trainer variables that are modified by the training procedure.
If all the variables modified are exclusively inside state (current index in loop, model, optimizers, callback statedicts, etc.) or derived from it, one can establish the invariant that a call to self.train(state) with the same state will always produce the same result.
A benefit is that one can then restrict to only passing the needed subset of the state to mutator functions, so bugs are easier to track down, or won't happen at all.
A more extreme version is to not mutate the state at all, and simply let functions return new objects (copy the old, modify and return modified versions) - a truly pure function. But this would likely not be of any benefit.