Resume Training with trainer.fit(model) from specific values of current_epoch and global_step
🚀 Feature

Currently, trainer.fit(model) ignores values of model.global_step and trainer.current_epoch and overwrites them with 0s at the beginning of each call.
The Resume Training feature will enable you to "continue where you left off" with the training.
Motivation

This is useful for many research use-cases, where something new is being tried and user wants to "interrogate" their model dynamically and irregularly between subsequent training sessions.
When tracking the training, whether using custom logging or some standard tool like TensorBoard, most people will use current_epoch and / or global_step somewhere in their logging pipeline to "timestamp" their logs / metrics.
Resetting these values on each training session forces user to duplicate the management of current_epoch and global_step variables somewhere in their code, adding boilerplate code.
Additionally, it is desirable to have this the "state of training" of each model preserved between saving and loading of model checkpoints. It is always good to know passed_epochs of any particular checkpoint (e.g. "how many times did this particular model checkpoint see the full dataset so far").
Whether user is in a jupyter notebook environment or running python scripts is irrelevant, as in both cases it is useful and desired to have a mechanism for preservation of training progress of a model checkpoint from save to load operation.
Pitch

One could achieve this by manipulating trainer.max_nb_epochs, trainer.min_nb_epochs and obviously trainer.current_epoch inbetween subsequent calls to trainer.fit(model).
Minimum change proposal

trainer.fit(model) re-uses values of trainer.current_epoch and model.global_step.

Extended change proposal

expose model.save_to_checkpoint() to the user, as we already have model.load_from_checkpoint() available
implement preservation of "training state" (namely values of current_epoch and global_step) in save_to_checkpoint and load_from_checkpoint

ideally in a way such that user doesn't have to do trainer.current_epoch = model.passed_epochs between calls to model.load_from_checkpoint and trainer.fit(model)



Alternatives

Alternatively, we can choose not to support this feature, but I can imagine other (more specific) use-cases where this feature is useful than stated above, so if nothing else, at least I'm interested how many people would find this useful.
Additional context
In research, training sessions might be desirably long and interactive. Currently they are treated as atomic operations from user perspective, where one needs to think ahead of everything they will want to examine about their model during training. To add something requires adjustment of the logging pipeline and starting the training session from scratch.
It goes without saying that lots of things are desirable to examine sparsely. The constraint to have all of this handled via prepared logging routines enforces regularity of examination along with need to determine appropriate intervals for these examinations ahead of time, which does not seem too research-friendly.
Alternative is to start the training over each time, which is what most people probably do currently, and might suffice as long as you're not logging scalars / images / metrics regularly with training time-stamps (resetting of which causes huge mess in TensorBoard for example).

Discussion
The extent and realization of this change are open to further considerations.
Stemming from Slack: @williamFalcon @neggert let's discuss here.