[RFC] Create a ModelCheckpointBase callback
🚀 Feature
Create a ModelCheckpointBase callback, and have the existing checkpoint callback extend it
Motivation
The model checkpoint callback is growing in complexity. Features that have been recently added or will soon be proposed:

Checkpoint every n train steps: #6146
Checkpointing using a time-based interval during training: #6286
Checkpointing on train end to fix this hack

  
    
      pytorch-lightning/pytorch_lightning/trainer/training_loop.py
    
    
        Lines 152 to 163
      in
      680e83a
    
  
  
    

        
          
           def check_checkpoint_callback(self, should_update, is_last=False): 
        

        
          
               # TODO bake this logic into the ModelCheckpoint callback 
        

        
          
               if should_update and self.trainer.checkpoint_connector.has_trained: 
        

        
          
                   callbacks = self.trainer.checkpoint_callbacks 
        

        
          
            
        

        
          
                   if is_last and any(cb.save_last and cb.verbose for cb in callbacks): 
        

        
          
                       rank_zero_info("Saving latest checkpoint...") 
        

        
          
            
        

        
          
                   model = self.trainer.lightning_module 
        

        
          
            
        

        
          
                   for cb in callbacks: 
        

        
          
                       cb.on_validation_end(self.trainer, model) 
        
    
  



The decision was made in #6146 to keep these triggers mutually exclusive, at least based on the phase they run in. Why? It's very hard to get the state management right. For instance, the monitor might be added for something that's available only during validation, but the checkpoint callback is configured to run during training too, and crashes when it tries to look up the monitor key in the available metrics for tracking. Tracking top-K models and scores is another huge pain. Supporting multiple monitor metrics on top of this is another beast.
cc @Borda @carmocca @awaelchli @ninginthecloud @jjenniferdai @rohitgr7
Pitch
Move the existing logic for the following into a base class:

Core saving functionality
Top-K model management
formatting checkpoint names
validation (though sub-classes should override this)

And have thin wrappers on top which extend this class and implement callback hook(s) for when to save the checkpoint.
Alternatives
The checkpoint callback gets bigger and bigger as we add more features to it.
Additional context