HPC Save Writes Multiple Checkpoints
🐛 Bug
Currently the hpc_save function (
  
    
      pytorch-lightning/pytorch_lightning/trainer/connectors/checkpoint_connector.py
    
    
         Line 201
      in
      6e8721e
    
  
  
    

        
          
           def hpc_save(self, folderpath: str, logger): 
        
    
  

) doesn't respect the behavior of save_last meaning that only one checkpoint should be written. This means that every time the job is preempted (on slurm) it writes another checkpoint which caused me to run out of disk space recently.
To Reproduce
This can't exactly be reproduced using the requested BoringModel method, it requires a cluster (I know for sure slurm will repro this), set a short timeout, and run. Each time the limit is reached there will be a new checkpoint written. If this should be controlled separately from the existing save_last flag, then another flag should be introduced. This should be an easy fix and I'd be happy to PR it if the owners are agreeable to the solution.
Expected behavior
Only one checkpoint is written.
Environment

CUDA:
- GPU:
- available:         False
- version:           10.2
Packages:
- numpy:             1.20.1
- pyTorch_debug:     False
- pyTorch_version:   1.7.1
- pytorch-lightning: 1.2.0
- tqdm:              4.57.0
System:
- OS:                Linux
- architecture:
- 64bit
- ELF
- processor:         x86_64
- python:            3.8.1
- version:           #1 SMP Thu Jan 21 16:15:07 EST 2021