How to ignore certain "parameters" with model checkpointing?
I have some data that I store on my LightningModule during validation.  I want to prevent this from being saved by the model checkpoint.  They are not actually parameters and do not affect the state at all.  I want to maintain other parts of the state, I don't want to use weights only.
Is it possible to do this?
Hey @jmerkow,
The checkpoint is generated from the dump_checkpoint function of the CheckpointConnector. One of the latest hooks called is lightning_module.on_save_checkpoint(checkpoint) here: https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/trainer/connectors/checkpoint_connector.py#L386
This is actually done in-place.
Therefore, you could do the following.
class MyModel(LightningModule):

    def on_save_checkpoint(self, checkpoint):
        # pop the keys you are not interested by
        checkpoint