Saving and loading HF transformer model fine tuned with PL?
I am fine-tuning hugging face transformer models, essentially exactly as shown in the following example found in the pytorch lightning docs:
https://pytorch-lightning.readthedocs.io/en/latest/notebooks/lightning_examples/text-transformers.html
Where we instantiate the LightningModule doing something like this:
class GLUETransformer(LightningModule):

    def __init__(self, ... ):
        super().__init__()
        self.config = AutoConfig.from_pretrained(model_name_or_path, num_labels=num_labels)
        self.model = AutoModelForSequenceClassification.from_pretrained(
            model_name_or_path, config=self.config
        )
But I have been confused about how I should be saving and loading checkpoints.
When saving checkpoints, should I be using
mymodel.model.save_pretrained("model_save_dir"),
and reloading from this checkpoint using
AutoModelForSequenceClassification.from_pretrained("model_save_dir"),
or saving with
trainer.save_checkpoint("model_save_dir/checkpoint.ckpt"),
and reloading with
GLUETransformer.load_from_checkpoint("model_save_dir/checkpoint.ckpt")?
Dear @brijow,
You should be using the second approach. An even better one would be to rely on ModelCheckpoint to save the checkpoints and provide Trainer(resume_from_checkpoint=...) for reloading all the states.
Best,
T.C