How to redefine optimizer for pl.LightningModule?
The great advantage and convenient feature of PyTorch-Lightning over vanilla Pytorch is the more convenient interface for optimization loops.  Instead of reinventing the wheel every time with manual train and test function, one can just define
training_step and validation_step, e.t.c.
The setup of the optimizer is done in the configure_optimizers method https://pytorch-lightning.readthedocs.io/en/latest/common/lightning_module.html#configure-optimizers, where one can create optimizers and schedulers.
However, it seems, that this choice fixes the optimizer properties in the definition of the class, and often you would like to try several optimizers and change their params. One way to do it - is to pass additional arguments to the constructor of
pl.LightningModule to be processed in configure_optimizers, where an appropriate optimizer will be chosen.
Also one may try to access directly the optimizers outside the class, but probably it is an unsafe approach.
Is there some alternative way to reconfigure optimizer, like in Tensorflow, where models have complie method https://www.tensorflow.org/api_docs/python/tf/keras/Model#compile?
Hi @Godofnothing ,
I am afraid there is no clean way to do so. What I personally do to hack around this (if I need it) is that I have arguments to the model that define what is to be returned by configure_optimizers and then call self.trainer.accelerator.setup_optimizers(self.trainer) in any of the other hooks. That makes sure configure_optimizers is called again and converts everything to the correct types. Inside your configure_optimizers function you would have to handle the creation of the optimizers for the different stages in training then.