How can one use an external optimizer with LightningCLI?
I would like to use Adafactor as my optimizer with LightningCLI. I've tried the method described in the documentation for custom optimizers but it didn't work. Can anybody tell me how they would train a model with this optimizer using LightningCLI?
Hi! I got it to work in the meantime. I added this to the main file where I call CLI:
import transformers
from pytorch_lightning.utilities.cli import OPTIMIZER_REGISTRY

@OPTIMIZER_REGISTRY
class Adafactor(transformers.Adafactor):
    def __init__(self, *args: Any, **kwargs: Any) -> None:
        super().__init__(*args, **kwargs)

The main issue was in the config file---apparently one needs to write:
optimizer:
  class_path: __main__.Adafactor

instead of:
optimizer:
  class_path: Adafactor

Doing the former, I got it to work.
By the way, is there a way to have the optimizer register in a separate file than the one that calls CLI?