Repository: Lightning-AI/pytorch-lightning
Pull request: Add support for optimizers and learning rate schedulers to LightningCLI
## What does this PR do?

New feature for LightningCLI to make optimizers and learning rate schedulers easily configurable. It implements the proposal mentioned in https://github.com/PyTorchLightning/pytorch-lightning/issues/7576#issuecomment-852387071 except for minor differences required during development. How it should be used is explained in lightning_cli.rst.

This PR does not close #7576.

## Before submitting
- [x] Was this discussed/approved via a GitHub issue? (not for typos and docs)
- [x] Did you read the [contributor guideline](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/.github/CONTRIBUTING.md), **Pull Request** section?
- [x] Did you make sure your PR does only one thing, instead of bundling different changes together?
- [x] Did you make sure to update the documentation with your changes? (if necessary)
- [x] Did you write any new necessary tests? (not for typos and docs)
- [x] Did you verify new and existing tests pass locally with your changes?
- [x] Did you update the [CHANGELOG](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/CHANGELOG.md)? (not for typos, docs, test updates, or internal minor changes/refactorings)

<!-- For CHANGELOG separate each item in the unreleased section by a blank line to reduce collisions -->

## PR review
Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read [Review guidelines](https://github.com/PyTorchLightning/pytorch-lightning/wiki/Review-guidelines). In short, see the following bullet-list:

 - [x] Is this pull request ready for review? (if not, please submit in draft mode)
 - [x] Check that all items from **Before submitting** are resolved
 - [x] Make sure the title is self-explanatory and the description concisely explains the PR
 - [x] Add labels and milestones (and optionally projects) to the PR so it can be classified

## Did you have fun?
Make sure you had fun coding 🙃


Problem statement:
Easier way to configure optimizers and schedulers in the CLI
## 🚀 Feature
Right now if you want to work with multiple optimizers and/or learning rate schedulers you need to write a whole bunch of ugly boilerplate.  I suggest that the LightningCLI in one way or another to enable that sort of optimizer configuration code in a clean way, which would take care of most use cases.

### Motivation

Right now I have the following code in `configure_optimizers` callback:
```python
    # Configuration.  Add more for learning schedulers, etc.?
    def configure_optimizers(self):
        if self.hparams.optimizer == "Adam":
            optimizer = torch.optim.Adam(
                self.parameters(),
                lr=self.hparams.learning_rate,
                weight_decay=self.hparams.weight_decay,
            )
        elif self.hparams.optimizer == "SGD":
            # Left out the momentum options for now
            optimizer = torch.optim.SGD(
                self.parameters(),
                lr=self.hparams.learning_rate,
                weight_decay=self.hparams.weight_decay,
            )
        elif self.hparams.optimizer == "LBFGS":
            optimizer = torch.optim.LBFGS(
                self.parameters(),
                # or can have self.hparams.learning_rate with warning if too low.
                lr=1,
                tolerance_grad=1e-5,  # can add to parameters if useful.
                tolerance_change=1e-9,  # can add to parameters if useful.
            )
        else:
            print("Invalid optimizer.  See --help")
            sys.exit()

        if self.hparams.lr_scheduler == None:
            return optimizer

        # Setup the scheduler
        scheduler = None
        try:
            if self.hparams.lr_scheduler == "StepLR":
                step_size = self.hparams.StepLR_step_size
                gamma = self.hparams.StepLR_gamma
                scheduler = torch.optim.lr_scheduler.StepLR(
                    optimizer, step_size, gamma)
            elif self.hparams.lr_scheduler == "ReduceLROnPlateau":
                factor = self.hparams.lr_factor
                patience = self.hparams.lr_patience
                scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
                    optimizer, mode="min", factor=factor, patience=patience
                )
                return {
                    "optimizer": optimizer,
                    "lr_scheduler": scheduler,
                    "monitor": self.hparams.LRScheduler_metric,
                }
            elif self.hparams.lr_scheduler == "OneCycleLR":
                max_lr = self.hparams.OneCycleLR_max_lr
                epochs = self.hparams.OneCycleLR_epochs
                steps_per_epoch = self.hparams.train_trajectories * (
                    self.hparams.T + 1
                )  # I think this is the same as len(train_data_loader)? #TODO
                scheduler = torch.optim.lr_scheduler.OneCycleLR(
                    optimizer,
                    max_lr=max_lr,
                    epochs=epochs,
                    steps_per_epoch=steps_per_epoch,
                )
        except:
            print("Invalid scheduler configuration.  See --help")
            raise
        return [optimizer], [scheduler]
```
And the commandline configuration looks like
```python
        parser.add_argument("--learning_rate", type=float, default=5e-3)
        parser.add_argument("--LRScheduler_metric",
                            type=str, default="val_loss")
        parser.add_argument(
            "--lr_scheduler",
            default="ReduceLROnPlateau",
            help="Learning rate scheduler name, it can be StepLR, ReduceLROnPlateau, or, OneCycleLR. Defaults to None.",
        )
        parser.add_argument(
            "--optimizer",
            type=str,
            default="Adam",
            help="Choice optimizer: Adam, SGD, or LBFGS.",
        )
        parser.add_argument(
            "--StepLR_step_size",
            type=int,
            default=30,  # TODO there was no actual default in the source code
            help="Step size for StepLR scheduler. Defaults to 30.",
        )
        parser.add_argument(
            "--StepLR_gamma",
            type=float,
            default=0.1,
            help="Gamma for StepLR scheduler. Defaults to 0.1 ",
        )
        # plateau based learning rate
        parser.add_argument(
            "--lr_factor",
            type=float,
            default=0.1,
            help="Factor by which the learning rate will be reduced. new_lr = lr * factor.",
        )
        parser.add_argument(
            "--lr_patience",
            type=int,
            default=5,
            help="Number of epochs with no improvement after which learning rate will be reduced. For example, if patience = 2, then we will ignore the first 2 epochs with no improvement, and will only decrease the LR after the 3rd epoch if the loss still hasn’t improved then",
        )
        parser.add_argument(
            "--OneCycleLR_max_lr",
            type=float,  # TODO no default on the source code
            default=0.1,
            help="Upper learning rate boundary in the cycle. Note that it cannot be a list",
        )
        parser.add_argument(
            "--OneCycleLR_epochs",
            type=int,
            default=10,
            help="The number of epochs to train for. This is used along with steps_per_epoch in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Default: None",
        )
```
or maybe even more.... 

### Pitch

Not exactly sure how you are planning to do recursive configuration stuff, but I can imagine the CLI having a baseline `configure_optimizers` on its own so that the user doesn't need to write it for simple patterns.  Then maybe have defaults in the CLI arguments to have things like
```python
python train.py --model.learning_rate 1e-4 --model.lr_scheduler.type ReduceLROnPlateau --model.lr_scheduler.factor 0.1
```
and
```python
python train.py --model.learning_rate 1e-4 --model.lr_scheduler.type ReduceLROnPlateau --model.lr_scheduler.factor 0.1 --model.optimizer.type Adam --model.optimizer.weight_decay 1e-5
```
or whatever.   And the user just wouldn't implement a `configure_optimizers` or at least would have something simple to call.

Writing this out as a configuration, I have in mind that the user could have something like
```yaml
model:
  decoder_layers:
  - 2
  - 4
  encoder_layers: 12
optimizer:
  Adam:
    weight_decay: 1e-5
lr_scheduler:
   ReduceLROnPlateau:
     factor: 0.1     
trainer:
  accelerator: null
  accumulate_grad_batches: 1
  amp_backend: native
  amp_level: O2
```
or maybe instead:
```yaml
model:
  decoder_layers:
  - 2
  - 4
  encoder_layers: 12
  optimizer:
    Adam:
      weight_decay: 1e-5
  lr_scheduler:
     ReduceLROnPlateau:
       factor: 0.1     
trainer:
  accelerator: null
  accumulate_grad_batches: 1
  amp_backend: native
  amp_level: O2
```

etc.

### Alternatives

See the above monstrosity of what is done now.

Hints from issue discussion:
cc @carmocca @mauvilsa 
> but I can imagine the CLI having a baseline configure_optimizers on its own so that the user doesn't need to write it for simple patterns.

The CLI should not modify or override any code inside the `LightningModule`. To accomplish what you want, there are several ways you could go about it. What I would do is:

```python
from enum import Enum

import torch

import pytorch_lightning as pl
from pytorch_lightning.utilities.cli import LightningCLI


# could be just strings but enum forces the set of choices
class OptimizerEnum(str, Enum):
    Adam = "Adam"
    SGD = "SGD"
    LBFGS = "LBFGS"


class LRSchedulerEnum(str, Enum):
    ...


class MyModel(pl.LightningModule):
    def configure_optimizer(self, optimizer: OptimizerEnum, learning_rate: float = 1e-3, weight_decay: float = 0.0):
        if optimizer == "Adam":
             return torch.optim.Adam(
                self.parameters(),
                lr=learning_rate,
                weight_decay=weight_decay,
            )
        elif optimizer == "SGD":
            return torch.optim.SGD(
                self.parameters(),
                lr=learning_rate,
                weight_decay=weight_decay,
            )
        elif optimizer == "LBFGS":
            return torch.optim.LBFGS(self.parameters(), lr=1)
        raise ValueError(f"Invalid optimizer {optimizer}. See --help")

    def configure_scheduler(self, optimizer: torch.optim.Optimizer, lr_scheduler: LRSchedulerEnum):
        # same structure as `configure_optimizer`
        ...

    def configure_optimizers(self):
        optimizer = self.configure_optimizer(
            self.hparams.optimizer,
            learning_rate=self.hparams.learning_rate,
            weight_decay=self.hparams.weight_decay,
        )
        if self.hparams.lr_scheduler is None:
            return optimizer
        scheduler = self.configure_scheduler(optimizer, self.hparams.lr_scheduler)
        return [optimizer], [scheduler]


class MyLightningCLI(LightningCLI):
    def add_arguments_to_parser(self, parser):
        # add the `configure_optimizer` arguments under the key optimizer
        parser.add_method_arguments(self.model_class, "configure_optimizer", nested_key="optimizer")
        # add the `configure_scheduler` arguments under the key lr_scheduler
        parser.add_method_arguments(self.model_class, "configure_scheduler", nested_key="lr_scheduler", skip={"optimizer"})


MyLightningCLI(MyModel)
```

> And the commandline configuration looks like

Now all those defaults and types are part of the function signatures
And the help messages part of the docstrings (not included in the ported code)
I know this is "working as intended" but I also think you see missing a key feature. The code given doesnt tackle the bigger issue of configuring the learning rate schedulers with their different parameters, etc. I am not even sure how much wrapper code that would take or how to write it.

Also, the optimizers don't always have the same options so I think architecture for your code example just wouldn't work with more generality. Not to mention, how are new optimizers that lightning adds going to be supported? It would require basically hardcoding something in user code for the dispatching  rather than an OO design within lightning or lightning CLI code.

Something is missing. @zippeurfou was there a structure that works for this with hydra?

This is very un-lightning in the quantity of copy paste boilerplate required for something almost everyone would want the same features work. I am positive  there is something which would require zero code for users who want the CLI to use config for standard optimizer and scheduler usage 
@jlperla I agree this is a missing feature. One of the objectives of LightningCLI is that it can be used without people having to learn about a configuration framework. This at least for most common cases since the possibilities of what people might need are endless.

I have some ideas and will describe them here in a bit. Though I will not manage to do this today.
Hey @jlperla you have good points and I agree with you that doing "if" statement with manually linking your hparams for each of the different options for optimizer and LR scheduler is not a great experience. 
@mauvilsa have some idea let's wait and see if it works.


Thanks. Yeah, the other consideration is that if you can move stuff into your libraries in one form or another, it makes user code far less fragile to any refactoring or additional pytorch optimizer options. Otherwise there will be a half dozen, almost identical versions of the same code out there...none of them really customized for the user 
I give here some initial ideas to start the discussion. Some objectives could be:

- Be possible to add groups in the config for settings of optimizers/schedulers. The groups are dicts that can be used to instantiate, no automatic instantiation.
- There could be one or more independent optimizers/schedulers.
- A single optimizer/scheduler group can be configured to accept settings for one class or multiple classes using `class_path` and `init_args` to follow the same pattern that LightningCLI already uses.
- There would be a function to ease instantiation. This is particularly important when a class is defined using `class_path` and `init_args`.

My initial idea is the following (not yet fully thought through). There would be a method in `LightningArgumentParser` to add groups in the config, e.g. `add_optimizer_args`. This method could be used to configure a single optimizer class, e.g. `parser.add_optimizer_args(Adam, 'optimizer')`, or multiple, e.g. `parser.add_optimizer_args(BaseOptimizer, 'optimizer', subclass_mode=True)` or `parser.add_optimizer_args(Union[Adam, SGD], 'optimizer')`. These groups would need to be linked to arguments of the module. Would be up to the user to store them in the class and then use them in `configure_optimizers` to instantiate.
> Would be up to the user to store them in the class and then use them in configure_optimizers to instantiate.

I think I would have to get a sense of what you mean by that, but sounds like boilerplate to me :-)  The source of the difficulty in all of this stuff is writing the code to configure the class and store the parameters.  I still don't understand why the user would need to write a single line of code for this since it is always the same stuff (except for weird cases where you need two optimizers/etc. but users can then do what they want for that).  Why can't PL and the CLI could do everything internally and have things nice and consistent?

But... the stuff in the parser groups seem to make sense to me and the grouping "dicts" as constructors for the optimizers and scheudlers also make plenty of sense.  Now I think it just needs to be taken to its inevitable conclusion of an option which formalizes that stuff inside of PL itself :-)

> > Would be up to the user to store them in the class and then use them in configure_optimizers to instantiate.
> 
> I think I would have to get a sense of what you mean by that, but sounds like boilerplate to me :-) The source of the difficulty in all of this stuff is writing the code to configure the class and store the parameters. I still don't understand why the user would need to write a single line of code for this since it is always the same stuff (except for weird cases where you need two optimizers/etc. but users can then do what they want for that). Why can't PL and the CLI could do everything internally and have things nice and consistent?

You are right. It wouldn't be difficult to automatically implement the model's `configure_optimizers` in the case of a single optimizer and scheduler. It could be as simple as

```python
parser.add_optimizer_args(Adam)  # 'optimizer' key would be default
parser.add_scheduler_args(StepLR)  # 'scheduler' key would be default
```

In the case of multiple optimizers/schedulers it becomes manual. The parser could be configured like

```python
parser.add_optimizer_args(Adam, 'optimizer1', link_to='model.optim1')
parser.add_optimizer_args(Adam, 'optimizer2', link_to='model.optim2')
```

Then the user needs to store the parameters and implement `configure_optimizers`.
>It wouldn't be difficult to automatically implement the model's configure_optimizers in the case of a single optimizer and scheduler

I am not sure I completely follow, but if it means I can have a configurable and swappable single optimizer and single scheduler in my code without any manual boilerplate, then I am happy.  But just to confirm, the key here is that it is all swappable and configurable in the CLI wihtout me needing to manually write a bunch of `elif` etc. statements and manipulating arguments in my model.

That is, if you are saying that I have to go `parser.add_scheduler_args(StepLR)` in my code, then it seems like you are forcing me to choose a scheduler at parsetime?  Whereas what we really want is to let the users swap out different parsers and schedulers in the CLI or json configuration so we can try different things in different runs to see what works.  I still see no reason that all of that stuff can't be basically automatic for single optimizer/single scheudler setups.  At least if the user uses the
```
class MyModel(pl.LightningModule): 
    def __init__(self, **kwargs): 
        super().__init__()
        self.save_hyperparameters()
``` 
Then if the user doesn't provide a `def configure_optimizers(self):` method, the default one could look into the module to see if it has the standard thing stored.  e.g. maybe it stores a `optimizer_dict` and `lr_scheduler_dict` or whatever in it, which gets populated  in the CLI config from 
```yaml
model:
  decoder_layers:
  - 2
  - 4
  encoder_layers: 12
  optimizer:
    Adam:
      weight_decay: 1e-5
  lr_scheduler:
     ReduceLROnPlateau:
       factor: 0.1     
trainer:
  accelerator: null
  accumulate_grad_batches: 1
  amp_backend: native
  amp_level: O2
```
And if they wanted to try a different optimizer, they just pass in a different YAML or CLI.
```
  optimizer:
     SGD:
      weight_decay: 1e-5
```
etc.
Etc.    The user wouldn't need to write any code specific to the optimizers or lr_schedulers if they want to just use the basic method which looks for variables associated with those?

Then you could either use dispatching on the type (e.g. the `Adam`) or fill things in with a dictionary and a bunch of internal `elif`.

Commit message: - Consider the case of ReduceLROnPlateau.
- Change manual configure_optimizers example to return dict.

Patch to review:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 666438b872b9c..d78142d24e3c1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -111,6 +111,9 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Add support for calling scripts using the module syntax (`python -m package.script`) ([#8073](https://github.com/PyTorchLightning/pytorch-lightning/pull/8073))
 
 
+- Add support for optimizers and learning rate schedulers to `LightningCLI` ([#8093](https://github.com/PyTorchLightning/pytorch-lightning/pull/8093))
+
+
 ### Changed
 
 
diff --git a/docs/source/common/lightning_cli.rst b/docs/source/common/lightning_cli.rst
index 1fd313244383b..82eb5e5fe3f3c 100644
--- a/docs/source/common/lightning_cli.rst
+++ b/docs/source/common/lightning_cli.rst
@@ -1,6 +1,7 @@
 .. testsetup:: *
     :skipif: not _JSONARGPARSE_AVAILABLE
 
+    import torch
     from unittest import mock
     from typing import List
     from pytorch_lightning.core.lightning import LightningModule
@@ -385,7 +386,7 @@ instantiating the trainer class can be found in :code:`self.config['trainer']`.
 
 
 Configurable callbacks
-~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^
 
 As explained previously, any callback can be added by including it in the config via :code:`class_path` and
 :code:`init_args` entries. However, there are other cases in which a callback should always be present and be
@@ -417,7 +418,7 @@ To change the configuration of the :code:`EarlyStopping` in the config it would
 
 
 Argument linking
-~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^
 
 Another case in which it might be desired to extend :class:`~pytorch_lightning.utilities.cli.LightningCLI` is that the
 model and data module depend on a common parameter. For example in some cases both classes require to know the
@@ -470,3 +471,82 @@ Instantiation links are used to automatically determine the order of instantiati
     The linking of arguments can be used for more complex cases. For example to derive a value via a function that takes
     multiple settings as input. For more details have a look at the API of `link_arguments
     <https://jsonargparse.readthedocs.io/en/stable/#jsonargparse.core.ArgumentParser.link_arguments>`_.
+
+
+Optimizers and learning rate schedulers
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Optimizers and learning rate schedulers can also be made configurable. The most common case is when a model only has a
+single optimizer and optionally a single learning rate scheduler. In this case the model's
+:class:`~pytorch_lightning.core.lightning.LightningModule` could be left without implementing the
+:code:`configure_optimizers`, method since it is normally always the same and just adds boilerplate. The following code
+snippet shows how to implement it:
+
+.. testcode::
+
+    import torch
+    from pytorch_lightning.utilities.cli import LightningCLI
+
+    class MyLightningCLI(LightningCLI):
+
+        def add_arguments_to_parser(self, parser):
+            parser.add_optimizer_args(torch.optim.Adam)
+            parser.add_lr_scheduler_args(torch.optim.lr_scheduler.ExponentialLR)
+
+    cli = MyLightningCLI(MyModel)
+
+With this the :code:`configure_optimizers` method is automatically implemented and in the config the :code:`optimizer`
+and :code:`lr_scheduler` groups would accept all of the options for the given classes, in this example :code:`Adam` and
+:code:`ExponentialLR`.
+
+There is also the possibility of selecting among multiple classes by giving them as a tuple. For example:
+
+.. testcode::
+
+    class MyLightningCLI(LightningCLI):
+
+        def add_arguments_to_parser(self, parser):
+            parser.add_optimizer_args((torch.optim.SGD, torch.optim.Adam))
+
+In this case in the config the :code:`optimizer` group instead of having directly init settings, it should specify
+:code:`class_path` and optionally :code:`init_args`. Sub-classes of the classes in the tuple would also be accepted.
+
+The automatic implementation of :code:`configure_optimizers` can be disabled by linking the configuration group. An
+example can be :code:`ReduceLROnPlateau` which requires to specify a monitor. This would be:
+
+.. testcode::
+
+    from pytorch_lightning.utilities.cli import instantiate_class, LightningCLI
+
+    class MyModel(LightningModule):
+
+        def __init__(self, optimizer_init: dict, lr_scheduler_init: dict):
+            super().__init__()
+            self.optimizer_init = optimizer_init
+            self.lr_scheduler_init = lr_scheduler_init
+
+        def configure_optimizers(self):
+            optimizer = instantiate_class(self.parameters(), self.optimizer_init)
+            scheduler = instantiate_class(optimizer, self.lr_scheduler_init)
+            return {"optimizer": optimizer, "lr_scheduler": scheduler, "monitor": "metric_to_track"}
+
+    class MyLightningCLI(LightningCLI):
+
+        def add_arguments_to_parser(self, parser):
+            parser.add_optimizer_args(
+                torch.optim.Adam,
+                link_to='model.optimizer_init',
+            )
+            parser.add_lr_scheduler_args(
+                torch.optim.lr_scheduler.ReduceLROnPlateau,
+                link_to='model.lr_scheduler_init',
+            )
+
+    cli = MyLightningCLI(MyModel)
+
+For both possibilities of using :meth:`pytorch_lightning.utilities.cli.LightningArgumentParser.add_optimizer_args` with
+a single class or a tuple of classes, the value given to :code:`optimizer_init` will always be a dictionary including
+:code:`class_path` and :code:`init_args` entries. The function
+:func:`~pytorch_lightning.utilities.cli.instantiate_class` takes care of importing the class defined in
+:code:`class_path` and instantiating it using some positional arguments, in this case :code:`self.parameters()`, and the
+:code:`init_args`. Any number of optimizers and learning rate schedulers can be added when using :code:`link_to`.
diff --git a/pytorch_lightning/utilities/cli.py b/pytorch_lightning/utilities/cli.py
index 387bbdeb85319..d5a1b7dbae166 100644
--- a/pytorch_lightning/utilities/cli.py
+++ b/pytorch_lightning/utilities/cli.py
@@ -12,8 +12,13 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import os
+import warnings
 from argparse import Namespace
-from typing import Any, Dict, List, Optional, Type, Union
+from types import MethodType
+from typing import Any, Callable, Dict, List, Optional, Tuple, Type, Union
+
+from torch.optim import Optimizer
+from torch.optim.lr_scheduler import _LRScheduler, ReduceLROnPlateau
 
 from pytorch_lightning.callbacks import Callback
 from pytorch_lightning.core.datamodule import LightningDataModule
@@ -25,10 +30,14 @@
 _JSONARGPARSE_AVAILABLE = _module_available("jsonargparse")
 if _JSONARGPARSE_AVAILABLE:
     from jsonargparse import ActionConfigFile, ArgumentParser, set_config_read_mode
+    from jsonargparse.util import import_object
     set_config_read_mode(fsspec_enabled=True)
 else:
     ArgumentParser = object
 
+LRScheduler = (_LRScheduler, ReduceLROnPlateau)
+LRSchedulerType = Union[Type[_LRScheduler], Type[ReduceLROnPlateau]]
+
 
 class LightningArgumentParser(ArgumentParser):
     """Extension of jsonargparse's ArgumentParser for pytorch-lightning"""
@@ -49,6 +58,7 @@ def __init__(self, *args: Any, parse_as_dict: bool = True, **kwargs: Any) -> Non
             '--config', action=ActionConfigFile, help='Path to a configuration file in json or yaml format.'
         )
         self.callback_keys: List[str] = []
+        self.optimizers_and_lr_schedulers: Dict[str, Tuple[Union[Type, Tuple[Type, ...]], str]] = {}
 
     def add_lightning_class_args(
         self,
@@ -76,6 +86,64 @@ def add_lightning_class_args(
             instantiate=not issubclass(lightning_class, Trainer),
         )
 
+    def add_optimizer_args(
+        self,
+        optimizer_class: Union[Type[Optimizer], Tuple[Type[Optimizer], ...]],
+        nested_key: str = 'optimizer',
+        link_to: str = 'AUTOMATIC',
+    ) -> None:
+        """
+        Adds arguments from an optimizer class to a nested key of the parser
+
+        Args:
+            optimizer_class: Any subclass of torch.optim.Optimizer.
+            nested_key: Name of the nested namespace to store arguments.
+            link_to: Dot notation of a parser key to set arguments or AUTOMATIC.
+        """
+        if isinstance(optimizer_class, tuple):
+            assert all(issubclass(o, Optimizer) for o in optimizer_class)
+        else:
+            assert issubclass(optimizer_class, Optimizer)
+        kwargs = {
+            'instantiate': False,
+            'fail_untyped': False,
+            'skip': {'params'},
+        }
+        if isinstance(optimizer_class, tuple):
+            self.add_subclass_arguments(optimizer_class, nested_key, required=True, **kwargs)
+        else:
+            self.add_class_arguments(optimizer_class, nested_key, **kwargs)
+        self.optimizers_and_lr_schedulers[nested_key] = (optimizer_class, link_to)
+
+    def add_lr_scheduler_args(
+        self,
+        lr_scheduler_class: Union[LRSchedulerType, Tuple[LRSchedulerType, ...]],
+        nested_key: str = 'lr_scheduler',
+        link_to: str = 'AUTOMATIC',
+    ) -> None:
+        """
+        Adds arguments from a learning rate scheduler class to a nested key of the parser
+
+        Args:
+            lr_scheduler_class: Any subclass of torch.optim.lr_scheduler.{_LRScheduler, ReduceLROnPlateau}.
+            nested_key: Name of the nested namespace to store arguments.
+            link_to: Dot notation of a parser key to set arguments or AUTOMATIC.
+        """
+        if isinstance(lr_scheduler_class, tuple):
+            assert all(issubclass(o, LRScheduler) for o in lr_scheduler_class)
+        else:
+            assert issubclass(lr_scheduler_class, LRScheduler)
+        kwargs = {
+            'instantiate': False,
+            'fail_untyped': False,
+            'skip': {'optimizer'},
+        }
+        if isinstance(lr_scheduler_class, tuple):
+            self.add_subclass_arguments(lr_scheduler_class, nested_key, required=True, **kwargs)
+        else:
+            self.add_class_arguments(lr_scheduler_class, nested_key, **kwargs)
+        self.optimizers_and_lr_schedulers[nested_key] = (lr_scheduler_class, link_to)
+
 
 class SaveConfigCallback(Callback):
     """Saves a LightningCLI config to the log_dir when training starts
@@ -192,11 +260,13 @@ def __init__(
         self.init_parser()
         self.add_core_arguments_to_parser()
         self.add_arguments_to_parser(self.parser)
+        self.link_optimizers_and_lr_schedulers()
         self.parse_arguments()
         if self.config['seed_everything'] is not None:
             seed_everything(self.config['seed_everything'], workers=True)
         self.before_instantiate_classes()
         self.instantiate_classes()
+        self.add_configure_optimizers_method_to_model()
         self.prepare_fit_kwargs()
         self.before_fit()
         self.fit()
@@ -228,6 +298,17 @@ def add_arguments_to_parser(self, parser: LightningArgumentParser) -> None:
             parser: The argument parser object to which arguments can be added
         """
 
+    def link_optimizers_and_lr_schedulers(self) -> None:
+        """Creates argument links for optimizers and lr_schedulers that specified a link_to"""
+        for key in self.parser.optimizers_and_lr_schedulers.keys():
+            class_type, link_to = self.parser.optimizers_and_lr_schedulers[key]
+            if link_to != 'AUTOMATIC':
+                if isinstance(class_type, tuple):
+                    self.parser.link_arguments(key, link_to)
+                else:
+                    add_class_path = _add_class_path_generator(class_type)
+                    self.parser.link_arguments(key, link_to, compute_fn=add_class_path)
+
     def parse_arguments(self) -> None:
         """Parses command line arguments and stores it in self.config"""
         self.config = self.parser.parse_args()
@@ -260,6 +341,60 @@ def instantiate_trainer(self) -> None:
             self.config_init['trainer']['callbacks'].append(config_callback)
         self.trainer = self.trainer_class(**self.config_init['trainer'])
 
+    def add_configure_optimizers_method_to_model(self) -> None:
+        """Adds to the model an automatically generated configure_optimizers method
+
+        If a single optimizer and optionally a scheduler argument groups are added to the parser as 'AUTOMATIC',
+        then a `configure_optimizers` method is automatically implemented in the model class.
+        """
+
+        def get_automatic(class_type: Type) -> List[str]:
+            automatic = []
+            for key, (base_class, link_to) in self.parser.optimizers_and_lr_schedulers.items():
+                if not isinstance(base_class, tuple):
+                    base_class = (base_class, )
+                if link_to == 'AUTOMATIC' and any(issubclass(c, class_type) for c in base_class):
+                    automatic.append(key)
+            return automatic
+
+        optimizers = get_automatic(Optimizer)
+        lr_schedulers = get_automatic(LRScheduler)
+
+        if len(optimizers) == 0:
+            return
+
+        if len(optimizers) > 1 or len(lr_schedulers) > 1:
+            raise RuntimeError(
+                f"`{self.__class__.__name__}.add_configure_optimizers_method_to_model` expects at most one optimizer "
+                f"and one lr_scheduler to be 'AUTOMATIC', but found {optimizers+lr_schedulers}."
+            )
+
+        if type(self.model).configure_optimizers != LightningModule.configure_optimizers:
+            warnings.warn(
+                f"`{self.model.__class__.__name__}.configure_optimizers` will be overridden by "
+                f"`{self.__class__.__name__}.add_configure_optimizers_method_to_model`."
+            )
+
+        optimizer_class = self.parser.optimizers_and_lr_schedulers[optimizers[0]][0]
+        optimizer_init = self.config_init.get(optimizers[0], {})
+        if not isinstance(optimizer_class, tuple):
+            optimizer_init = _global_add_class_path(optimizer_class, optimizer_init)
+        lr_scheduler_init = None
+        if lr_schedulers:
+            lr_scheduler_class = self.parser.optimizers_and_lr_schedulers[lr_schedulers[0]][0]
+            lr_scheduler_init = self.config_init.get(lr_schedulers[0], {})
+            if not isinstance(lr_scheduler_class, tuple):
+                lr_scheduler_init = _global_add_class_path(lr_scheduler_class, lr_scheduler_init)
+
+        def configure_optimizers(self: LightningModule) -> Union[Optimizer, Tuple[List[Optimizer], List[LRSchedulerType]]]:
+            optimizer = instantiate_class(self.parameters(), optimizer_init)
+            if not lr_scheduler_init:
+                return optimizer
+            lr_scheduler = instantiate_class(optimizer, lr_scheduler_init)
+            return [optimizer], [lr_scheduler]
+
+        self.model.configure_optimizers = MethodType(configure_optimizers, self.model)
+
     def prepare_fit_kwargs(self) -> None:
         """Prepares fit_kwargs including datamodule using self.config_init['data'] if given"""
         self.fit_kwargs = {'model': self.model}
@@ -275,3 +410,35 @@ def fit(self) -> None:
 
     def after_fit(self) -> None:
         """Implement to run some code after fit has finished"""
+
+
+def _global_add_class_path(class_type: Type, init_args: Dict[str, Any]) -> Dict[str, Any]:
+    return {
+        'class_path': class_type.__module__ + '.' + class_type.__name__,
+        'init_args': init_args,
+    }
+
+
+def _add_class_path_generator(class_type: Type) -> Callable[[Dict[str, Any]], Dict[str, Any]]:
+
+    def add_class_path(init_args: Dict[str, Any]) -> Dict[str, Any]:
+        return _global_add_class_path(class_type, init_args)
+
+    return add_class_path
+
+
+def instantiate_class(args: Union[Any, Tuple[Any, ...]], init: Dict[str, Any]) -> Any:
+    """Instantiates a class with the given args and init.
+
+    Args:
+        args: Positional arguments required for instantiation.
+        init: Dict of the form {"class_path":...,"init_args":...}.
+
+    Returns:
+        The instantiated class object.
+    """
+    args_class = import_object(init['class_path'])
+    kwargs = init.get('init_args', {})
+    if not isinstance(args, tuple):
+        args = (args, )
+    return args_class(*args, **kwargs)
diff --git a/requirements/extra.txt b/requirements/extra.txt
index 1dd8a5184ad8c..291813e05edcd 100644
--- a/requirements/extra.txt
+++ b/requirements/extra.txt
@@ -7,4 +7,4 @@ torchtext>=0.5
 # onnx>=1.7.0
 onnxruntime>=1.3.0
 hydra-core>=1.0
-jsonargparse[signatures]>=3.14.0
+jsonargparse[signatures]>=3.15.0
diff --git a/tests/utilities/test_cli.py b/tests/utilities/test_cli.py
index 4e37f511f0954..b672a2596b7cf 100644
--- a/tests/utilities/test_cli.py
+++ b/tests/utilities/test_cli.py
@@ -32,7 +32,7 @@
 from pytorch_lightning.callbacks import LearningRateMonitor, ModelCheckpoint
 from pytorch_lightning.plugins.environments import SLURMEnvironment
 from pytorch_lightning.utilities import _TPU_AVAILABLE
-from pytorch_lightning.utilities.cli import LightningArgumentParser, LightningCLI, SaveConfigCallback
+from pytorch_lightning.utilities.cli import instantiate_class, LightningArgumentParser, LightningCLI, SaveConfigCallback
 from pytorch_lightning.utilities.imports import _TORCHVISION_AVAILABLE
 from tests.helpers import BoringDataModule, BoringModel
 
@@ -614,3 +614,117 @@ def test_cli_config_overwrite(tmpdir):
         LightningCLI(BoringModel, trainer_defaults=trainer_defaults)
     with mock.patch('sys.argv', ['any.py']):
         LightningCLI(BoringModel, save_config_overwrite=True, trainer_defaults=trainer_defaults)
+
+
+def test_lightning_cli_optimizer(tmpdir):
+
+    class MyLightningCLI(LightningCLI):
+
+        def add_arguments_to_parser(self, parser):
+            parser.add_optimizer_args(torch.optim.Adam)
+
+    cli_args = [
+        f'--trainer.default_root_dir={tmpdir}',
+        '--trainer.max_epochs=1',
+    ]
+
+    with mock.patch('sys.argv', ['any.py'] + cli_args):
+        cli = MyLightningCLI(BoringModel)
+
+    assert len(cli.trainer.optimizers) == 1
+    assert isinstance(cli.trainer.optimizers[0], torch.optim.Adam)
+    assert len(cli.trainer.lr_schedulers) == 0
+
+
+def test_lightning_cli_optimizer_and_lr_scheduler(tmpdir):
+
+    class MyLightningCLI(LightningCLI):
+
+        def add_arguments_to_parser(self, parser):
+            parser.add_optimizer_args(torch.optim.Adam)
+            parser.add_lr_scheduler_args(torch.optim.lr_scheduler.ExponentialLR)
+
+    cli_args = [
+        f'--trainer.default_root_dir={tmpdir}',
+        '--trainer.max_epochs=1',
+        '--lr_scheduler.gamma=0.8',
+    ]
+
+    with mock.patch('sys.argv', ['any.py'] + cli_args):
+        cli = MyLightningCLI(BoringModel)
+
+    assert len(cli.trainer.optimizers) == 1
+    assert isinstance(cli.trainer.optimizers[0], torch.optim.Adam)
+    assert len(cli.trainer.lr_schedulers) == 1
+    assert isinstance(cli.trainer.lr_schedulers[0]['scheduler'], torch.optim.lr_scheduler.ExponentialLR)
+    assert cli.trainer.lr_schedulers[0]['scheduler'].gamma == 0.8
+
+
+def test_lightning_cli_optimizer_and_lr_scheduler_subclasses(tmpdir):
+
+    class MyLightningCLI(LightningCLI):
+
+        def add_arguments_to_parser(self, parser):
+            parser.add_optimizer_args((torch.optim.SGD, torch.optim.Adam))
+            parser.add_lr_scheduler_args((torch.optim.lr_scheduler.StepLR, torch.optim.lr_scheduler.ExponentialLR))
+
+    optimizer_arg = dict(
+        class_path='torch.optim.Adam',
+        init_args=dict(lr=0.01),
+    )
+    lr_scheduler_arg = dict(
+        class_path='torch.optim.lr_scheduler.StepLR',
+        init_args=dict(step_size=50),
+    )
+    cli_args = [
+        f'--trainer.default_root_dir={tmpdir}',
+        '--trainer.max_epochs=1',
+        f'--optimizer={json.dumps(optimizer_arg)}',
+        f'--lr_scheduler={json.dumps(lr_scheduler_arg)}',
+    ]
+
+    with mock.patch('sys.argv', ['any.py'] + cli_args):
+        cli = MyLightningCLI(BoringModel)
+
+    assert len(cli.trainer.optimizers) == 1
+    assert isinstance(cli.trainer.optimizers[0], torch.optim.Adam)
+    assert len(cli.trainer.lr_schedulers) == 1
+    assert isinstance(cli.trainer.lr_schedulers[0]['scheduler'], torch.optim.lr_scheduler.StepLR)
+    assert cli.trainer.lr_schedulers[0]['scheduler'].step_size == 50
+
+
+def test_lightning_cli_optimizers_and_lr_scheduler_with_link_to(tmpdir):
+
+    class MyLightningCLI(LightningCLI):
+
+        def add_arguments_to_parser(self, parser):
+            parser.add_optimizer_args(torch.optim.Adam, nested_key='optim1', link_to='model.optim1')
+            parser.add_optimizer_args((torch.optim.ASGD, torch.optim.SGD), nested_key='optim2', link_to='model.optim2')
+            parser.add_lr_scheduler_args(torch.optim.lr_scheduler.ExponentialLR, link_to='model.scheduler')
+
+    class TestModel(BoringModel):
+
+        def __init__(
+            self,
+            optim1: dict,
+            optim2: dict,
+            scheduler: dict,
+        ):
+            super().__init__()
+            self.optim1 = instantiate_class(self.parameters(), optim1)
+            self.optim2 = instantiate_class(self.parameters(), optim2)
+            self.scheduler = instantiate_class(self.optim1, scheduler)
+
+    cli_args = [
+        f'--trainer.default_root_dir={tmpdir}',
+        '--trainer.max_epochs=1',
+        '--optim2={"class_path": "torch.optim.SGD", "init_args": {"lr": 0.01}}',
+        '--lr_scheduler.gamma=0.2',
+    ]
+
+    with mock.patch('sys.argv', ['any.py'] + cli_args):
+        cli = MyLightningCLI(TestModel)
+
+    assert isinstance(cli.model.optim1, torch.optim.Adam)
+    assert isinstance(cli.model.optim2, torch.optim.SGD)
+    assert isinstance(cli.model.scheduler, torch.optim.lr_scheduler.ExponentialLR)


Code review comments:
diff_hunk: @@ -511,31 +511,41 @@ There is also the possibility of selecting among multiple classes by giving them
 In this case in the config the :code:`optimizer` group instead of having directly init settings, it should specify
 :code:`class_path` and optionally :code:`init_args`. Sub-classes of the classes in the tuple would also be accepted.
 
-For more complex cases in which there could more than one optimizer and/or learning rate scheduler, the automatic
-implementation of :code:`configure_optimizers` can be disabled by linking the configuration group. An example is:
+The automatic implementation of :code:`configure_optimizers` can be disabled by linking the configuration group. An
+example can be :code:`ReduceLROnPlateau` which requires to specify a monitor. This would be:
 
 .. testcode::
 
     from pytorch_lightning.utilities.cli import instantiate_class, LightningCLI
 
     class MyModel(LightningModule):
 
-        def __init__(self, optimizer1_init: dict):
+        def __init__(self, optimizer_init: dict, lr_scheduler_init: dict):
             super().__init__()
-            self.optimizer1_init = optimizer1_init
+            self.optimizer_init = optimizer_init
+            self.lr_scheduler_init = lr_scheduler_init
 
         def configure_optimizers(self):
-            return instantiate_class(self.parameters(), self.optimizer1_init)
+            optimizer = instantiate_class(self.parameters(), self.optimizer_init)
+            scheduler = instantiate_class(optimizer, self.lr_scheduler_init)
path: docs/source/common/lightning_cli.rst
text: @user1:
This has the caveat that your model needs to know about the CLI - which hasn't been the case at all until now.

Just pointing it out.

@user2:
I think it is a reasonable compromise in this case, given how powerful it is.  Also, presumably other non Lightning CLI initialization could use the same approach with `optimizer_init` if they really wanted to.

@author:
> This has the caveat that your model needs to know about the CLI - which hasn't been the case at all until now.

Yes, I would have preferred to avoid this. But I didn't see any other way. If these classes were more composable instead of requiring params and optimizer to instantiate it could have been done in a cleaner way.

Though now that you mention it, as it is now `instantiate_class` depends on jsonargparse being installed. There is no reason why a model should depend on this. I will change it to remove this dependency.