Repository: mlcommons/GaNDLF

Issue:
Add another optimizer (or 2) as option
**Is your feature request related to a problem? Please describe.**
Currently, GaNDLF only offers Adam and SGD as options for optimizers, which could sometimes be a bit limiting.

**Describe the solution you'd like**
Add ~[RMSProp](https://pytorch.org/docs/stable/generated/torch.optim.RMSprop.html#torch.optim.RMSprop), and optionally~ one or more of the following:
- [LBFGS](https://pytorch.org/docs/1.13/generated/torch.optim.LBFGS.html#torch.optim.LBFGS) 
- [RAdam](https://pytorch.org/docs/1.13/generated/torch.optim.RAdam.html#torch.optim.RAdam)
- ~[Rprop](https://pytorch.org/docs/1.11/generated/torch.optim.Rprop.html#torch.optim.Rprop)~
- ~[ASGD](https://pytorch.org/docs/1.11/generated/torch.optim.ASGD.html#torch.optim.ASGD)~

**Describe alternatives you've considered**
N.A.

**Additional context**
N.A.

EDIT [2023/03/20]: updated to the optimizers that are actually missing.


Discussion:
@sarthakpati , Unless I am mistaken from looking at LOC from the [file](https://github.com/mlcommons/GaNDLF/blob/master/GANDLF/optimizers/wrap_torch.py), we are supporting `SGD, ASGD, Rprop, Adam, AdamW, Adamax, Adadelta, Adagrad, RMSprop`. Also, any specific reason why we need [LBFGS](https://pytorch.org/docs/1.11/generated/torch.optim.LBFGS.html#torch.optim.LBFGS) or is it just an enhancement? 
Ah, I completely forgot that we had incorporated these. 😵‍💫 @AdiSir05 and I had discussed adding an optimizer would be a good exercise as a "getting to know the guts" of GaNDLF. Perhaps he can still add LBFGS using the template we have for the rest before moving onto his main work. What do you think, @Geeks-Sid and @AdiSir05?
Yeah, sounds good. He could try adding all the [ones](https://pytorch.org/docs/1.11/optim.html#algorithms) (LBFGS, RAdam) that are not added in GaNDLF as a part of the practice. :)
> Yeah, sounds good. He could try adding all the [ones](https://pytorch.org/docs/1.11/optim.html#algorithms) (LBFGS, RAdam) that are not added in GaNDLF as a part of the practice. :)

Sounds good to me. Does that work, @AdiSir05? 
That sounds good to me! I can start with LBFGS and RAdam. If there are more optimizers to add, I can do those after. 

Resolving patch:
diff --git a/GANDLF/optimizers/__init__.py b/GANDLF/optimizers/__init__.py
--- a/GANDLF/optimizers/__init__.py
+++ b/GANDLF/optimizers/__init__.py
@@ -9,6 +9,7 @@
     adadelta,
     adagrad,
     rmsprop,
+    radam,
 )
 
 global_optimizer_dict = {
@@ -22,6 +23,7 @@
     "adadelta": adadelta,
     "adagrad": adagrad,
     "rmsprop": rmsprop,
+    "radam": radam,
 }
 
 
diff --git a/GANDLF/optimizers/wrap_torch.py b/GANDLF/optimizers/wrap_torch.py
--- a/GANDLF/optimizers/wrap_torch.py
+++ b/GANDLF/optimizers/wrap_torch.py
@@ -9,6 +9,7 @@
     Adadelta,
     Adagrad,
     RMSprop,
+    RAdam
 )
 
 
@@ -223,3 +224,23 @@ def rmsprop(parameters):
         weight_decay=parameters["optimizer"].get("weight_decay", 0),
     )
 
+def radam(parameters):
+    """
+    Creates a RAdam optimizer from the PyTorch `torch.optim` module using the input parameters. 
+    
+    Args: 
+        parameters (dict): A dictionary containing the input parameters for the optimizer.
+        
+    Returns: 
+        optimizer (torch.optim.RAdam): A RAdam optimizer.
+    """
+    # Create the optimizer using the input parameters
+    return RAdam(
+        parameters["model_parameters"],
+        lr=parameters.get("learning_rate"),
+        betas=parameters["optimizer"].get("betas", (0.9, 0.999)),
+        eps=parameters["optimizer"].get("eps", 1e-8),
+        weight_decay=parameters["optimizer"].get("weight_decay", 0),
+        foreach=parameters["optimizer"].get("foreach", None),
+    )
+


Test patch: