Repository: ludwig-ai/ludwig

Issue:
Inconsistent learning rates shown in output when specifying learning rate in config 
When specifying the learning_rate parameter in the training section of the config, `description.json` shows a different default learning rate in the optimizer section:

config.yaml:
<img width="231" alt="Screen Shot 2021-12-21 at 4 46 42 PM" src="https://user-images.githubusercontent.com/687280/147016130-2856cfda-c399-4fc8-b79b-2fe96003b559.png">

description.json:
<img width="698" alt="Screen Shot 2021-12-21 at 4 47 04 PM" src="https://user-images.githubusercontent.com/687280/147016222-60e44fd7-fd92-474d-b73d-17c5c62d5954.png">

Expected behavior is that there is one learning rate, and if it is specified in the config it should match that.



Discussion:
Example config for repro (titanic example):
```
input_features:
    -
        name: Pclass
        type: category
    -
        name: Sex
        type: category
    -
        name: Age
        type: numerical
        preprocessing:
          missing_value_strategy: fill_with_mean
    -
        name: SibSp
        type: numerical
    -
        name: Parch
        type: numerical
    -
        name: Fare
        type: numerical
        preprocessing:
          missing_value_strategy: fill_with_mean
    -
        name: Embarked
        type: category

output_features:
    -
        name: Survived
        type: binary
        norm: null
        class_weights: 1
training:
  epochs: 100
  learning_rate: 0.05
  optimizer:
    type: sgd
  decay: false
  regularization_lambda: 0
```
Curious about this as well. I think this traces to `merge_with_defaults` : As an example, in the case of `sgd`, `merge_with_defaults` will check `type` and load the defaults (including `lr`) specified in `default_optimizer_params_registry` regardless of how `learning_rate` is set: https://github.com/ludwig-ai/ludwig/blob/cfa01f07568d545167f36a2ef171a1d6ddb7dab5/ludwig/utils/defaults.py#L244
I believe the reason is that in TF the lr was not an init parameter of the optimizer but a parameter of the step function. As we have external mechanisms outside of the optimizer to set the lr (warmup and decay) we need to change this and reconcile them.
Could this one of culprits for [training differences between Torch and Tensorflow](https://github.com/ludwig-ai/ludwig/issues/1613)?

EDIT: I just saw your comment in Slack about how this doesn't seem to affect training. Nevermind!

Resolving patch:
diff --git a/ludwig/utils/defaults.py b/ludwig/utils/defaults.py
--- a/ludwig/utils/defaults.py
+++ b/ludwig/utils/defaults.py
@@ -76,10 +76,10 @@
 }
 
 default_optimizer_params_registry = {
-    "sgd": {"lr": 0.001},
-    "stochastic_gradient_descent": {"lr": 0.001},
-    "gd": {"lr": 0.001},
-    "gradient_descent": {"lr": 0.001},
+    "sgd": {},
+    "stochastic_gradient_descent": {},
+    "gd": {},
+    "gradient_descent": {},
     "adam": {
         "betas": (0.9, 0.999),
         # 'beta_1': 0.9,
@@ -247,6 +247,7 @@ def merge_with_defaults(config):
 
     # ===== Training Optimizer =====
     optimizer = config[TRAINING]["optimizer"]
+    set_default_value(optimizer, "lr", config[TRAINING]["learning_rate"])
     default_optimizer_params = get_default_optimizer_params(optimizer[TYPE])
     for param in default_optimizer_params:
         set_default_value(optimizer, param, default_optimizer_params[param])


Test patch: