Repository: ludwig-ai/ludwig

Issue:
decay, decay_rate and decay_steps not implemented
Using latest master it seems to me that `decay`, `decay_rate `and `decay_steps `are not affecting the learning rate at all. Looking in the trainer model, they don't even seem to be used in the train function.
https://github.com/uber/ludwig/blob/62430e4a0dd7a4fda08d6dcd615fbdbbf53c5377/ludwig/models/trainer.py#L166-L195

`learning_rate `and `learning_rate_warmup_epochs` instead work fine (and I see them parsed in the train function)
Am I missing something?
Maybe it's related to the TF2 porting?


Discussion:
using `decay` often, need this feature too
I can't really seem to find the point in where those parameters were dropped in the commit history... much as like they were meant to be, they were documented as they should have worked... but heh... never actually implemented those :)

@w4nderlust: this would have been obvious if there was such a "unknown parameter" warning\error in the yaml parsing, just to reference another use case for https://github.com/uber/ludwig/issues/822#issuecomment-674425054
Only references to `decay` in all the codebase are the default parameters. I can't find any other reference for the parameters in the subject.
This is weird, I have to go back in time to see what happened to these parameters. Thank you for posting this.
Indeed... And it's not TF2 fault, since a year ago they still aren't there.

Thanks
@w4nderlust I noticed that you replaced TF2 with the `master` branch and this affected a lot of commits. Don't know how. When I was switching back to TF1 I needed to use my own branch inside the fork
It was inside the deleted file

<img width="1014" alt="Screenshot 2020-08-22 at 00 25 13" src="https://user-images.githubusercontent.com/2017148/90936522-043cf400-e40e-11ea-82d1-b6340d2d5ab0.png">

This actually helped me rememeber: in TF1 ther was this exponential decay function i was using, inf TF2 thigns work differently, now I'm adding a todo before the release of v0.3 to re-enable the exponencial decay logic. Thank you for noticing this guys, it's great to release the alpha version and having members of the community spotting these things, will make the v0.3 the most robust version yet :)
#898 solves this. Let me know if it's not ok.
I am finally able to test decay and decay_rate, but it seems that the learning rate reduction due to decay is only applied one time, and then never again

ie: I set LR to 0.1, decay to true, decay_rate to 0.9 and LR goes down to 0.09986 after the first epoch and then stays that way forever

Can you check this? Do you prefer to have a new issue opened?

Thanks
So this is kinda weird. inspecting this I figured out some issues with the way the learning rate is computed. I'm fixing them, but yours is different. Also, changing the reporting of the learning rate per batch, as the decay (and the warm up) ar applead on a per batch basis.
Will probably put up a PR soonfor you to try.

Resolving patch:
diff --git a/ludwig/models/trainer.py b/ludwig/models/trainer.py
--- a/ludwig/models/trainer.py
+++ b/ludwig/models/trainer.py
@@ -67,6 +67,7 @@ def __init__(
             decay=False,
             decay_rate=0.96,
             decay_steps=10000,
+            staircase=False,
             batch_size=128,
             eval_batch_size=0,
             bucketing_field=None,
@@ -187,6 +188,7 @@ def __init__(
         self.decay = decay
         self.decay_rate = decay_rate
         self.decay_steps = decay_steps
+        self.staircase = staircase
         self.batch_size = batch_size
         self.eval_batch_size = batch_size if eval_batch_size < 1 else eval_batch_size
         self.bucketing_field = bucketing_field
@@ -231,7 +233,6 @@ def write_epoch_summary(
             summary_writer,
             metrics,
             step,
-            learning_rate=None
     ):
         if not summary_writer:
             return
@@ -244,9 +245,6 @@ def write_epoch_summary(
                     )
                     metric_val = output_feature[metric][-1]
                     tf.summary.scalar(metric_tag, metric_val, step=step)
-            if learning_rate:
-                tf.summary.scalar("combined/epoch_learning_rate",
-                                  learning_rate, step=step)
         summary_writer.flush()
 
     @classmethod
@@ -255,7 +253,8 @@ def write_step_summary(
             train_summary_writer,
             combined_loss,
             all_losses,
-            step
+            step,
+            learning_rate=None
     ):
         if not train_summary_writer:
             return
@@ -270,6 +269,10 @@ def write_step_summary(
                 loss_tag = "{}/step_training_loss".format(feature_name)
                 tf.summary.scalar(loss_tag, loss, step=step)
 
+            if learning_rate:
+                tf.summary.scalar("combined/step_learning_rate",
+                                  learning_rate, step=step)
+
         train_summary_writer.flush()
 
     def train(
@@ -476,7 +479,7 @@ def train(
                         digits=digits_per_epochs
                     )
                 )
-            current_learning_rate = progress_tracker.learning_rate
+
             # needed because batch size may change
             batcher.batch_size = progress_tracker.batch_size
 
@@ -495,6 +498,39 @@ def train(
 
             # training step loop
             while not batcher.last_batch():
+
+                # Set learning rate for this batch
+                current_learning_rate = progress_tracker.learning_rate
+
+                if self.decay:
+                    current_learning_rate = exponential_decay(
+                        current_learning_rate,
+                        self.decay_rate,
+                        self.decay_steps,
+                        progress_tracker.steps,
+                        self.staircase
+                    )
+
+                if self.horovod:
+                    current_learning_rate = learning_rate_warmup_distributed(
+                        current_learning_rate,
+                        progress_tracker.epoch,
+                        self.learning_rate_warmup_epochs,
+                        self.horovod.size(),
+                        batcher.step,
+                        batcher.steps_per_epoch
+                    ) * self.horovod.size()
+                else:
+                    current_learning_rate = learning_rate_warmup(
+                        current_learning_rate,
+                        progress_tracker.epoch,
+                        self.learning_rate_warmup_epochs,
+                        batcher.step,
+                        batcher.steps_per_epoch
+                    )
+                self.optimizer.set_learning_rate(current_learning_rate)
+
+                # obtain batch
                 batch = batcher.next_batch()
                 inputs = {
                     i_feat.feature_name: batch[i_feat.feature_name]
@@ -531,6 +567,7 @@ def train(
                         combined_loss=loss,
                         all_losses=all_losses,
                         step=progress_tracker.steps,
+                        learning_rate=current_learning_rate,
                     )
 
                 if self.horovod and first_batch:
@@ -545,33 +582,6 @@ def train(
                     self.horovod.broadcast_variables(
                         self.optimizer.variables(), root_rank=0)
 
-                if self.decay:
-                    current_learning_rate = exponential_decay(
-                        current_learning_rate,
-                        self.decay_rate,
-                        self.decay_steps,
-                        batcher.step
-                    )
-
-                if self.horovod:
-                    current_learning_rate = learning_rate_warmup_distributed(
-                        current_learning_rate,
-                        progress_tracker.epoch,
-                        self.learning_rate_warmup_epochs,
-                        self.horovod.size(),
-                        batcher.step,
-                        batcher.steps_per_epoch
-                    ) * self.horovod.size()
-                else:
-                    current_learning_rate = learning_rate_warmup(
-                        current_learning_rate,
-                        progress_tracker.epoch,
-                        self.learning_rate_warmup_epochs,
-                        batcher.step,
-                        batcher.steps_per_epoch
-                    )
-                self.optimizer.set_learning_rate(current_learning_rate)
-
                 progress_tracker.steps += 1
                 if is_on_master():
                     progress_bar.update(1)
@@ -606,7 +616,6 @@ def train(
                 summary_writer=train_summary_writer,
                 metrics=progress_tracker.train_metrics,
                 step=progress_tracker.epoch,
-                learning_rate=current_learning_rate,
             )
 
             if validation_set is not None and validation_set.size > 0:
diff --git a/ludwig/utils/math_utils.py b/ludwig/utils/math_utils.py
--- a/ludwig/utils/math_utils.py
+++ b/ludwig/utils/math_utils.py
@@ -45,8 +45,15 @@ def convert_size(size_bytes):
     return '{} {}'.format(s, size_name[i])
 
 
-def exponential_decay(initial_learning_rate, decay_rate, decay_steps, step):
-    return initial_learning_rate * decay_rate ** (float(step) / decay_steps)
+def exponential_decay(initial_learning_rate, decay_rate, decay_steps, step,
+                      staircase=False):
+    decay_rate = float(decay_rate)
+    decay_steps = float(decay_steps)
+    step = float(step)
+    exponent = step / decay_steps
+    if staircase:
+        exponent = math.ceil(exponent)
+    return initial_learning_rate * math.pow(decay_rate, exponent)
 
 
 def learning_rate_warmup_distributed(
@@ -78,7 +85,7 @@ def learning_rate_warmup_distributed(
                                   size
             lr'(epoch = warmup) = lr
     """
-    if epoch > warmup_epochs:
+    if epoch >= warmup_epochs:
         return learning_rate
     else:
         epoch_adjusted = float(epoch) + (curr_step / steps_per_epoch)
@@ -93,19 +100,22 @@ def learning_rate_warmup(
         curr_step,
         steps_per_epoch
 ):
-    global_curr_step = 1 + curr_step + epoch * steps_per_epoch
-    warmup_steps = warmup_epochs * steps_per_epoch
+    if epoch >= warmup_epochs:
+        return learning_rate
+    else:
+        global_curr_step = 1 + curr_step + epoch * steps_per_epoch
+        warmup_steps = warmup_epochs * steps_per_epoch
 
-    warmup_percent_done = global_curr_step / warmup_steps
-    warmup_learning_rate = learning_rate * warmup_percent_done
+        warmup_percent_done = global_curr_step / warmup_steps
+        warmup_learning_rate = learning_rate * warmup_percent_done
 
-    is_warmup = int(global_curr_step < warmup_steps)
-    interpolated_learning_rate = (
-            (1.0 - is_warmup) * learning_rate +
-            is_warmup * warmup_learning_rate
-    )
+        is_warmup = int(global_curr_step < warmup_steps)
+        interpolated_learning_rate = (
+                (1.0 - is_warmup) * learning_rate +
+                is_warmup * warmup_learning_rate
+        )
 
-    return interpolated_learning_rate
+        return interpolated_learning_rate
 
 
 def round2precision(val, precision: int = 0, which: str = ''):


Test patch: