Repository: spotify/luigi

Issue:
format for config times is inconsistent
<!---
We use GitHub issues mainly for tracking bugs and feature requests.
Questions for how to use luigi can be sent to the mailing list.

Currently, there are no strict procedures or guidelines for submitting issues.
In short, please just use common sense.

Common sense includes this at bare-minimum:

 * search for similar issues posted before creating a new issue.
 * Use markdown to format all code/logs. Issues which are hard to read
   when rendered on GitHub might be closed with a friendly reminder of this.
 * If applicable, reading relevant parts of the documentation.

Also, add steps to reproduce the bug, if applicable. Sample code would be nice too :)

For more information on how to submit valuable contributions,
see https://opensource.guide/how-to-contribute/#how-to-submit-a-contribution
-->
My `luigi.cfg` looked like this:
```
[core]
# Hide DeprecationWarning:
autoload_range: false

[scheduler]
# Time before a finished task graph is deleted (in seconds):
remove_delay: 10

[worker]
# Keep workers alive as long as they have pending jobs
keep_alive: true
max_keep_alive_idle_duration: 3600
```
Now luigi can parse the 10 seconds but it can't pass the 3600 seconds. That's surprising.

Turns out I had to specify `max_keep_alive_idle_duration: 1 hour` but that can't be found in the documentation without inspecting the source code. Yes, it's in the docs for the TimedeltaParameter but nowhere does it say that this option is interpreted as a TimedeltaParameter.

However, it would be better if all the config values specifying a time can be given in the same format. I'll see if I can get in a PR to solve this. The idea would be for all times to accept all formats accepted by TimeDeltaParameter plus to interpret an integer as seconds.




Discussion:
Thanks for reporting inconsistencies. If you make a PR for this, could you consider adding the time unit to the config parameter names, e.g. `remove_delay_seconds`? IMHO that decreases risk for config mistakes. I realise that it makes the PR scope larger, since it would be necessary to keep the old names for compatibility. But worth consideration.
I don't like that idea. I'm making it so that you can also enter something like `remove_delay: 1 hour 30 minutes` or the seconds as float. I think I'll keep the names as they are but perhaps it would make sense to actually remove the unit then.
I was looking into it and seems what I was proposing is way too much effort because it would break a bunch of tests. The idea was to create a hybrid between TimeDeltaParameter and FloatParameter.

New proposal:

- change TimedeltaParameters like `max_keep_alive_idle_duration` to also accept a float or int input and interpret it as seconds.
- mention the word "seconds" for every parameter in the docs where it's applicable
- but keep the names of the parameters as they are

Resolving patch:
diff --git a/luigi/batch_notifier.py b/luigi/batch_notifier.py
--- a/luigi/batch_notifier.py
+++ b/luigi/batch_notifier.py
@@ -15,7 +15,9 @@
 
 class batch_email(luigi.Config):
     email_interval = luigi.parameter.IntParameter(
-        default=60, description='Number of minutes between e-mail sends (default: 60)')
+        default=60,
+        config_path=dict(section="batch-notifier", name="email-interval-minutes"),
+        description='Number of minutes between e-mail sends (default: 60)')
     batch_mode = luigi.parameter.ChoiceParameter(
         default='unbatched_params', choices=('family', 'all', 'unbatched_params'),
         description='Method used for batching failures in e-mail. If "family" all failures for '
diff --git a/luigi/parameter.py b/luigi/parameter.py
--- a/luigi/parameter.py
+++ b/luigi/parameter.py
@@ -810,6 +810,7 @@ class TimeDeltaParameter(Parameter):
     """
     Class that maps to timedelta using strings in any of the following forms:
 
+     * A bare number is interpreted as duration in seconds.
      * ``n {w[eek[s]]|d[ay[s]]|h[our[s]]|m[inute[s]|s[second[s]]}`` (e.g. "1 week 2 days" or "1 h")
         Note: multiple arguments must be supplied in longest to shortest unit order
      * ISO 8601 duration ``PnDTnHnMnS`` (each field optional, years and months not supported)
@@ -857,6 +858,10 @@ def parse(self, input):
 
         See :py:class:`TimeDeltaParameter` for details on supported formats.
         """
+        try:
+            return datetime.timedelta(seconds=float(input))
+        except ValueError:
+            pass
         result = self._parseIso8601(input)
         if not result:
             result = self._parseSimple(input)


Test patch:
diff --git a/test/parameter_test.py b/test/parameter_test.py
--- a/test/parameter_test.py
+++ b/test/parameter_test.py
@@ -854,6 +854,16 @@ def f():
         self.assertRaises(ValueError, f)  # ISO 8601 durations with months are not supported
         exc.assert_called_once_with("Invalid time delta - could not parse P6M")
 
+    @with_config({"foo": {"bar": "12.34"}})
+    def testTimeDeltaFloat(self):
+        p = luigi.TimeDeltaParameter(config_path=dict(section="foo", name="bar"))
+        self.assertEqual(timedelta(seconds=12.34), _value(p))
+
+    @with_config({"foo": {"bar": "56789"}})
+    def testTimeDeltaInt(self):
+        p = luigi.TimeDeltaParameter(config_path=dict(section="foo", name="bar"))
+        self.assertEqual(timedelta(seconds=56789), _value(p))
+
     def testHasDefaultNoSection(self):
         self.assertRaises(luigi.parameter.MissingParameterException,
                           lambda: _value(luigi.Parameter(config_path=dict(section="foo", name="bar"))))