Repository: spotify/luigi

Issue:
Visualizer fails with similar range targets.
I'm using Range Targets to guide the loading of some information that appears on a daily basis that we want to load ASAP. (We have the configuration set to retry on external task failures and to keep the workers alive)

When you run two Range Targets with the same parameters at the same time, but the target task has different parameters, the visualizer will only display one of the range targets in any given screen. The sub-tasks are of course still present and Luigi doesn't appear to have any issue with the Range targets being identical, so I am presuming this is an okay way to use Range. I don't know enough about JS to dig in and figure out what's going wrong, but I imagine it has something to do with the task parameters being identical.

My command lines look something like this:

```
# Note that I use 2016.04.01 to mean, start loading today's data. This is set in our cron job to change to whatever the current date is whenever it re-executes (which would be as soon as the job exits)
#2 Days forward takes care of any issues with the international date line
luigi --module mymodule RangeDaily --start 2016.04.01 --days-forward 2 --of MyTask --MyTask-myparam 1
luigi --module mymodule RangeDaily --start 2016.04.01 --days-forward 2 --of MyTask --MyTask-myparam 2
```

Am I just going about this backassward? I feel like this would be an issue if you had to run a range on a parameterized task for any reason. Just to reiterate, there's _no_ problem with the jobs running or anything, this only shows up as an issue in the visualizer.



Discussion:
I wish this day would never come ... I was aware of this design issue when redesigning the parameter system. Though I opted for it to make the common case easy.

Yea the problem is indeed that the parameter set to the RangeDaily tasks is identical. If you can't live with it, just hack around and set different `--start` parameters. If you can't live with that bad hack, we should think about a proper solution ...

For me, it's a bit of an issue since I actually wrote a subclass of RangeDaily to let me _just_ specify `--days-forward` and `--days-back` (so I could have jobs that are always looking forward and filling in anything missing from the past few days).

I mean, it's not a high priority for me since everything seems to work and the Scheduler doesn't really seem to have any problem with it, and we're not heavily relying on the visualizer, but it'd be _nice_ if it worked right.

Now that each job is in part associated with a hash identifier, maybe that could be used to differentiate between instances in the visualizer?

Actually. I totally forgot about this simple solution:

---

Use a new `DictParameter` called "pass-along" or something. This line:

```
luigi --module mymodule RangeDaily --start 2016.04.01 --days-forward 2 --of MyTask --MyTask-myparam 1
```

becomes

```
luigi --module mymodule RangeDaily --start 2016.04.01 --days-forward 2 --of MyTask --pass-along {myparam:1}
```

---

It doesn't solve the global problem, sure. But this common use case is solved.

edit: To clarify, this does solution means to alter the code for the RangeBase task.

Just to make sure I am correctly understanding the work around: I would modify RangeBase (or a sublcass, presumably) to have the pass-along parameter, then actually instantiate the of-class instance by using that dictionary as it's keyword arguments?

Yes, pretty much. You have to change RangeBase itself. You would need to send a PR to this repository.

In particular this code needs to be changed to use the new parameter.: https://github.com/spotify/luigi/blob/8d39715092a67de0537e7c173babe6b8279d74ef/luigi/tools/range.py#L176-L181

Would you be interested to send a PR? :)

Not really. That sounds like a pretty screwy hack, especially if it _has_ to be on the RangeBase class.

If I knew JS at all, I'd offer to try and figure out how to fix the visualizer.

Disagree. I think it resolves the problem by removing a hack. The current hack is that in different execution environments (different global parameters), the one and same instantiated task (some RangeBase-child) requires _different_ tasks for the same set of parameters it is input with. It's like using global variables. If we explicitly thread along all parameters from RangeBase, it becomes more like a pure and global-state-independent task.

What makes you think the problem is in the visualiser? There's no way the scheduler (which only acts on taskids) can resolve this.

Hmm, guess I'm a little confused on how the scheduler is identifying the different instances of the Range tasks.

I'll look at the scheduler code more in depth so I can understand it better.

Hey, sorry. I've been a little busy. Do you know if this would also affect the worker count_uniques option?

Yes I think it will affect count_unqiues. Because the metatask (RangeBlah) is going to be assigned different ids when it's passing along different parameters to the child tasks.

Heh. Turns out it is causing an operational issue for me. Because I've got keep alive set to wait on some external tasks, some of the jobs aren't terminating so they can be rescheduled appropriately. I'll try and implement your suggested solution. After thinking about it more, I think you're right that it's the right way of taking care of it.

Also, I was also thinking about whether or not TaskParameters should _always_ be accompanied by a pass-through or not. I couldn't come up with a great example that wouldn't be taken care of by a subclass of Range though. Any thoughts on whether pass-through is a general concept or not? I'll just work on the Range version, but I wonder if this'll come up again.


Resolving patch:
diff --git a/luigi/tools/range.py b/luigi/tools/range.py
--- a/luigi/tools/range.py
+++ b/luigi/tools/range.py
@@ -26,6 +26,7 @@
 """
 
 import itertools
+import functools
 import logging
 import warnings
 import operator
@@ -88,6 +89,7 @@ class RangeBase(luigi.WrapperTask):
     # TODO lift the single parameter constraint by passing unknown parameters through WrapperTask?
     of = luigi.TaskParameter(
         description="task name to be completed. The task must take a single datetime parameter")
+    of_params = luigi.DictParameter(default=dict(), description="Arguments to be provided to the 'of' class when instantiating")
     # The common parameters 'start' and 'stop' have type (e.g. DateParameter,
     # DateHourParameter) dependent on the concrete subclass, cumbersome to
     # define here generically without dark magic. Refer to the overrides.
@@ -125,6 +127,18 @@ def datetime_to_parameter(self, dt):
     def parameter_to_datetime(self, p):
         raise NotImplementedError
 
+    def datetime_to_parameters(self, dt):
+        """
+        Given a date-time, will produce a dictionary of of-params combined with the ranged task parameter
+        """
+        raise NotImplementedError
+
+    def parameters_to_datetime(self, p):
+        """
+        Given a dictionary of parameters, will extract the ranged task parameter value
+        """
+        raise NotImplementedError
+
     def moving_start(self, now):
         """
         Returns a datetime from which to ensure contiguousness in the case when
@@ -174,11 +188,19 @@ def _format_range(self, datetimes):
         return '[%s, %s]' % (param_first, param_last)
 
     def _instantiate_task_cls(self, param):
+        return self.of(**self._task_parameters(param))
+
+    @property
+    def _param_name(self):
         if self.param_name is None:
-            return self.of_cls(param)
+            return next(x[0] for x in self.of.get_params() if x[1].positional)
         else:
-            kwargs = {self.param_name: param}
-            return self.of_cls(**kwargs)
+            return self.param_name
+
+    def _task_parameters(self, param):
+        kwargs = dict(**self.of_params)
+        kwargs[self._param_name] = param
+        return kwargs
 
     def requires(self):
         # cache because we anticipate a fair amount of computation
@@ -282,6 +304,19 @@ def datetime_to_parameter(self, dt):
     def parameter_to_datetime(self, p):
         return datetime(p.year, p.month, p.day)
 
+    def datetime_to_parameters(self, dt):
+        """
+        Given a date-time, will produce a dictionary of of-params combined with the ranged task parameter
+        """
+        return self._task_parameters(dt.date())
+
+    def parameters_to_datetime(self, p):
+        """
+        Given a dictionary of parameters, will extract the ranged task parameter value
+        """
+        dt = p[self._param_name]
+        return datetime(dt.year, dt.month, dt.day)
+
     def moving_start(self, now):
         return now - timedelta(days=self.days_back)
 
@@ -332,6 +367,18 @@ def datetime_to_parameter(self, dt):
     def parameter_to_datetime(self, p):
         return p
 
+    def datetime_to_parameters(self, dt):
+        """
+        Given a date-time, will produce a dictionary of of-params combined with the ranged task parameter
+        """
+        return self._task_parameters(dt)
+
+    def parameters_to_datetime(self, p):
+        """
+        Given a dictionary of parameters, will extract the ranged task parameter value
+        """
+        return p[self._param_name]
+
     def moving_start(self, now):
         return now - timedelta(hours=self.hours_back)
 
@@ -529,7 +576,8 @@ class RangeDaily(RangeDailyBase):
 
     def missing_datetimes(self, finite_datetimes):
         try:
-            complete_parameters = self.of_cls.bulk_complete(map(self.datetime_to_parameter, finite_datetimes))
+            cls_with_params = functools.partial(self.of, **self.of_params)
+            complete_parameters = self.of.bulk_complete.__func__(cls_with_params, map(self.datetime_to_parameter, finite_datetimes))
             return set(finite_datetimes) - set(map(self.parameter_to_datetime, complete_parameters))
         except NotImplementedError:
             return infer_bulk_complete_from_fs(
@@ -557,7 +605,8 @@ class RangeHourly(RangeHourlyBase):
     def missing_datetimes(self, finite_datetimes):
         try:
             # TODO: Why is there a list() here but not for the RangeDaily??
-            complete_parameters = self.of_cls.bulk_complete(list(map(self.datetime_to_parameter, finite_datetimes)))
+            cls_with_params = functools.partial(self.of, **self.of_params)
+            complete_parameters = self.of.bulk_complete.__func__(cls_with_params, list(map(self.datetime_to_parameter, finite_datetimes)))
             return set(finite_datetimes) - set(map(self.parameter_to_datetime, complete_parameters))
         except NotImplementedError:
             return infer_bulk_complete_from_fs(


Test patch:
diff --git a/test/range_test.py b/test/range_test.py
--- a/test/range_test.py
+++ b/test/range_test.py
@@ -566,6 +566,35 @@ def output(self):
         actual = [str(t) for t in task.requires()]
         self.assertEqual(actual, expected)
 
+    def test_bulk_complete_of_params(self):
+        class BulkCompleteDailyTask(luigi.Task):
+            non_positional_arbitrary_argument = luigi.Parameter(default="whatever", positional=False, significant=False)
+            d = luigi.DateParameter()
+            arbitrary_argument = luigi.BoolParameter()
+
+            @classmethod
+            def bulk_complete(cls, parameter_tuples):
+                ptuples = list(parameter_tuples)
+                for t in map(cls, ptuples):
+                    assert t.arbitrary_argument
+                return ptuples[:-2]
+
+            def output(self):
+                raise RuntimeError("Shouldn't get called while resolving deps via bulk_complete")
+
+        task = RangeDaily(now=datetime_to_epoch(datetime.datetime(2015, 12, 1)),
+                          of=BulkCompleteDailyTask,
+                          of_params=dict(arbitrary_argument=True),
+                          start=datetime.date(2015, 11, 1),
+                          stop=datetime.date(2015, 12, 1))
+        expected = [
+            'BulkCompleteDailyTask(d=2015-11-29, arbitrary_argument=True)',
+            'BulkCompleteDailyTask(d=2015-11-30, arbitrary_argument=True)',
+        ]
+
+        actual = [str(t) for t in task.requires()]
+        self.assertEqual(actual, expected)
+
     @mock.patch('luigi.mock.MockFileSystem.listdir',
                 new=mock_listdir([
                     '/data/2014/p/v/z/2014_/_03-_-21octor/20/ZOOO',
@@ -651,6 +680,35 @@ def output(self):
         actual = [str(t) for t in task.requires()]
         self.assertEqual(actual, expected)
 
+    def test_bulk_complete_of_params(self):
+        class BulkCompleteHourlyTask(luigi.Task):
+            non_positional_arbitrary_argument = luigi.Parameter(default="whatever", positional=False, significant=False)
+            dh = luigi.DateHourParameter()
+            arbitrary_argument = luigi.BoolParameter()
+
+            @classmethod
+            def bulk_complete(cls, parameter_tuples):
+                for t in map(cls, parameter_tuples):
+                    assert t.arbitrary_argument
+                return parameter_tuples[:-2]
+
+            def output(self):
+                raise RuntimeError("Shouldn't get called while resolving deps via bulk_complete")
+
+        task = RangeHourly(now=datetime_to_epoch(datetime.datetime(2015, 12, 1)),
+                           of=BulkCompleteHourlyTask,
+                           of_params=dict(arbitrary_argument=True),
+                           start=datetime.datetime(2015, 11, 1),
+                           stop=datetime.datetime(2015, 12, 1))
+
+        expected = [
+            'BulkCompleteHourlyTask(dh=2015-11-30T22, arbitrary_argument=True)',
+            'BulkCompleteHourlyTask(dh=2015-11-30T23, arbitrary_argument=True)',
+        ]
+
+        actual = [str(t) for t in task.requires()]
+        self.assertEqual(actual, expected)
+
     @mock.patch('luigi.mock.MockFileSystem.exists',
                 new=mock_exists_always_false)
     def test_missing_directory(self):
@@ -738,3 +796,44 @@ def output(self):
                                 param_name='date_param')
         expected_task = MyTask('woo', datetime.date(2015, 12, 1))
         self.assertEqual(expected_task, list(range_task._requires())[0])
+
+    def test_of_param_distinction(self):
+        class MyTask(luigi.Task):
+            arbitrary_param = luigi.Parameter(default='foo')
+            arbitrary_integer_param = luigi.IntParameter(default=10)
+            date_param = luigi.DateParameter()
+
+            def complete(self):
+                return False
+
+        range_task_1 = RangeDaily(now=datetime_to_epoch(datetime.datetime(2015, 12, 2)),
+                                  of=MyTask,
+                                  start=datetime.date(2015, 12, 1),
+                                  stop=datetime.date(2015, 12, 2))
+        range_task_2 = RangeDaily(now=datetime_to_epoch(datetime.datetime(2015, 12, 2)),
+                                  of=MyTask,
+                                  of_params=dict(arbitrary_param="bar", abitrary_integer_param=2),
+                                  start=datetime.date(2015, 12, 1),
+                                  stop=datetime.date(2015, 12, 2))
+        self.assertNotEqual(range_task_1.task_id, range_task_2.task_id)
+
+    def test_of_param_commandline(self):
+        class MyTask(luigi.Task):
+            task_namespace = "wohoo"
+            date_param = luigi.DateParameter()
+            arbitrary_param = luigi.Parameter(default='foo')
+            arbitrary_integer_param = luigi.IntParameter(default=10)
+            state = (None, None)
+            comp = False
+
+            def complete(self):
+                return self.comp
+
+            def run(self):
+                self.comp = True
+                MyTask.state = (self.arbitrary_param, self.arbitrary_integer_param)
+
+        now = str(int(datetime_to_epoch(datetime.datetime(2015, 12, 2))))
+        self.run_locally(['RangeDailyBase', '--of', 'wohoo.MyTask', '--of-params', '{"arbitrary_param":"bar","arbitrary_integer_param":5}',
+                          '--now', '{0}'.format(now), '--start', '2015-12-01', '--stop', '2015-12-02'])
+        self.assertEqual(MyTask.state, ('bar', 5))