Repository: spotify/luigi

Issue:
Fix external dynamic deps
Since running tasks communicate with worker via a queue, all dynamic dependencies that they yield must be serialized and then deserialized back. This doesn't work if a task has `run = NotImplemented`, since there was a specific check for that in Register for unclear reason.

This PR adds a test case to reproduce the issue and fixes it by removing the check.



Discussion:
Oh, it breaks tests, this behavior with `run = NotImplemented` seems to be intentional.
Could someone please explain the logic behind looking at `run` method existence to me? I'd also love to hear any ideas on how to fix the issue. Thanks a lot!

Yeah this sounds weird. I think the idea is you shouldn't be able to run external tasks from the command line. Not sure though. It seems like bad introspection which should be removed (I've seen same thing in a couple of places).

Do you have a breaking example with dynamic requirements btw?

Trivial example:

``` (python)
import luigi

class Dependency(luigi.ExternalTask):
    def output(self):
        return luigi.LocalTarget("/tmp/i_am_not_here")

class MainTask(luigi.Task):
    def output(self):
        return luigi.LocalTarget("/tmp/i_am_not_here_as_well")

    def run(self):
        yield Dependency()

if __name__ == "__main__":
    luigi.run()
```

Running `MainTask` produces the following:

```
$ python test.py MainTask --local-scheduler
DEBUG: Checking if MainTask() is complete
INFO: Scheduled MainTask() (PENDING)
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 1
INFO: [pid 45372] Worker Worker(salt=369848895, workers=1, host=f355-mbpr.local, username=konstantin, pid=45372) running   MainTask()
INFO: [pid 45372] Worker Worker(salt=369848895, workers=1, host=f355-mbpr.local, username=konstantin, pid=45372) new requirements      MainTask()
DEBUG: 1 running tasks, waiting for next task to finish
Traceback (most recent call last):
  File "test.py", line 15, in <module>
    luigi.run()
  File "/Users/konstantin/src/luigi/luigi/interface.py", line 425, in run
    return interface.run(tasks, worker_scheduler_factory, override_defaults=override_defaults)
  File "/Users/konstantin/src/luigi/luigi/interface.py", line 174, in run
    success &= w.run()
  File "/Users/konstantin/src/luigi/luigi/worker.py", line 743, in run
    self._handle_next_task()
  File "/Users/konstantin/src/luigi/luigi/worker.py", line 666, in _handle_next_task
    for module, name, params in new_requirements]
  File "/Users/konstantin/src/luigi/luigi/task_register.py", line 201, in load_task
    task_cls = Register.get_task_cls(task_name)
  File "/Users/konstantin/src/luigi/luigi/task_register.py", line 175, in get_task_cls
    raise TaskClassException('Task %r not found. Candidates are: %s' % (name, cls.tasks_str()))
luigi.task_register.TaskClassException: Task 'Dependency' not found. Candidates are: Config,MainTask,RangeBase,RangeDaily,RangeDailyBase,RangeHourly,RangeHourlyBase,Task,WrapperTask,core,scheduler,worker
```

Couple more thoughts on it:
1. If we remove the check, it will make this task ambiguous: https://github.com/spotify/luigi/blob/master/test/cmdline_test.py#L60 Even though picking the right class that way doesn't seem to be a good programming practice, some people might have existing code that relies on it, intentionally or not.
2. We still need to have a way of distinguishing external tasks here: https://github.com/spotify/luigi/blob/master/luigi/worker.py#L509 There are several ways of doing it, and I'm not sure the one in question is the worst of them. Ideas?

According to `git bisect`, the commit that broke external tasks is f8cdfedeee5894eb094d535972e95b75b77de255.

Yes that makes sense. I replaced a custom task importer with the standard one. But the standard one doesn't handle external tasks. Need to fix this

Nice catch @f355. Spotify haven't started to be power users of the dynamic require feature. Maybe @tommyengstrom want to comment? Anyway, kind of weird that we don't have tests for dynamically requiring a dynamic dependency.

Well, I don't really have anything to add. I never considered this case but I agree that there should be tests for it. 


Resolving patch:
diff --git a/luigi/task_register.py b/luigi/task_register.py
--- a/luigi/task_register.py
+++ b/luigi/task_register.py
@@ -135,8 +135,6 @@ def __get_reg(cls):
         # We return this in a topologically sorted list of inheritance: this is useful in some cases (#822)
         reg = OrderedDict()
         for cls in cls._reg:
-            if cls.run == NotImplemented:
-                continue
             name = cls.task_family
 
             if name in reg and reg[name] != cls and \


Test patch:
diff --git a/test/cmdline_test.py b/test/cmdline_test.py
--- a/test/cmdline_test.py
+++ b/test/cmdline_test.py
@@ -53,16 +53,6 @@ class AmbiguousClass(luigi.Task):
     pass
 
 
-class NonAmbiguousClass(luigi.ExternalTask):
-    pass
-
-
-class NonAmbiguousClass(luigi.Task):
-
-    def run(self):
-        NonAmbiguousClass.has_run = True
-
-
 class TaskWithSameName(luigi.Task):
 
     def run(self):
@@ -115,12 +105,6 @@ def test_cmdline_other_task(self, logger):
     def test_cmdline_ambiguous_class(self, logger):
         self.assertRaises(Exception, luigi.run, ['--local-scheduler', '--no-lock', 'AmbiguousClass'])
 
-    @mock.patch("logging.getLogger")
-    @mock.patch("warnings.warn")
-    def test_cmdline_non_ambiguous_class(self, warn, logger):
-        luigi.run(['--local-scheduler', '--no-lock', 'NonAmbiguousClass'])
-        self.assertTrue(NonAmbiguousClass.has_run)
-
     @mock.patch("logging.getLogger")
     @mock.patch("logging.StreamHandler")
     def test_setup_interface_logging(self, handler, logger):
diff --git a/test/task_test.py b/test/task_test.py
--- a/test/task_test.py
+++ b/test/task_test.py
@@ -21,6 +21,7 @@
 
 import luigi
 import luigi.task
+from luigi.task_register import load_task
 
 
 class DummyTask(luigi.Task):
@@ -55,6 +56,10 @@ def test_task_to_str_to_task(self):
         other = DummyTask.from_str_params(original.to_str_params())
         self.assertEqual(original, other)
 
+    def test_external_tasks_loadable(self):
+        task = load_task("luigi", "ExternalTask", {})
+        assert(isinstance(task, luigi.ExternalTask))
+
     def test_id_to_name_and_params(self):
         task_id = "InputText(date=2014-12-29)"
         (name, params) = luigi.task.id_to_name_and_params(task_id)