spotify/luigi — pull request #540 by daveFNbuck (2014-11-21 01:31:35+00:00)
Title: Allows complete functions to be computed in parallel during scheduling

I've noticed that most of the time spent during scheduling is spend computing
complete functions, as these can involve various kinds of expensive checks
against remote file systems and services. This modifies the add function in
worker to allow complete functions to be computed asynchronously while
traversing the dependency graph. This can result in significant savings,
lowering the time needed for my twice-hourly scheduling from about 9 minutes to
about 3.5 minutes.

In order to gain the full benefit, I added an attribute ignored_root to Task.
If true, the task will not be directly scheduled when run as the root task from
the command line. This allows for scheduling many tasks via a WrapperTask
without actually scheduling the WrapperTask. Thus, we don't need to compute the
expensive complete() function of the WrapperTask on a single processor.

Using multiprocess requires picklable tasks, and this is incompatible with many
of the unit tests in worker_test. In order to get around this issue and allow
maximum compatibility with client code, multiprocess is not used at all unless
enabled via the command line or client.cfg. In order to minimize the code
difference between these cases, I added wrapper objects simulating the necessary
pool and queue interfaces for the single-processor case. This also allows the
existing unit tests to test most of the new code path, and I've only had to
write a few new unit tests to exercise the multiprocessing code.