spotify/luigi — pull request #479 by erikbern (2014-09-30 12:39:21+00:00)
Title: Initial prototype of a "slave" worker

If a worker is created using slave=True, then it will pull *any* tasks from the scheduler,
not just tasks that are assigned to it.

The biggest limitation atm is that the slave needs to be aware of what those tasks map to,
i.e. somehow they have to be imported for the worker to know what to do with the tasks.

It also doesn't handle failure very well.

Changed the semantics of how interface.load_task works a bit. It will now rely on the
Register class to locate task classes rather than importing modules dynamically. There is
still an option to import modules but it's not run by default.