spotify/luigi — pull request #320 by daleroberts (2014-05-01 23:52:11+00:00)
Title: Minor refactoring of Worker to allow easier subclassing

I've made a few minor tweaks to the `Worker` class to allow easier subclassing:
- I've changed a few (private) variable names so that name mangling does not occur.
- Instead of calling `task.complete()` directly in `_add` function, I've put that call in a function called `_check_complete`. This allows a subclass to overwrite this method and handle this in their own way.

This pull request is preempting a custom MPI scheduler (and hopefully some other cool stuff) that I hope to push to luigi soon. If you want a sneak peek and want to understand why I would like these tweaks have a look at https://github.com/daleroberts/luigi/tree/daleroberts/mpi-scheduler

This is the start of getting luigi to do some massive big data and big compute stuff on the 57,472 core Raijin supercomputer (http://nci.org.au/nci-systems/national-facility/peak-system/raijin).