spotify/luigi — pull request #419 by erikbern (2014-08-31 20:56:13+00:00)
Title: Refactored luigi.worker a bit to use multiprocessing

...instead of os.fork and os.wait

This commit features the following stuff
- multiprocessing instead of os.fork
- Added a bunch of unit tests to run with multiple workers
- Fixed MockFile to support multiple processes
- Multiple processes are now supported even with a local scheduler

Other things I had to fix that was marginally related:
- Added --no-lock to a bunch of tests
- Fixed an issue in lock_test.py that triggered an error on my machine

I have no idea if/how this will work on non-Unix platforms (eg. Windows). On thing that should work (in theory) is to replace multiprocessing.Process with threading.Thread. I doubt that the previous code was working since it was relying directly on os.fork.