spotify/luigi — pull request #429 by daveFNbuck (2014-09-08 04:13:26+00:00)
Title: Adds luigi global state tracking to interface

This tracks luigi's global state for worker processes of either scheduling or running. My main use case is to batch and cache complete checks during scheduling and then compute them as normal during running for more safety. For example, when scheduling 24 hours of scp tasks, it can take about a minute to check all 24 output files for existence. However, we can do one ls to check all the files at once and just refer to that until scheduling is over to do all the checks in a few seconds.

To help with this sort of use case, I wrote a decorator that caches a function's output only during scheduling and returns its value without caching after that. I don't know whether that belongs in luigi (and if so, where) so I haven't included it for now.