spotify/luigi — pull request #33 by JoeEnnever (2013-04-09 22:06:53+00:00)
Title: Makes scheduler configurable, no retry failed task

luigi.server.RPCHandler passes arguments from luigi.interface.get_config
to the scheduler it creates.

Passing a negative retry_delay to CentralPlannerScheduler will cause
tasks to never be retried, until a new luigi invocation. This is needed
for our very long running pipelines. A task that continually fails uses
up cluster resources and luigi worker processes.