spotify/luigi — pull request #569 by daveFNbuck (2014-12-11 22:01:29+00:00)
Title: Times out long-running workers

I woke up this morning to find that very few jobs had run overnight because
something weird happened with our hadoop cluster that caused all of our jobs to
hang permanently. When I checked in the morning, all workers were busy running
jobs that had been going for about 10 hours, and thousands of pending jobs that
would normally have been run overnight had piled up.

After killing and restarting all of my workers, everything started running fine
again because whatever weird issue had caused everything to hang was over. In
order to automate this fix, I've added a worker_timeout property to tasks that
tells the worker how long to attempt to run the task before killing it.

This property defaults to 0, meaning do not time tasks out. It can be changed
individually for a task by override or globally for all non-overridden tasks
via client.cfg. This only applies when running with more than 1 worker process,
as we need to run the task in a separate process in order to be able to kill it
after the timeout.