spotify/luigi — pull request #572 by daveFNbuck (2014-12-12 01:37:25+00:00)
Title: Fixes multiple purging bug.

If two worker processes get killed at the same time, we'll add them both to the
task handling queue as failures, then remove one of them from running tasks.
After this, we'll add the other one to the queue and remove it. On the third
attempt at task handling, we'll have the second task in the queue but not in
the running dict, resulting in duplicated error messages and a KeyError when
popping from running.

Even if we cut off the purge loop after it finds its first item, this could
still happen if there is already stuff in the queue or other workers insert
into the queue during/after the loop. In order to cleanly avoid removing items
multiple times, this ignores queue items that aren't in the running dict.