spotify/luigi — pull request #584 by daveFNbuck (2014-12-23 23:30:57+00:00)
Title: Keeps track of tasks in scheduler grouped by status

I've been seeing half-second get_work runtimes recently on my scheduler. The
cause of this has been a large backfill that scheduled a lot of done tasks. With
about 100k done tasks and about 350 pending tasks, a lot of time is wasted
looping over done tasks just to ignore them. This essentially caches these
computations by keeping a dictionary to group tasks by status and updating it
whenever task statuses are changed. This has sped my scheduling up by an order
of magnitude, now taking about 20ms per get_work.

In order to ensure updates happen every time, the task state has to take over
any functions that modify status. This is a necessary for remote task storage
anyway, so it had to be done eventually. There are no new tests here because
there are no functionality changes, only improvements in efficiency and
different internal representations.