scraperwiki/cobalt — pull request #44 by pwaller (2014-04-17 17:27:44+00:00)
Title: Job daemon

Not complete, but proof of concept

This daemon runs a limited number of jobs in parallel, and is
in principle capable of cancelling duplicate jobs.

It would be a good idea to have such a thing to prevent runaway
load as the result of many jobs running.

We've observed the queue length on free grow as high as ~70-80.