pivotal/projectmonitor — pull request #10 by excepttheweasel (2012-09-18 22:55:53+00:00)
Title: Polling refactor

This is a huge refactor of the polling system to use eventmachine for more performant polling. The old polling system had several problems:

* It needed a delayed job just to launch the other delayed jobs that polled projects, with the standard heroku limit of 3 delayed jobs at one time, this meant only two projects could be polled simultaneously
* It used threads to do network bound IO, meaning extra startup, memory and resource overhead with little benefit
* If 2 jobs in a row had a network timeout, nothing happened for 30 seconds
* It typically took around 15 minutes to go through the entire list of production CI servers

This new polling code handles all updates in a single parallel block, in theory this should allow tens of thousands of project updates per second.

Note: the most optimal polling mode doesn't work on Heroku as they have hacked the Delayed::Job code to ignore a user specified timeout. So it has to be configured to run periodically. The poller only needs to run for 30 seconds however, as all tasks will have either completed or timed out by this time.