anroots/Valify — Issue #3: Use a queue for W3C checks
Author: anroots
Created: 2012-02-28 20:25:14+00:00
State: open

Site checking is done all at once at the moment. That's bad for performance.

Implement checking using a queue: a table with sites to check, filled every ... hour.
A cron runs every X minutes, grabs the bottom X rows and does the checking.

That spreads the load out a bit and enables scaling.