guardian/frontend — pull request #4957 by janua (2014-07-02 13:57:14+00:00)
Title: Change pressing variables

We are having a few issues with pressing, which is causing the tool to slow down. From next week (hopefully), this will no longer slow down the tool, but these variables will still apply.

We would like to change the variables around while pressing happens on the tool to see how it behaves.

Currently, we feel that we are taking too many items off the queue too frequently. Once there is more than one instance (3 on `PROD`), this is not a problem, but on `CODE` it tries to take so many things off the queue that if it gets held up, it gets stuck concurrently trying too many things at once, and doesn't give itself enough time to finish. This is also not helped by the amount of `paths` that exist for code.

Taking too many things off the queue causes a lot of spikes in the CPU load. Reducing this should make it less spikey and more flat. The knock on effect is that we should increase the duration to wait before filling the queue with jobs.

We have 3 new variables:
- `pressJobBatchSize` - The maximum amount of work requested from the queue
- `pressJobConsumeRateInSeconds` - The duration to wait before trying to take things off the queue to work on
- `adminPressJobPushRateInMinutes` - The duration to wait before pushing all the paths from `admin` to the queue.

@stephanfowler