silverstripe-labs/silverstripe-fulltextsearch — pull request #45 by tractorcow (2014-05-29 04:55:36+00:00)
Title: API SearchUpdateQueuedJobProcessor now uses batching

While the queued jobs module doesn't have an ability for jobs to suppress themselves between batch runs, breaking a job into multiple steps (one run after one another) will allow the `QueuedJobsService` to check `isMemoryTooHigh` between each batch, and will continue at the next batch in a subsequent run if memory usage becomes too high.

If this doesn't solve the problem of jobs fatally using excessive resources, then it may be necessary to extend QueuedJobs to allow jobs to decide themselves when further processing should be delayed.

Some refactor of `SearchUpdateProcessor` was done in order to better support subclasses overriding specific methods.

Graceful recovery of partially completed batches is enabled so that if a batch contains multiple indexes, then it won't attempt to re-process successfully processed indexes for that batch.