clarencen/lshift — Issue #261: Batching of ingress to the Version Correlation Store
Author: clarencen
Created: 2011-11-14 22:42:18+00:00
State: open

If a large number of events are provided to the VersionCorrelationStore, then we can quite rapidly end up with a backlog (especially when events are arriving via the AMQP transport whereby the sender isn't blocked waiting for indexing).

This causes substantial problems in the UI, where errors that no longer exist are shown - potentially leading to rather confusing user situations. I suspect this is probably very much a "real world" scenario, given that a bulk repair of a system would lead to exactly this scenario.

My proposal is for the following enhancement:

* Create a "Session" api on the version correlation store. This way, syncs could create a session and then explicitly commit it when done - allowing the correlation store to understand the scope, and batch flushes;
* The changes handler should also use this session api, but with a time based mechanism:
* If there have been no events for 5 seconds, then close and re-open the session;
* Ensure a flush occurs once every 10 seconds;

In the case of the AMQP transport, we'd just need to make sure that the ACK behaviour is sensible. Perhaps for the HTTP transport, we should pass a flag through with the event saying it is non-recoverable - hence requiring it to be flushed synchronously.