SyncFree/floppystore — pull request #81 by aletomsic (2014-10-14 15:08:32+00:00)
Title: Materialiser fix

The materialiser cache stores snapshots and operations for each key. Here I present a basic, per key mechanism to garbage collect its state with the intention of releasing memory and avoiding reading from the log (stable storage). Keep in mind that this is the first and most basic approach ever taken, and there is no particular reason for not doing it in any other different way, so all feedback and ideas are welcome.

As there are object views (snapshots) and operations, the growing number of any of them needs to trigger the GC mechanism.

1. Snapshot (or read-triggered) GC:
In case the number of snapshots stored for a key grows up to a value “snapshot_threshold”, keep just the latest “snapshot_min” snapshots. Proposed initial values (for no particular reason) are 10 and 2 respectively. Note that the snapshot generation is done on reads, that’s why this mechanism is triggered only on reads. All the operations that are not included in the remaining snapshots and are not older than the oldest one, remain in the operation cache. The rest are removed.

2. Operation (or write triggered) GC:
In case the number of update operations stored for a key grows up to a number “ops_threshold”, just perform a read. This read operation will generate snapshots, therefore eventually triggering the Snapshot GC mechanism. Initial value for ops_threshold is 50.

Note that right now each update generates a snapshot in order to generate the downstream operation. That's why if snapshot_threshold is bigger than ops_thresold the Operation GC mechanism will never be triggered.