Materialiser fix

The materialiser cache stores snapshots and operations for each key. Here I present a basic, per key mechanism to garbage collect its state with the intention of releasing memory and avoiding reading from the log (stable storage). Keep in mind that this is the first and most basic approach ever taken, and there is no particular reason for not doing it in any other different way, so all feedback and ideas are welcome.

As there are object views (snapshots) and operations, the growing number of any of them needs to trigger the GC mechanism. 

1. Snapshot (or read-triggered) GC: 
     In case the number of snapshots stored for a key grows up to a value “snapshot_threshold”, keep just the latest “snapshot_min” snapshots. Proposed initial values (for no particular reason) are 10 and 2 respectively. Note that the snapshot generation is done on reads, that’s why this mechanism is triggered only on reads. All the operations that are not included in the remaining snapshots and are not older than the oldest one, remain in the operation cache. The rest are removed.

2. Operation (or write triggered) GC:
      In case the number of update operations stored for a key grows up to a number “ops_threshold”, just perform a read. This read operation will generate snapshots, therefore eventually triggering the Snapshot GC mechanism. Initial value for ops_threshold is 50.

Note that right now each update generates a snapshot in order to generate the downstream operation. That's why if snapshot_threshold is bigger than ops_thresold the Operation GC mechanism will never be triggered. 



Because the intention is not to get a read value, but to trigger the GC mechanism that a read triggers. That case happens when there have been many updates and no reads.

Alejandro Z. Tomsic
aletomsic@gmail.com



On Oct 14, 2014, at 18:34, Christopher Meiklejohn <notifications@github.com> wrote:

> In src/materializer_vnode.erl:
> 
> > +
> > +
> > +%% @doc Insert an operation and start garbage collection triggered by writes.
> > +%% the mechanism is very simple; when there are more than OPS_THRESHOLD
> > +%% operations for a given key, just perform a read, that will trigger
> > +%% the GC mechanism.
> > +-spec op_insert_gc(term(), clocksi_payload(), 
> > +	orddict:orddict(), atom() , atom() )-> true.
> > +op_insert_gc(Key,DownstreamOp, OpsDict, OpsCache, SnapshotCache)-> 
> > +    case (orddict:size(OpsDict))>=?OPS_THRESHOLD of 
> > +    true ->
> > +	    Type=DownstreamOp#clocksi_payload.type,
> > +	    SnapshotTime=DownstreamOp#clocksi_payload.snapshot_time,
> > +	    Type=DownstreamOp#clocksi_payload.type,
> > +	    SnapshotTime=DownstreamOp#clocksi_payload.snapshot_time,
> > +	    _Read=internal_read(Key, Type, SnapshotTime, OpsCache, SnapshotCache),
> Why ignore the return value?
> 
> —
> Reply to this email directly or view it on GitHub.
>