basho/riak_kv — pull request #676 by buddhisthead (2013-09-30 21:18:16+00:00)
Title: Implement timeboxed changeset tracking of riak object writes

It is desirable to track which objects have changed during regular time intervals; this is useful for implementing differential extraction of riak objects and possibly also differential fullsync. This implementation uses bloom filters to record which objects have changed and periodically writes them to disk ( in order to reduce the impact of saving the bloom filter, it is not written to disk on the write-path of the riak object put). Writes are defaulted at once every 60 seconds, so there is the potential for loss of change set data on a node crash. That is something that warrants discussion; there is a tradeoff between impact to the node for writes (which are on the order of 10MB per bloom filter) vs data loss.

Changes are shown in this PR to kv_vnode in the write path. It's modeled after the AAE update and thus does a gen_server:call once out of every 100 gen_server:casts to create some back pressure without the frequency expense of a call.

There is a unit test.

Open for discussion!