basho/yokozuna — pull request #347 by kuenishi (2014-03-20 12:08:10+00:00)
Title: tools to check and repair inconsistency between kv data and solr index

In current implementation, there could be inconsistency between
kv data and solr index, in case of write-intensive workloads where
just a log "failed to index object ...." left in console.log.
This cannot be fixed by AAE because AAE only fixes inconsistency
between replicas, but not kv data and index.

Mainly used functions would be `yz_repair:check_type/1` and
`yz_repair:check_bucket/1` , each for bucket types and default buckets.
Both functions iterates over all vnodes along the cluster and compares
keys from vnode and keys from solr with `_yz_pn:N` where N is the
partition number.

Potential problems: Those functions runs sequencially, thus very slow
in large datasets. This is because correctness is the priority for now
and it is easy to make repair scripts work in parallel. Also keylisting
strategy is awful because both keylists from kv and solr are stored in
memory at once for subtraction. This is also a future work to seek for
and use smarter join algorithm.

--

Writing riak_test is also future work.