basho/yokozuna — pull request #171 by rzezeski (2013-09-12 02:46:29+00:00)
Title: Remove the default index

The default index, `_yz_default`, was added to prevent AAE from doing
unnecessary repairs after tree re-build. After a Yokozuna hashtree is
cleared it is rebuilt from data in Solr. If Solr only contains data
for indexed buckets then its hashtree will be missing hashes for
non-indexed buckets. Since AAE exchanges based on partition the
system will think keys are missing and potentially invoke lots of
repair. The default index avoids this by indexing the needed data for
AAE for buckets which are not indexed. AAE works, but at the cost of
affecting write throughput on non-indexed buckets and increasing the
index size. On my local workstation the default index reduced write
throughput to 56% of baseline.

This tradeoff is unnecessary. Write throughput on a non-indexed
bucket can remain much closer to baseline without causing trouble for
AAE.

* Determine if the object needs to be indexed.

* If it does then write to Solr and update the hashtree. If it
doesn't then just update the hashtree. This ensures that the
hashtrees are kept up to date with significantly less cost than
sending an update to Solr

* After a Yokozuna hashtree is cleared it will be missing keys stored
in non-indexed buckets.

* During exchange each missing key will cause the repair function to
be invoked.

* This repair function will now check if the object needs to be
indexed. I.e. an actual missing index entry versus an object that
doesn't need to be indexed. If the object is not to be indexed then
the Yokozuna hashtree is simply sent an update.

* So the first exchange of a tree after re-build will invoke the
repair function potentially many times. But these invocations will
consist of a simple check + a hashtree update. This is far less
work than minimally indexing all data.