basho/yokozuna — pull request #163 by rzezeski (2013-08-30 18:35:22+00:00)
Title: Riak Search Migration Test 1st Cut + Several Bug Fixes

NOTE: Looking at the individual commits will make this easier to review.

This branch started as a test for Riak Search to Yokozuna migration
but then evolved into a series of major bugfixes. I could try to make
an individual patch for each bug with a minimal test but it would eat
up a bunch or time and there's lots to get done for Riak 2.0. I also
didn't want this PR to get any bigger without review so I've decided
to push a functional, but incomplete, version of the migraiton test.
The test has some TODOs to make it even better and a method for
immediate re-indexing needs to be addded. But the current test is
functional and verifies that AAE can be used to migrate data. Now the
bug fixes.

### AAE Deadlock ###

The `yz_kv:index` function which runs on the vnode process makes a
blocking call to `yz_entropy_mgr`. The `yz_entropy_mgr` makes a
blocking call to the same KV vnode. Deadlock.

The fix I made was to make the call from entropy mgr to vnode async.

### Mixed Cluster Issues ###

During upgrade from 1.4.x there will be a period of time where only a
subset of the nodes have Yokozuna running. In that case Yokozuna
nodes cannot assume that other nodes speak Yokozuna. There is a case
where Yokozuna makes rpc calls to the module `yz_solr` thus causing
`undef` failures.

I fixed this by simplifying the entire process of port mapping. It is
now part of the query plan building process and only uses nodes
running the Yokozuna service.

### Switch To Yokozuna Function ###

This is actually a feature to support migraiton. This provides a live
switch of query servces from Riak Search to Yokozuna.

### Only Pass PB Params Which Are Set ###

The Protobuff server was sending query params to Solr which had new
values. If a query (`q`) was provided with a term with no field and
the default field (`df`) is not set then you get a parse issue.

I.e. `q=foo&df=&wt=json...` will cause Solr to blow up.

### Constant Repair ###

Yokozuna's use of `rpc:call` and the process dictionary along with a
small ring and 3 nodes combined to form constant repair of Yokozuna
indexes. The indexes were actually there but AAE didn't think so
because the trees were incorrect. The trees were incorrect because in
some cases hash inserts were going to the wrong tree. Inserts were
going to the wrong tree because the process dictionary key being used
was not unique and two different partitions were going to the same
hashtree pid. By changing the pdcit key to `{tree,Partition}` this
collision is avoided.

While in the repair code I made two other adjustments.

1. Don't RPC call all replicas. Just repair the local index.

2. Only hash the object once, not twice.