basho/yokozuna — pull request #133 by rzezeski (2013-06-25 21:34:14+00:00)
Title: Add map-reduce support

Add ability to use Yokozuna query as input for map-reduce job.

* Add a `mapred_search/3` API used to run map-reduce jobs with query
results generating bucket/key pairs as input.

* Add a `search_fold/5` API used to fold over all results matching a
query. Thus, if a query has 143 matches all 143 will be iterated
and fed to the fold function `F` passed as argument.

* Add a test to verify map-reduce support works. It writes 100
objects and runs a map-reduce job whose query matches all 100
objects. The job sums the number of objects matched and verifies
that the value is 100.

* Added the KVC repo as a dependency. It allows convenient extraction
of proplists and mochijson structures.

* The `Timeout` parameter to `mapred_search` is ignored. According to
Bryan Fink this timeout is for the duration of the entire job, not
per-batch (which is what Riak Search is using it for). He also
mentioned the existing endpoints such as PB and HTTP already enforce
this timeout and it probably doesn't make sense to also track it in
Yokozuna. I think this makes sense because if Yokozuna also tracked
it then what happens when both systems decide to deal with the
timeout and kill various processes? I imagine that could cause
other complications. In the future this parameter should be looked
at carefully and potentially removed from this part of the
map-reduce system completely.

* Yokozuna feeds results into pipe/map-reduce one at a time. It would
be nice to have a batch insert to save a bit of work.

* Yokozuna doesn't return positions and score in the `KeyData` but
instead passes a hard-coded empty list (`[]`). Passing in positions
is expensive and not guaranteed to be available for every field in
Yokozuna. I'm not sure if anyone was actually using this key data
so I opted to hard-code it to an empty list for now.

* The `search_fold` iteration is hard-coded to batches of 10. In the
future this should be configurable and benchmarked.