basho/yokozuna — pull request #103 by rzezeski (2013-05-02 21:58:00+00:00)
Title: Use mochiglobal to cache + get coverage plans

Creating a coverage plan consumes a lot of CPU and creates potentially
large lists (garbage) on the process potentially leading to GC. This
gets worse as the ring size increases.

Currently Yokozuna creates a coverage plan _per_ request. I.e. every
search request must first pay the expensive cost of calculating a plan
before running the search. But it doesn't have to be this way. The
plans could be periodically calculated in an out-of-band process and
cached in a good read-heavy solution like mochiglobal.

There is no reason to calc a plan per request. The argument could be
made that this is the only way to avoid sending requests to downed
nodes but that is false. Coverage relies on nodewatcher which uses
net-kernel tick which runs on a delay of ~15s.

The other argument could be ring change (join/remove a node). In this
case the cached plan might be in disagreement with the ring. The
divergence would last for a maximum upper bound of T where T is the
period of time before a new plan is written to the index. This is
already a problem in Riak where the ring changes between the time a
coordinator fires of a request and the vnodes see it. I'm just going
to punt and assume vnode forwarding should take care of this. Even if
this isn't the case this scenario is made more rare by the fact that
the partition that has moved must be part of the plan. This can also
be made even more rare by invalidating the cache on ring change events.

This patch makes the following changes:

* A new plan is generated for each index every 2 seconds.

* During a search request the cache is first checked. The cache uses
mochiglobal which generates BEAM code dynamically so that the cost
of a fetch is that of a function call.

* If the cache doesn't have a plan stored for the index then a plan
will be generated as part of the request.

* The `yz_cover` module is now a `gen_sever`. It's responsibility is
to periodically re-generate plans for each index and cache them.

NOTE: Another thing that might worry someone about this patch is that
using the same plan means not load-balancing well. But this matters
less re Yokozuna for two reasons.

1. Yokozuna doesn't have vnodes, so balancing across vnodes doesn't
matter.

2. Coverage currently seems to always select all nodes. Once coverage
gets better at optimizing for nodes then it would be wise to cache
multiple plans and round-robin them.