basho/yokozuna — pull request #76 by rzezeski (2013-03-15 03:30:29+00:00)
Title: Add filter query compression

After running some benchmarks I discovered that query throughput can
be increased by reducing the number of term queries in the 'fq' param
passed to Solr.

Currently Yokozun passes an 'fq' like so:

(nodeA AND p1) OR (nodeB AND p4) OR (nodeC AND p7) OR (nodeA AND p10) ...

Notice that the nodes repeat for each node-partition combo. This
"compresses" the 'fq' to the following:

(nodeA AND (p1 OR p10 OR ...)) OR (nodeB AND (p4 OR ...)) OR ...

Notice that the partitions are now grouped with the corresponding
owning node. On my local workstation I noticed a ~17% improvement in
throughput. As a lot of the filter query work is done in memory it
should be CPU bound so the slower the CPU the more this change should
benefit.