basho/riak_kv — pull request #331 by beerriot (2012-05-25 15:48:11+00:00)
Title: Speed up accumulation of results for single-keep MR queries

When delivering non-streamed MapReduce results to an HTTP client, we have to group the results by the phase that produced them. This PR improves the performance of that grouping when only a single phase is producing results, by avoiding a repeated append operation on a continually growing list.

Simple testing can be done by loading 10k or more keys into a bucket, then MapReducing on the keylist. The magnitude of the improvement is more obvious if an empty query, or a single reduce phase is used. If a map is added to the query, the time spent reading and decoding objects can push the improvement into the noise.

```
# Load 10k objects into bucket `fmgr`
for i in {1..10000}; do curl -X PUT -H "content-type: text/plain" http://localhost:8091/riak/fmgr/$i --data "hello"; done

# Keylist -> Reduce-identity query
time curl -s -X POST -H "content-type: application/json" http://localhost:8091/mapred --data "{\"inputs\":\"fmgr\",\"query\":[{\"reduce\":{\"language\":\"erlang\",\"module\":\"riak_kv_mapreduce\",\"function\":\"reduce_identity\",\"keep\":true}}]}" > /dev/null

# Keylist -> Empty query
time curl -s -X POST -H "content-type: application/json" http://localhost:8091/mapred --data "{\"inputs\":\"fmgr\",\"query\":[]}" > /dev/null
```

On a Mac laptop with the eleveldb backend, I see the reduce-identity query drop from ~2.1 without this patch to ~1.3 seconds with this patch. On the same system, I see the empty query drop from ~1.5 to ~0.7 seconds.