basho/riak-erlang-client — pull request #59 by beerriot (2012-05-25 21:23:50+00:00)
Title: Accumulate single-keep mapred query outputs more efficiently

As https://github.com/basho/riak_kv/pull/333 was to https://github.com/basho/riak_kv/pull/331, so is this PR to #58. That is, this patch substitutes a reverse list building with the cheaper cons for the repeated more expensive list appending.

Testing should run similarly:

```erlang
%% open a socket
{ok, S} = riakc_pb_socket:start("127.0.0.1", 8081).

%% Load 50k objects into the gmrf bucket -- no need if you've done this in PR #58
[ riakc_pb_socket:put(S, riakc_obj:new(<<"gmrf">>, list_to_binary(integer_to_list(N)), <<"hello">>)) || N <- lists:seq(1, 50000) ].

%% Two reduces in a row
timer:tc(riakc_pb_socket, mapred, [S, <<"fmgr">>, [{reduce, {modfun, riak_kv_mapreduce, reduce_identity}, undefined, true}, {reduce, {modfun, riak_kv_mapreduce, reduce_identity}, undefined, true}]]).

%% Two maps in a row
timer:tc(riakc_pb_socket, mapred, [S, <<"gmrf">>, [{map, {jsanon, <<"function(v) { return [ [v.bucket, v.key] ]; }">>}, undefined, true},{map, {jsanon, <<"function(v) { return [v.key]; }">>}, undefined, true}]]).
```

On a Mac laptop with the eleveldb backend, I see the two-reduce query drop from 43 seconds without the patch to 30 seconds with the patch. I also see the two-map query drop from 63 seconds to 43 seconds.