basho/riak_kv — pull request #263 by beerriot (2011-12-14 22:47:35+00:00)
Title: Ack-based Listkeys Backpressure

To prevent keylist results from piling up in a requester's mailbox, this patch changes the protocol between the pieces to require acknowledgements of each batch before the next batch is sent. It depends on new code in riak_core: https://github.com/basho/riak_core/pull/123

Specifically, each vnode (or async vnode worker) requires that its keylist batch messages to the keylist fsm or pipe worker be acknowledged before the next batch is sent. In addition, the keylist fsm also requires that its consumers (the PB socket and WM resource) acknowledge batch deliveries before continuing as well. These acks are invisible to the outside, handled internally by all user interfaces.

## Notes
- this probably could have been done without a new `listkeys_req` vnode command, but the separation of code that it allowed made things a bit clearer, imo
- performance testing on a 4-node VM dev box showed this to increase latency slightly, but with the benefit of never triggering the memory high watermark or other warnings
- the latency/memory usage tradeoff can be tweaked by the `key_buffer_size` appenv setting in riak_kv (a higher value will mean fewer messages to ack, but also larger messages waiting in mailboxes)
- this is specifically meant to solve https://issues.basho.com/show_bug.cgi?id=1286

## Testing

Set the riak_kv environment variable `liskeys_backpressure` to `true`.

Load a large number of keys into a Riak bucket. Request the keys as a stream. Over HTTP, this looks like:

curl http://localhost:8098/riak/bucket?keys=stream

All listkeys requests over PB are streamed. Using non-streamed keylist requests will confuse the results, as the keylist will still accumulate in memory, just not in a process mailbox.

Watch the console for memory errors. To reproduce the exact case of BZ1286, you'll need a keyspace large enough to trick the Erlang VM into attempting to allocate all of it. With slightly smaller, yet still large keyspaces you'll be watching for `long_gc` and `memory_high_watermark`.