basho/riak_cs — pull request #570 by arekinath (2013-05-24 01:38:05+00:00)
Title: Limit number of blocks buffered in memory during fetch

We've been trying out Riak CS for storing large downloads for our users (usually 1-4GB files) and finding that our nodes were randomly falling over!

After some investigation, we found that the Riak CS processes were dying after hitting their zones' swap memory caps (this is on Solaris), and that this was very easily reproduced simply by downloading a 4GB file from multiple clients at once.

What's happening is that the `riak_cs_get_fsm` is managing to fetch most of the blocks for these huge files very very quickly, and then they sit in RAM (inside that process' `got_block` orddict) for ages while waiting to be written out to the client. This seems to be due to the fact that our intra-cluster links are much much faster than the links to the clients downloading files. This is fine if the file is small, but on a 4GB file we are finding that that one dictionary can expand to 3 or 3.5GB in RAM for every single client trying to download it. This is a really big problem.

I've put together a general idea of a solution here, which I've tested and is working fine for us. In `riak_cs_get_fsm:waiting_chunks/2`, instead of unconditionally fetching the next block if one is on the queue as soon as the last one is received, I check whether we already have "too many" blocks in the orddict (the definition of "too many" is controlled by a new config tweakable). If we do, then no new block is fetched at that point. Then, in the handler for the `get_next_chunk` message, after sending out the block, we check if we are below the no-fetch threshold. If we are, we re-enter the `waiting_chunks` state with a 0 timeout. Then in the `timeout` message handler we start any fetch operations that we can.

The reason why I defer the call to `read_chunks/1` in the `get_next_chunk` handler using a `gen_fsm` timeout is to avoid any unnecessary delay when there lots of `get_next_chunk` messages queued. Fetching new chunks will wait until the current messages in the queue have all been handled. I found this to give better performance than calling `read_chunks/1` straight away.

As an aside, in the `timeout` handler, I have added a manual call to `erlang:collect_garbage/0`, because I was finding that the GC would not run often enough on the `riak_cs_get_fsm`s without it when they got very busy. This is quite common with processes that deal with relatively few messages/reductions that each move a lot of data (exactly what the `get_fsm` does)