basho/bitcask — pull request #122 by evanmcc (2013-11-19 23:14:47+00:00)
Title: Bound merge queue to number of partitions.

### What happens:

In situations where new merge items are getting enqueued more quickly than the merge worker can work through them, memory can grow without bound. This is made especially bad by the representation of the files, which are very long erlang strings, which are then reproduced in the next merge of the same partition. In certain cases, we've seen nodes fall so far behind that their queues have reached into the thousands and the nodes have crashed with the `bitcask_merge_worker` having a heap size > 10GB.

### What's changing:

Rather than use erlang's `queue`, this change uses a plain list of tuples, and uses `keyfind` on the directory name to find items from the merging partition that have been previously enqueued. It then merges the two lists, and the list of options, and replaces the partition in place. New items are appended to the end of the list, which should be cheap, given that the list should never get very long.

#### Testing notes:

I tested this patch in an indentical set up to the one described in #116. On top of the patch, I added a 5 second sleep to `bitcask_merge_worker:do_merge()` to cause it to fall behind. I then monitored the log output, the size of the files on disk, and checked the queue length with the following lovely snippet:

```erlang
length(element(2, element(2,lists:nth(1,element(2,lists:nth(3,lists:nth(5, element(4, sys:get_status(bitcask_merge_worker))))))))).
```

The expected behavior without the patch is that riak will crash when it runs out of memory, if it doesn't run out of space first, and the queue will become very long. With it, the queue length should be limited to the number of vnodes (less one, really, as it will always be processing at least one) and disk growth will be checked but perhaps not eliminated entirely. Merge messages will become larger as more files are added, and merges will take longer.