basho/bitcask — pull request #121 by evanmcc (2013-11-15 21:47:06+00:00)
Title: Fix GC issue with long-lived worker process.

There exists a degenerate case where the merge worker seems to bloat its heap more or less infinitely until the machine crashes. The preconditions are large amounts of merge-deletable data and settings that effectively guarantee constant merging.

The theory is that this keeps the bitcask_merge_worker gen_server from ever going idle and getting its heap size reduced. Thus, every garbage collection can potentially cause the heap to increase in size. This will cause the nursery generation to grow in size, causing longer and longer gaps between GCs, until we run out of memory.

Since solving the garbage collection problem is hard, we punt: spawn the actual merge in its own process, keeping the amount of work done by the merge_worker to a minimum. The heap will be cleaned up when the process in which it happened exits, so it won't have a chance to engage in unbounded growth.