basho/bitcask — pull request #61 by slfritchie (2012-10-17 04:43:36+00:00)
Title: Fix file descriptor leak after start() after AC power failure

A customer reports an eleveldb problem (`emfile` errors due to file
descriptor leaking) that was apparently the result of the twisted
flaming wreckage of:

1. A large ring size on a small cluster of machines (exact ring
size unknown, cluster size likely 5 or 7 nodes)

2. Under an extremely heavy Riak CS workload, which was under a
high stress workload involving exclusively files at 10+ MByte and
thus writing much more data to Bitcask than a high stress workload
involving 10 byte files.

3. The power plug was pulled/tripped/popped on one of the Riak
nodes.

4. The machine promptly rebooted, checked file system state, and
attempted to restart Riak.

5. Riak restarted with most/all? Bitcask instances complaining about
truncated data & hint files but appeared to start them successfully.

6. However, file descriptor leaks caused eleveldb instances to crash
with `emfile` errors.

This fix introduces a new parameter to `scan_key_files()`, telling
it to close files (or not) as it folds across each data file.

The two test functions, `t0/0` and `t1/0`, are added to show the
effect of the file descriptor leakage. The former shows how
the leak at `bitcask:start()` is addressed by this patch. Leaks
via merge or delete as shown by `t1/0` are not addressed. Both
functions are not really demos and not yet eunit tests (due to
relying on platform-specific utilities like `seq` and `lsof`).