basho/bitcask — pull request #73 by jtuple (2012-12-03 21:36:24+00:00)
Title: Use Erlang file I/O from dedicated procs rather than NIFs

Bitcask previously used raw file I/O to read/write files. However, since raw file I/O uses a non-optimized selective receive to wait for a reply back from the efile driver, this approach had numerous problems when Bitcask was used within processes with many incoming messages (such as how Bitcask is used in Riak).

In commit 79d5eb3470db7691f4584004563c1b2f19eb0ad4, NIFs were introduced to solve this problem. The file I/O NIFs would block the Erlang scheduler, but solve the issue encountered with selective receive. Unfortunately, using blocking NIFs is much worse than originally thought. Thus, blocking NIFs are not the right solution to this problem.

This commit changes Bitcask to once again use Erlang's built-in file I/O, but now wraps each open file in a separate gen_server that interacts with the raw port. The original process now waits on a gen_server reply which uses an optimized selective receive, while the file process handles the unoptimized selective receive from the port driver. In our usage, the file process only has a single request outstanding, and therefore does not run into the selective receive issue.