neo4j/community — pull request #51 by vivekprahlad (2011-10-08 17:56:56+00:00)
Title: Fixed issue with lucene indexes: too many open files

Hi,

I've been contributing recently to the Neo4j ruby bindings. Recently, we started running into an issue where the tests would fail with a cryptic 'too many open files' message. (See http://travis-ci.org/#!/andreasronge/neo4j/builds/202125). On my mac, lsof showed that a large number of lucene index files remained open while the neo4j rb tests were running. After some research, I figured out that the problem is that the Lucene IndexWriters and IndexSearchers in the LuceneDataSource are always kept open, and the cache is essentially unbounded.

On linux, the default per process limit for the maximum number of open files is 1024 (see http://www.cs.wisc.edu/condor/condorg/linux_scalability.html). The only time the searchers / writers are closed is when an index is deleted. As each index seems to result in approximately 6-7 files being opened, it is possible that with a large number of indices, the limit for the maximum number of open files can be reached fairly quickly. It looks like Neo4j users have run into this issue in the past as well, but the diagnosis was inconclusive: http://neo4j.org/forums/#nabble-td2754679

Implementation notes:
Two LRU caches have been introduced for Lucene IndexWriters and IndexSearchers. The configuration is provided by the following parameters:

lucene_searcher_cache_size (integer, specifies the maximum number of open searchers)
lucene_writer_cache_size (integer, specifies the maximum number of open writers)

A unit test has been added for the LuceneDataSource. This test checks the behaviour of the cache. The way the searchers were being invalidated has been modified slightly, which resulted in a slightly simpler implementation.

I will appreciate any feedback about this pull request. BTW, I've already sent the CLA as per the instructions here: http://wiki.neo4j.org/content/About_Contributor_License_Agreement

Cheers!
Vivek