Fix issue where entire shards are lost due to too many open files exceptions and a but in Lucene's
IndexWriter / DirectoryReader where an existsing index was not detected and then deleted due to a wrong
creation mode. See LUCENE-4870

Closes #2812