elasticsearch/elasticsearch — pull request #8436 by s1monw (2014-11-11 13:01:43+00:00)
Title: [CORE] Intorduce shards level locks to prevent concurrent shard modifications

Today it's possible that the data directory for a single shard is used by more than on
IndexShard->Store instances. While one shard is already closed but has a concurrent recovery
running and a new shard is creating it's engine files can conflict and data can potentially
be lost. We also remove shards data without checking if there are still users of the files
or if files are still open which can cause pending writes / flushes or the delete operation
to fail. If the latter is the case the index might be treated as a dangeling index and is brought
back to life at a later point in time.

This commit introduces a shard level filesystem lock that prevents modifications to the shard data
while it's still in use. Locks are created per shard following this structure:

`{data.dir}/locks/{index.name}/shard_{id}`

This commit also adds infrastructure that checks if all shard locks are released after tests.