[TEST] wait unitl all machines joined the cluster
Snapshot/Restore: switch to write once mode for snapshot metadata files

This commit removes creation of in-progress snapshot file and makes creation of the final snapshot file atomic.

Fixes #8696
Clarify InternalIndexShard callbacks

This commit changes internal callback to be clear
about when they are called and also provide the
exception that was potentially thrown as a callback argument.

Closes #5945
Docs: add randomizedtesting-runner to testing-framework.asciidoc

Close #8450
[ALLOCATION] Speed-up disk-threshold decider

Instead of iterating all shards of all indices to get all relocating
shards for a given node we can just use the RoutingNode#shardsWithState
method and fetch all INITIALIZING / RELOCATING shards and check if they
are relocating. This operation is much faster and uses pre-build
data-structures.

Relates to #6372
Tests: move RecoverAfterNodesTests to org.elasticsearch.gateway.local and increase BLOCK_WAIT_TIMEOUT to 10s

The tests were still in org.elasticsearch.gateway.none but the none gateway was removed.
Packaging: Check if proc file exists before calling sysctl

The packaged init scripts could return an error, if the file
/proc/sys/vm/max_map_count was not existing and we still called
sysctl.

This is primarly to prevent confusing error messages when elasticsearch
is started under virtualized environments without a proc file system.

Closes #4978
Scripting: Add explicit error message when script_score script returns NaN

When a scoring script returns not a number, the current message is confusing (IllegalArgumentException[docID must be >= 0 and < maxDoc=3 (got docID=2147483647)]). This commit adds the error message ScriptException[script score function returns a wrong score: NaN].

Closes #2426
Add more trace loggin to LocalIndexShardGateway.java
[STORE] Expose ShardId via LeafReader rather than Direcotry API

Today we try to fetch a shard Id for a given IndexReader / LeafReader
by walking it's tree until the lucene internal SegmentReader and then
casting the directory into a StoreDirecotory. This class is fully internal
to Elasticsearch and should not be exposed outside of the Store.

This commit makes StoreDirectory a private inner class and adds dedicated
ElasticsearchDirectoryReader / ElasticserachLeafReader exposing a ShardId
getter to obtain information about the shard the index / segment belogs to.

These classes can be used to expose other segment specific information in
the future more easily.
[TEST] Add unit tests for DiskThresholdDecider settings
Internal: allow InternalEngine to be stopped and started

Once the current engine is started you can only close it once. Once closed the engine cannot be started again. This commit adds a stop method which signals the engine to free it's resources but in a way that allows restarting.

This is done by introducing InternalEngineHolder which is a wrapper around InternalEngine. This allows to add the stop() method without adding complexity the engine implementation. InternalEngineHolder also serves an entry point for listeners (incoming and outgoing) to other ES components, which removes the needs add/remove them if the engine is stopped.

Closes #8784
[TEST] Pass class level test logging to external nodes

This commit passes the test logging annotation from the class
level to the external nodes as well.

Closes #8552
docs: add pgp key to repositories page
Plugins: Plugin failed to load since #8666

The method Path.endsWith(String s) doesn't work exactly the same way as String.endsWith() (see http://docs.oracle.com/javase/7/docs/api/java/nio/file/Path.html#endsWith(java.nio.file.Path)). This blocks the loading of plugins.
Merge branch 'fix/plugins-loading'
[ENGINE] Add engine lifecycle store reference to EngineHolder

This commit add the engines reference to the store out of the actual
implementation into the hodler since the holder manages the actual lifcycle.
Engine internal references like per searcher or per recovery are kept inside
the actual implemenation since the have a different lifecycle.
Test: fix this test to work in IntelliJ
fix cache key methods on leaf reader
Add test that ES filterreader getCoreCacheKey() behaves correctly.

Closes #8831.