[TEST] wait for yellow else assertSearchResponse will trip
 [TEST] wait for yellow else assertSearchResponse will trip
Discovery: remove any local state and use clusterService.state instead

At the moment, ZenDiscovery contains a local copy of the disco nodes plus a flag that indicates whether the local node is master or not. This is redundant as the same information is stored in the cluster state. Have duplicate copy can lead to unneeded concurrency issues. This PR removes the duplication, including moving the ownership of the localNode creation to ClusterState

The PR introduces a tighter control of the background joining thread to make sure it is started and stopped together with any cluster state changes. This solves potentially concurrency bugs where a joining thread may fail to start.

Last we add a couple of safety checks to make sure that if a nodes receives a cluster state from a new master while actively trying to join another one (or electing itself) we go back to pinging to actively join it.

Closes #7834
NodesFD: simplify concurrency control to fully rely on a single map

The node fault detection class is used by the master node to ping the nodes in the cluster and verify they are alive. This PR simplifies the concurrency controls in the class + adds a test for a scenario that surfaced the problem.

Closes #7889
node shutdown: make close() syncronized

An example scenario where this will help:

When the node is shutdown via api call
(https://github.com/elasticsearch/elasticsearch/blob/master/src/test/java/org/elasticsearch/test/ExternalNode.java#L219 )
then the call returns immediately even if the node is not actually shutdown yet
(https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/action/admin/cluster/node/shutdown/TransportNodesShutdownAction.java#L226).
If at the same time the proces is killed, then the hook that would usually prevent
uncontrolled shutdown
(https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/bootstrap/Bootstrap.java#L75)
has no effect: It again calls close() which might then just return
for example because one of the lifecycles was moved to closed already.

The bwc test FunctionScoreBackwardCompatibilityTests.testSimpleFunctionScoreParsingWorks
failed because of this. The translog was not properly
written because if the shutdown was called via api, the following process.destroy()
(https://github.com/elasticsearch/elasticsearch/blob/master/src/test/java/org/elasticsearch/test/ExternalNode.java#L225)
killed the node before the translog was written to disk.

closes #7885
Internal: split internal fetch request used within scroll and search

Similar to #7856 but relates to the fetch shard level requests. We currently use the same internal request when we need to fetch within search and scroll. The two original requests though diverged after #6933 as SearchRequest implements IndicesRequest while SearchScrollRequest doesn't. That said, with #7319 we made `FetchSearchRequest` implement IndicesRequest by making it hold the original indices taken from the original request, which are null if the fetch was originated by a search scroll, and that is why original indices are optional there.

This commit introduces a separate fetch request and transport action for scroll, which doesn't hold original indices. The new action is only used against nodes that expose it, the previous action name will be used for nodes older than 1.4.0.Beta1.

As a result, in 1.4 we have a new `indices:data/read/search[phase/fetch/id/scroll]` action that is equivalent to the previous `indices:data/read/search[phase/fetch/id]` whose request implements now IndicesRequest and holds the original indices coming from the original request. The original indices in the latter request can only be null during a rolling upgrade (already existing version checks make sure that serialization is bw compatible), when some nodes are still < 1.4.

Closes #7870
Docs: Removed all the added/deprecated tags from 1.x
Docs: Fixed some deleted refs
If a node is being shutdown some in flight ping request may be executed. Make sure to keep track of those ping requests and close the unicast connect executor service.

Closes #7903
Internal: MulticastChannel should wait on receiver thread to stop during shutdown

This was signaled by our tests which shutdown class and check for thread leakage.

Closes #7835
Update suggesters.asciidoc

A request was malformed

Closes #7867
GeoJSON is lon,lat

Although emphasized in the text, the example was backwards.

Closes #7843
Improve sentence structure

Closes #7842
Remove comma in JSON

Closes #7827
Fix Idaho spelling

Doesn't have two `d`s.

Closes #7802
Clarify s3 snapshot compress behavior

Clarify s3 snapshot compress behavior only applies to metadata and no index files.

Closes #7654
fix concurrency bug in index throttling
Core: upgrade to Lucene 4.10.1
fix typo: sonartype -> Sonatype
Refactor the Translog.read(Location) method

It was only used by `readSource`, it has been changed to return a
Translog.Operation, which can have .getSource() called on it to return
the source. `readSource` has been removed.

This also removes the checked IOException, any exception thrown is
unexpected and should throw a runtime exception.

Moves the ReleasableBytesStreamOutput allocation into the body of the
try-catch block so the lock can be released in the event of an exception
during allocation.