If there is an exception that indicates that the shard isn't available then the exception should bubble up to the TransportShardSingleOperationAction class, so the shard level request can be retried on a different shard. (multi get api)
Added acknowledgment timeout test to OpenCloseIndexTests
Delete Template: When deleting with * and no templates exists, don't 404
closes #3723
improve logging on before/after shared cluster tests
Fix formatting of the documentation.

Remaining '@'s have been replaced with '`'s.
Removed wrong assertion from OpenCloseIndexTests

If a request hasn't been acknowledged, there's no guarantee for any node to hold the up-to-date cluster state (not even the master yet, as the execution is asynchronous)
improve failure message when timing out on green/yellow
call ensureGreen() to get more meaningful debug output on failure
GetActionTests: call ensureGreen() to get more meaningful debug output on failure
call ensureGreen instead of explicit call
escape spaces in JAVA_HOME

fix 3725

(cherry picked from commit a657a511e2231d13079070584b48b5dca0079e10)
Check if nodes have been shut down in TestCluster#beforeTest

The test cluster ignored randomly teared down nodes in tests.
This commit also adds some debug logging to the TestCluster#beforeTest
method.
Updated outdated default setting in doc.
Prevented empty filters from causing NPE

 Closes #3724
Discovery to support a timeout waiting for other nodes to processing new cluster state
The master node processing changes to cluster state, and part of the processing is publishing the cluster state to other nodes. It does not wait for the cluster state to be processed on the other nodes before it moves on to the next cluster state processing job.

This is fine, we support out of order cluster state events using versioning, and nodes can handle those cases. It does lead though to non optimal API semantics. For example, when issuing cluster health, and waiting for green state, the master node will report back once the cluster is green based on its cluster state, but that mentioned "green" state might not have been received by all other nodes yet.

Add a discovery.zen.publish_timeout setting, and default it to 5s. This will give a best effort into making sure all nodes will process a cluster state within a window of time.

closes #3736
Make sure the transport response handler is invoked only once.

There was a small window of time where the transport response handler's handException method was invoked twice. As far as I can tell this happened when node disconnect event was processed just after the request was registered and between a "Node not connected" error was thrown. The TransportService#sendRequest method would invoke the transport response handler's handException method regardless if it was already invoked. This resulted that for one request failure, two retries were executed.

The mpercolate api has an assert that tripped when more than the expected shard level responses were returned. This was caused by the issue described above. For the a single shard level request we had multiple responses and this broke the the the total excepted responses. Also the reduce could be started prematurely, which resulted in an incorrect final response (e.g. total count being incorrect). For example: two shards in total, shard 0 gets reduces twice. The second shard 0 response gets in just before shard 1 response gets in. The reduce starts without shard 1 response.
Added more information to assertion in completion suggest tests
add explicit refresh post failure to test
log the hits post explicit refresh, and post explicit wait for green
Add asserting searcher to track searcher references

This assertion module also injects an AssertingIndexSearcher that
checks if our queries are all compliant with the lucene specification
which is improtant for future updates and changes in the upstream project.
detect JRE (vs JDK) and invalid JAVA_HOME

fixes #3739

(cherry picked from commit 9fe99aa6e947a86f4558b56f691e2376da946897)