Fixed explanation for GaussDecayFunction

The explanation now gives the correct value instead of the negative.
[TEST] Fix test explain now that explanation is fixed
Tests: added trace action.search.type to GeoBoundsTests
[TEST] Expose ability to provide http headers when sending requests in our REST tests

ElasticsearchRestTests has now a `restClientSettings` method that can be overriden to provide headers as settings (similarly to what we do with transport client). Those headers will be sent together with every REST requests within the tests.

Closes #7710
Internal: make sure that original headers are used when executing search as part of put warmer

Closes #7711
Test: ZenUnicastDiscoveryTests.testNormalClusterForming should start unicast hosts first

The test starts a cluster with random nodes as unicast hosts but *doesn't* use min_master_nodes. If the unicast hosts are started last, nodes may elect themselves as master as they do not have mechanism yet to share information.
function score: fix cast in Gaussian decay function

Also fix the test
FunctionScoreTests#simpleWeightedFunctionsTestWithRandomWeightsAndRandomCombineMode
which sometimes failed due to rounding issues. Make sure
only floats are returned as scores to assure ratio of
expected and returned score is 1.0f.
Aggregations: Adds ability to sort on multiple criteria

The terms aggregation can now support sorting on multiple criteria by replacing the sort object with an array or sort object whose order signifies the priority of the sort. The existing syntax for sorting on a single criteria also still works.

Contributes to #6917
Replaces #7588
Only set `breaker` when stats are retrieved

When communicating with 1.3 and earlier nodes, it's possible that the
field data breaker info is not sent at all. When this happens, we should
leave the `breaker` variable as-is (unset) instead of creating an
AllCircuitBreakerStats object with a null fd breaker and fake request &
parent breakers.
Added scrollId/s setters to the different scroll requests/responses
[TEST] Minor  REST tests infra cleanup

Make the http addresses within the REST client final. It makes no sense to update them before each test if we don't check the version of the nodes again, which would mean adding too much overhead (an additional http call before each test) for no reason. We just reuse the same nodes for the whole suite and check the version once while initializing the client. Would be nice to make the REST client within the execution context final but its initialization still needs to happen after the `ElasticsearchIntegrationTest#beforeInternal` that assigns `GLOBAL_CLUSTER` to `currentCluster`.

Closes #7723
Recovery: remove unneeded waits on recovery cancellation

When cancelling recoveries, we wait for up to 10s for the source node to be notified before continuing. This is not needed in two cases:
1) The source node has been disconnected due to node shutdown (recovery is canceled as a response to cluster state processing)
2) The current thread is the one that will be notifying the source node (happens when one of the calls from the source nodes discoveres the local index is closed)

The first one is especially important as it may delay cluster state update processing with 10s.

Closes #7717
[TEST] Log if we use transport client in trace mode
Test: always run CheckIndex for these two tests
Discovery: UnicastZenPing - use temporary node ids if can't resolve node by it's address

The Unicast Zen Ping mechanism is configured to ping certain host:port combinations in order to discover other node. Since this is only a ping, we do not setup a full connection but rather do a light connect with one channel. This light connection is closed at the end of the pinging.

During pinging, we may discover disco nodes which are not yet connected (via temporalResponses). UnicastZenPing will setup the same light connection for those node. However, during pinging a cluster state may arrive with those nodes in it. In that case , we will mistakenly believe those nodes are connected and at the end of pinging we will mistakenly disconnect those valid node.

This commit makes sure that all nodes UnicastZenPing connects to have a unique id and can be safely disconnected.

Closes #7719
Netty: Make sure channel closing never happens on i/o thread

Similar to NettyTransport.doStop() all actions which disconnect
from a node (and thus call awaitUnterruptibly) should not be executed
on the I/O thread.

This patch ensures that all disconnects happen in the generic threadpool, trying to avoid unnecessary `disconnectFromNode` calls.

Also added a missing return statement in case the component was not yet
started when catching an exception on the netty layer.

Closes #7726
Tools: Add script to grab ES version for BWC tests.

closes #7653
Internal: ClusterHealthAPI does not respect waitForEvents when local flag is set

It uses a cluster state update task and it gets rejected if not run on a master node. We should enable running on non-masters if the local flag is set.

Also, report any unexpected error that may happen during this cluster state update task

Closes #7731
Discovery: not all master election related cluster state update task use Priority.IMMEDIATE

Most notably the elected_as_master task should run as soon as possible. This is an issue as node join request do use `Priority.IMMEDIATE` and can be unjustly rejected.

Closes #7718
[TEST] Use node names for transport clients an close them in tests