[ENGINE] try increment store before searcher is acquired

InternalEngine#refreshNeeded must increment the ref count on the
store used before it's checking if the searcher is current since
internally a searcher ref is acquired and if that happens concurrently
to a engine close it might violate the assumption that all files
are closed when the store is closed.

This commit also converts some try / finally into try / with.
[TEST] move REST tests to their own test group

Closes #7795
Bulk API: Do not fail whole request on closed index

The bulk API request was marked as completely failed,
in case a request with a closed index was referred in
any of the requests inside of a bulk one.

Implementation Note: Currently the implementation is a bit more verbose in order to prevent an instanceof check and another cast - if that is fast enough, we could execute that logic only once at the beginning of the loop (thinking this might be a bit overoptimization here).

Closes #6410
Bulk Request : Add Document Request

This file was missing.
Revert "Bulk Request : Add Document Request"

This reverts commit 86f575dcea31e2e1486a17ed248708084905aeea.
Revert "Bulk API: Do not fail whole request on closed index"

This reverts commit 405e5816b827db5823e6c2ed9c1bbe9fa4629355.
Bulk API: Do not fail whole request on closed index

The bulk API request was marked as completely failed,
in case a request with a closed index was referred in
any of the requests inside of a bulk one.

Implementation Note: Currently the implementation is a bit more verbose in order to prevent an instanceof check and another cast - if that is fast enough, we could execute that logic only once at the
beginning of the loop (thinking this might be a bit overoptimization here).

Closes #6410
Add option to take currently relocating shards' sizes into account

When using the DiskThresholdDecider, it's possible that shards could
already be marked as relocating to the node being evaluated. This commit
adds a new setting `cluster.routing.allocation.disk.include_relocations`
which adds the size of the shards currently being relocated to this node
to the node's used disk space.

This new option defaults to `true`, however it's possible to
over-estimate the usage for a node if the relocation is already
partially complete, for instance:

A node with a 10gb shard that's 45% of the way through a relocation
would add 10gb + (.45 * 10) = 14.5gb to the node's disk usage before
examining the watermarks to see if a new shard can be allocated.

Fixes #7753
Relates to #6168
Indexed Scripts/Templates :  Cleanup

This contains several cleanups to the indexed scripts.
Remove the unused FetchSourceContext from the Get request..
Add lang,_version,_id to the REST GET API.
Removes the routing from GetIndexedScriptRequest since the script index is a single shard that is replicated across all nodes.
Fix backward compatible template file reference
Before 1.3.0 on disk scripts could be referenced by requesting
````
_search/template

{
  "template" : "ondiskscript"
}
````
This was broken in 1.3.0 by requiring
````
{
  "template" :
  {
    "file" : "ondiskscript"
  }
}
````
This commit restores the previous behavior.
Remove support for preference, realtime and refresh
These parameters don't make sense anymore for indexed scripts as we always force the preference to _local and
always refresh after a Put to the indexed scripts index.

Closes #7568
Closes #7559
Closes #7647
Closes #7567
Internal: remove ForceSyncDirectory

Historical code, not used anymore.

Closes #7804
Core: Check if from + size don't cause overflow and fail with a better error.

Closes #7778
Discovery: Give a unique id to each ping response

During discovery a node gossips with other nodes to discover the current state of the cluster - what nodes are out there, what version they use and most importantly whether there is an active master out there. During this ping process we may end up in a situation where old information is mixed with new. This is comment if a couple of master election happen in rapid succession.

This commit adds a monotonically increasing id to each ping response. This makes it easy to always select the last ping from every node.

Closes #7769
Test: RecoveryWhileUnderLoadTests.recoverWhileRelocating should report cluster state when failing to reach green
Recovery: mapping check during phase2 should be done in cluster state update task

Before phase2 we check verify that the local mapping is in sync with the cluster state mapping (and send & wait on a master update mapping task if not). This check should be done under a cluster state update task to make sure an incoming cluster state update to do not change things while we check.

Closes #7744
Tests: extend testRecoverFromPreviousVersion to sometimes index during relocation

Relates to #7729

Closes #7768
Update java source example

From the version 1.0 FilterBuilders and QueryBuilders are not part from org.elasticsearch.index.query.xcontent package no more.

Closes #7701.

(cherry picked from commit 32d4200)
Apply bulk change to 1.3.3

relates to #7729
Docs: IndexResponse.matches() does not exist anymore

Since 1.0, percolator has been redesigned so percolator is not applied anymore at index time.

Closes #7548.
Document the Java BulkProcessor

Closes #7638.
[TEST] Ensure primaries are allocated before bulk indexing with dymamic mappings