Startup: Add ES_HOME to ES_INCLUDE search path

With this change, the elasticsearch script can be linked to another path
without having to set ES_INCLUDE to match the installation path.
Previously, the elasticsearch would find ES_HOME correctly even if linked
but could not find the include script, and finding it would be expected
behavior to me based on its current search path.

Closes #4958
Tests: Improve range tests to check inclusive/exclusive on ends.

closes #8199
TEST : Disable throttleStats test

This test has been intermittently failing, disabling while I dig into it.
Add EnhancedPatternLayout to logging.yml options
Enable ClusterInfoService by default

Since we enabled the disk threshold decider by default, we need to
enable the cluster info service so that disk usages and shard sizes can
be gathered also.

Adds a test that checks that we are gathering information by default.
Term Vectors: support for distributed frequencies

Adds distributed frequencies support for the Term Vectors API. A new parameter
called `dfs` is introduced which defaults to `false`.

Closes #8144
[SEARCH] Passing fieddata_fields as a non array causes OOM

If `fielddata_fields` are passed as a simple value instead of an array
we end up in an infinite loop createing parsed elements with null
values.
This commit validates the incoming token

Closes #8203
[ROUTING] Add rebalance enabled allocation decider

This commit adds the ability to enable / disable relocations
on an entire cluster or on individual indices for either:

 * `primaries` - only primaries can rebalance
 * `replica` - only replicas can rebalance
 * `all` - everything can rebalance (default)
 * `none` - all rebalances are disabled

similar to the allocation enable / disable functionality.

Relates to #7288
[TESTS] Change test node watermarks for DiskThresholdDecider

Ensures that we can still run tests if the machine running them does not
have enough free disk space to be below the high watermark
Recovery: refactor RecoveryTarget state management

This commit rewrites the state controls in the RecoveryTarget family classes to make it easier to guarantee that:
- recovery resources are only cleared once there are no ongoing requests
- recovery is automatically canceled when the target shard is closed/removed
- canceled recoveries do not leave temp files behind when canceled.

Highlights of the change:
1) All temporary files are cleared upon failure/cancel (see #7315 )
2) All newly created files are always temporary
3) Doesn't list local files on the cluster state update thread (which throw unwanted exception)
4) Recoveries are canceled by a listener to IndicesLifecycle.beforeIndexShardClosed, so we don't need to explicitly call it.
5) Simplifies RecoveryListener to only notify when a recovery is done or failed. Removed subtleties like ignore and retry (they are dealt with internally)

Closes #8092 , Closes #7315
Test: additional logging
test: RelocationTests.testCancellationCleansTempFiles may fail due to io errors while check temp files are deleted

Windows can throw NoSuchFileException when using File.walkFileTree and deleting files concurrently. This commit changes IO exceptions into assertion error so that assertBusy will wait for them as well.
[UTILITIES] Introduce a RefCounted interface and basic impl

We already have two places duplicating this rather hairy logic, this
commit intorduces a new RefCoutned interace and an abstract implementation
that can be used for delegation. It factors out all the reference counting
and adds single and multithreaded test for it.

Closes #8210
[CACHE] Expose concurrency_level on all caches

The concurrency level allows to configure the cache internal segments
used to cache data. This can have direct impact on evicition rates since
memory bound caches are equally divided into segments which can cause
early evictions if cache entries are not well balanced.

Relates to #7836
[STORE] Don't catch FNF/NSF exception when reading metadata

When reading metadata we do catch FileNotFound and NoSuchFileExceptions
today, log the even and return an empty metadata object. Yet, in some cases
this might be the wrong thing todo ie. if a commit point is provided these
situations are actually an error and should be rethrown. This commit
pushes the responsiblity to the caller to handle this exception.

Closes #8207
Term Vectors: removed Nullable annotation on a primitive type
ParseField: Support for when all fields are deprecated

Closes #8067
Bulk indexing: Fix 8125 hanged request when auto create index is off.
If a bulk request contains a mix of indexing requests for an existing index and one that needs to be auto-created but a cluster configuration prevents the auto-create of the new index the ingest process hangs. The exception for the failure to create an index was not caught or reported back properly. Added a Junit test to recreate the issue and the associated fix is in TransportBulkAction.

Closes #8125
Docs: fixed typo in documentation

Closes #8205
Docs: rolling upgrade process seems incorrect

When reading the [rolling upgrade process](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html#rolling-upgrades), you can see that we wrote:

* disable allocation
* upgrade node1
* upgrade node2
* upgrade node3
* ...
* enable allocation

That won't work as after a node has been removed and restarted, no shard will be allocated anymore.
So closing node2 and remaining nodes, won't help to serve index and search request anymore.

We should write:

* disable allocation
* upgrade node1
* enable allocation
* wait for shards being recovered on node1
* disable allocation
* upgrade node2
* enable allocation
* wait for shards being recovered on node2
* disable allocation
* upgrade node3
* enable allocation
* wait for shards being recovered on node3
* disable allocation
* ...
* enable allocation

I think this documentation update should go in 1.3, 1.4, 1.x and master branches.

Closes #8218
Closes #7973.