[Store] delete unallocated shards under a cluster state task

This is to prevent a rare racing condition where the very same shard gets allocated to the node after our sanity check that the cluster state didn't check and the actual deletion of the files.

Closes #6902
Added three frequency levels for resource watching

 It's now possible to register watchers along with a specified check frequency. There are three frequencies: low, medium, high. Each one is associated with a check interval that determines how frequent the watchers will check for changes and notify listeners if needed. By default, the intervals are 5s, 30s and 60s respectively, but they can also be customized in the settings. also:

  - Added the WatcherHandle construct by which one can stop it (remove it) and resume it (re add it). Also provices access to the watchers itself and the frequency by which it's checked
  - Change the default frequency to 30 seconds interval (used to be 60 seconds). The only watcher that is currently effected by this is the script watcher (now auto-loading scripts will auto-load every 30 seconds if changed)
[Bulk] Add blocking close method to BulkProcessor

Blocks until all bulk requests have completed.

Closes #4158
Closes #6314
[Bulk] Release semaphore again after close

The bulk processor tries to acquire all leases for the semaphore to wait
for all pending requests. Yet, we should release them afterwards again to
ensure we don't ever deadlock if there is a bug in the processor.
This commit also adds a testcase for this method
Return null info if service is not started

Closes #6906
[DOCS] Fixed typos
Term Vectors API: Computes term vectors on the fly if not stored in the index.

Adds the ability to the Term Vector API to generate term vectors for some
chosen fields, even though they haven't been explicitely stored in the index.

Relates to #5184
Closes #6567
[CLIENT] Unknown node version should be a lower bound

Today when we start a `TransportClient` we use the given transport
addresses and create a `DiscoveryNode` from it without knowing the
actual nodes version. We just use the `Version.CURRENT` which is an
upper bound. Yet, the other node might be a version less than the
currently running and serialisation of the nodes info might break. We
should rather use a lower bound here which is the version of the first
release with the same major version as `Version.CURRENT` since this is
what we officially support.

This commit moves to use the minimum major version or an RC / Snapshot
if the current version is a snapshot.

Closes #6894
[TEST] Randomize FS Translog implementation

Closes #6658
[TEST] Randomize Preference in search requests

Closes #6863
[DOCS]: Added link to saltstack module
Better message for invalid internal transport message format
closes ##6916
Docs: Add links to client helper classes for bulk/scroll/reindexing
Plugin Manager: Properly quote $JAVA in bin/plugin

The all new Oracle Java 7 on OSX has a space in its path
resulting in bin/plugin failing to work

Closes #5765
CORS: Support regular expressions for origin to match against

This commit adds regular expression support for the allow-origin
header depending on the value of the request `Origin` header.

Relates #5601
Closes #6891