Snapshot/Restore: make it possible to delete snapshots with missing metadata file

Fixes #7980
[TEST] Fix config directory for external nodes in bwc tests

The logging configuration was expected in the path.home folder which is set to
target/JX
when running the bwc tests from the console.
Therefore the logger could not be initialized with error message:

[INFO] Failed to configure logging...
org.elasticsearch.ElasticsearchException: Failed to load logging configuration
        at org.elasticsearch.common.logging.log4j.LogConfigurator.resolveConfig(LogConfigurator.java:117)
        at org.elasticsearch.common.logging.log4j.LogConfigurator.configure(LogConfigurator.java:81)
        at org.elasticsearch.bootstrap.Bootstrap.setupLogging(Bootstrap.java:96)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:180)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
Caused by: java.nio.file.NoSuchFileException: /home/britta/es/target/J0/config
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
        at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
        at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:97)
        at java.nio.file.Files.readAttributes(Files.java:1686)
        at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:109)
        at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:69)
        at java.nio.file.Files.walkFileTree(Files.java:2602)
        at org.elasticsearch.common.logging.log4j.LogConfigurator.resolveConfig(LogConfigurator.java:107)
        ... 4 more
log4j:WARN No appenders could be found for logger (node).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

Setting the config directory fixes this.

Logs from external nodes are still not printed properly. They are inserted to the log
whenever the stdout is printed ([WARNING] JVM J0: stdout was not empty...)

closes #7964
[TEST] fix CurrentTestFailedMarker to reset its state after each test

The currently used method `testRunStarted` is only called before any tests have been run, we need to reset that state before each test, that's why we need to use `testStarted`.
plugins: disable support for config dir

When removing and installing again the plugin all configuration files will be removed in `config/pluginname` dir.
This is bad as users may have set and added specific configuration files.

During an install, if we detect already existing files in `config/pluginname` directory, we simply copy the new file to the same dir but we append a `.new` at the end.

Related to  #5064.

(cherry picked from commit 5da028f)
[SNAPSHOT] Add repository validation

Fixes #7096
Add Upgrade API

This commit does the following:
* Add the new API at the rest layer, being backed by the optimize API
  with upgrade flag, and segments api to find upgrade status.
* Add `upgrade` flag to optimize API, and deprecate `force` flag (will
  remove in master)
* Add test for both synchronous and async upgrade

closes #7884
closes #7922
Remove test uses of optimize force flag.
Docs: fix formatting in snapshots documentation
[TEST] Fix content comparison to not take platform dependent control chars into account.
Docs: clarify wait_for_completion flag in snapshot documentation

Closes #7952
Parent/child: has_parent filter must take parent filter into account when executing the inner query/filter.

Closes #8020
Closes #7943
Test: Fixed test bug, expected string should take wrapped type filter into account
Internal: split shard search request into a local and a transport variant

In some cases a shard search request gets created on a node to be only used there and never sent over the transport. This commit clarifies that and creates a new base class called `ShardSearchLocalRequest` that can and will be only used locally. `ShardSearchTransportRequest` on the other hand delegates to the local version but extends `TransportRequest` and is `Streamable`, which means that it is supposed to be sent over the transport.

This way we can make the `OriginalIndices` only required (and mandatory now) in the transport variant.

Took the chance to remove an unused InternalScrollSearchRequest constructor and an empty else branch in `TransportSearchScrollQueryAndFetchAction`.

Closes #7855
add verbose logging during index close
[SPEC] Separated the "Upgrade Index" API into two methods

* `get_upgrade` => `GET _upgrade`  -- Return the status
* `upgrade`     => `POST _upgrade` -- Perform the operation

Original specification part of c021f22523a69583b94c56921792b727ab4030fb.

Related: #7884, #7922
[CORE] Log if CircuitBreaker is tripping

Today we only throw an exception which might not be logged at all. This
commit adds debug logging if we are tripping a CB.

Closes #8050
Spec: Added human flag to indices.get_upgrade
Add link to new upgrade docs from indices page.

See #7922.
Add forgotten include for upgrade docs.

See #7922
Internal: fixed bw comp in ShardSearchTransportRequest

Original indices were introduced in 1.4.0.Beta1 as optional, preceded by a boolean flag that told whether they were empty or not. We have to keep serializing as optional only when reading and writing from/to 1.4.0.Beta1, although the original indices are not optional whenever we go and serialize the request they belong to (which is why the boolean got removed in 1.4.0).