Snapshot/Restore: Make sure indices cannot be renamed into restored aliases

Fixes #7915
Aggregations: makes script params consistent with other APIs in scripted_metric

This change removes the script_type parameter form the Scripted Metric Aggregation and adds support for _file and _id suffixes to the init_script, map_script, combine_script and reduce_script parameters to make defining the source of the script consistent with the other APIs which use the ScriptService
Search: add `format` support for date range filter and queries

When the date format is defined in mapping, you can not use another format when querying using range date query or filter.

For example, this won't work:

```
DELETE /test

PUT /test/t/1
{
  "date": "2014-01-01"
}

GET /test/_search
{
  "query": {
    "filtered": {
      "filter": {
        "range": {
          "date": {
            "from": "01/01/2014"
          }
        }
      }
    }
  }
}
```

It causes:

```
Caused by: org.elasticsearch.ElasticsearchParseException: failed to parse date field [01/01/2014], tried both date format [dateOptionalTime], and timestamp number
```

It could be nice if we can support at query time another date format just like we support `analyzer` at search time on String fields.

Something like:

```
GET /test/_search
{
  "query": {
    "filtered": {
      "filter": {
        "range": {
          "date": {
            "from": "01/01/2014",
            "format": "dd/MM/yyyy"
          }
        }
      }
    }
  }
}
```

Same for queries:

```
GET /test/_search
{
  "query": {
    "range": {
      "date": {
        "from": "01/01/2014",
        "format": "dd/MM/yyyy"
      }
    }
  }
}
```

Closes #7189.
Discovery: remove MasterFaultDetection.Listener.notListedOnMaster

It is never used in practice. We treat it as a master failure (using NodeDoesNotExistOnMasterException).

Closes #7995
Core: Perform write consistency just before writing on the primary shard

Before this change the write consistency change was performed on the node that receives the write request and the node that holds the primary shard. This change removes the check on the node that receives the request, since it is redundant.

Also this change moves the write consistency check on the node that holds the primary shard to a later moment after forking of the thread to perform the actual write on the primary shard.

Closes #7873
Snapshot/Restore: make it possible to delete snapshots with missing metadata file

Fixes #7980
[TEST] Fix config directory for external nodes in bwc tests

The logging configuration was expected in the path.home folder which is set to
target/JX
when running the bwc tests from the console.
Therefore the logger could not be initialized with error message:

[INFO] Failed to configure logging...
org.elasticsearch.ElasticsearchException: Failed to load logging configuration
        at org.elasticsearch.common.logging.log4j.LogConfigurator.resolveConfig(LogConfigurator.java:117)
        at org.elasticsearch.common.logging.log4j.LogConfigurator.configure(LogConfigurator.java:81)
        at org.elasticsearch.bootstrap.Bootstrap.setupLogging(Bootstrap.java:96)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:180)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
Caused by: java.nio.file.NoSuchFileException: /home/britta/es/target/J0/config
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
        at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
        at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:97)
        at java.nio.file.Files.readAttributes(Files.java:1686)
        at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:109)
        at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:69)
        at java.nio.file.Files.walkFileTree(Files.java:2602)
        at org.elasticsearch.common.logging.log4j.LogConfigurator.resolveConfig(LogConfigurator.java:107)
        ... 4 more
log4j:WARN No appenders could be found for logger (node).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

Setting the config directory fixes this.

Logs from external nodes are still not printed properly. They are inserted to the log
whenever the stdout is printed ([WARNING] JVM J0: stdout was not empty...)

closes #7964
[TEST] fix CurrentTestFailedMarker to reset its state after each test

The currently used method `testRunStarted` is only called before any tests have been run, we need to reset that state before each test, that's why we need to use `testStarted`.
plugins: disable support for config dir

When removing and installing again the plugin all configuration files will be removed in `config/pluginname` dir.
This is bad as users may have set and added specific configuration files.

During an install, if we detect already existing files in `config/pluginname` directory, we simply copy the new file to the same dir but we append a `.new` at the end.

Related to  #5064.

(cherry picked from commit 5da028f)
(cherry picked from commit 4cb1f95)
[SNAPSHOT] Add repository validation

Fixes #7096
Add Upgrade API

This commit does the following:
* Add the new API at the rest layer, being backed by the optimize API
  with upgrade flag, and segments api to find upgrade status.
* Add `upgrade` flag to optimize API, and deprecate `force` flag (will
  remove in master)
* Add test for both synchronous and async upgrade

closes #7884
closes #7922
Remove deprecations from master (follow up to #7922)
Fix missed places referencing optimize force flag from removal in commit 1ae8195.
Docs: fix formatting in snapshots documentation
[TEST] Fix content comparison to not take platform dependent control chars into account.
Scripting: Created a parameter parse to standardised script options
Tests: Enable check index in tests again.

* Run flush in beforeIndexShardClosed to prevent an empty shard.
* Only run check index if the shard state before closing was: started, relocated or post_recovery
Admin: add total index memory in `_cat/indices`

This patch adds to `_cat/indices` information about memory usage per index by adding memory used by FieldData, IdCache, Percolate, Segments (memory, index writer, version map).

```
% curl 'localhost:9200/_cat/indices?v&h=i,tm'
i     tm
wiki  8.1gb
test  30.5kb
user  1.9mb
```

Closes #7008
Docs: clarify wait_for_completion flag in snapshot documentation

Closes #7952
Tests: two corruptions can fix a file

In some of our tests we corrupt multiple places in a file randomly. If we corrupt the same place twice, we shouldn't fix the file by mistake.