[TEST] Fix content comparison to not take platform dependent control chars into account.
Scripting: Created a parameter parse to standardised script options
Tests: Enable check index in tests again.

* Run flush in beforeIndexShardClosed to prevent an empty shard.
* Only run check index if the shard state before closing was: started, relocated or post_recovery
Admin: add total index memory in `_cat/indices`

This patch adds to `_cat/indices` information about memory usage per index by adding memory used by FieldData, IdCache, Percolate, Segments (memory, index writer, version map).

```
% curl 'localhost:9200/_cat/indices?v&h=i,tm'
i     tm
wiki  8.1gb
test  30.5kb
user  1.9mb
```

Closes #7008
Docs: clarify wait_for_completion flag in snapshot documentation

Closes #7952
Tests: two corruptions can fix a file

In some of our tests we corrupt multiple places in a file randomly. If we corrupt the same place twice, we shouldn't fix the file by mistake.
Docs: Added breaking changes docs for Indices APIs

Adds the breaking changes defaults for the change of default indices options for the GET Aliases API
Parent/child: has_parent filter must take parent filter into account when executing the inner query/filter.

Closes #8020
Closes #7943
[TEST] fix test after pr #8020
Internal: split shard search request into a local and a transport variant

In some cases a shard search request gets created on a node to be only used there and never sent over the transport. This commit clarifies that and creates a new base class called `ShardSearchLocalRequest` that can and will be only used locally. `ShardSearchTransportRequest` on the other hand delegates to the local version but extends `TransportRequest` and is `Streamable`, which means that it is supposed to be sent over the transport.

This way we can make the `OriginalIndices` only required (and mandatory now) in the transport variant.

Took the chance to remove an unused InternalScrollSearchRequest constructor and an empty else branch in `TransportSearchScrollQueryAndFetchAction`.

Closes #7855
Core: Make fetch phase nested doc aware

By letting the fetch phase understand the nested docs structure we can serve nested docs as hits.
The `top_hits` aggregation can because of this commit be placed in a `nested` or `reverse_nested` aggregation.

Closes #7164
Tests: add verbosity to this test case
add verbose logging during index close
MLT Query: don't fetch unnecessary stats
Fix serialization of short[] arays.

short[] were mistakenly encoded as a float[]. This is not an issue for the
text-based xcontents that we have (yaml, json) since floats can represent any
short value and are serialized as strings. However, this will make the binary
xcontents serialize shorts as int instead of floats.

Close #7845
Tests: dump all thread stacks on failure
[SPEC] Separated the "Upgrade Index" API into two methods

* `get_upgrade` => `GET _upgrade`  -- Return the status
* `upgrade`     => `POST _upgrade` -- Perform the operation

Original specification part of c021f22523a69583b94c56921792b727ab4030fb.

Related: #7884, #7922
Tests: move thread stacks on failure to a RunListener, so it actually works
Aggregations: Fixes scripted metrics aggregation when used as a sub aggregation

The scripted metric aggregation is now a PER_BUCKET aggregation so that parent buckets are evaluated independently. Also the params and reduceParams are copied for each instance of the aggregator (each parent bucket) so modifications to the values are kept only within the scope of its parent bucket

Closes #8036
Docs: Remove the note that fielddata doesn't support filtering.

This particular note was about fielddata filtering but could cause confusion
that fields that have doc values enabled cannot be used for filtering (as in
a `filtered_query`).