improve visibility of clusterState and shardsIt
In case of retries, we update the clusterState and shardsIt, make sure they are visible using volatile (even though updates will probably go through a memory barrier, this might explain rare failure we see when retry happens)
improve logging
the source of the default mappings should be trace logged
Allow packages to disable immediate restart on package upgrade

Both package types, RPM and deb now contain an option to not restart on upgrade.

This option can be configure in /etc/default/elasticsearch for dpkg based systems
and /etc/sysconfig/elasticsearch for rpm based systems.

By default the setting is as before, where a restart is executed on upgrade.

Closes #3685
Log failures at closing the SearcherManager.
Add TESTING document describing test settings
Add integration test for regex field data filter
allow refresh to execute on a relocating shard as well
Flush API: remove refresh flag
Refresh flag in flush is problematic, since the shards refresh is allowed to execute on is different compared to the flush shards. In order to do flush and then refresh, they should be executed as separate APIs when needed.
closes #3689
Optimize API: Remove refresh flag
Refresh flag in optimize is problematic, since the shards refresh is allowed to execute on is different compared to the optimize shards. In order to do optimize and then refresh, they should be executed as separate APIs when needed.
closes #3690
use built in flush/refresh helper methods
better logging
trace log the low level details on recovery
tag with @AwaitsFix because it causes the test suite to crash

see #3694
Also, this might have caused one of the problems described in issue #3674
Added third highlighter type based on lucene postings highlighter

Requires field index_options set to "offsets" in order to store positions and offsets in the postings list.
Considerably faster than the plain highlighter since it doesn't require to reanalyze the text to be highlighted: the larger the documents the better the performance gain should be.
Requires less disk space than term_vectors, needed for the fast_vector_highlighter.
Breaks the text into sentences and highlights them. Uses a BreakIterator to find sentences in the text. Plays really well with natural text, not quite the same if the text contains html markup for instance.
Treats the document as the whole corpus, and scores individual sentences as if they were documents in this corpus, usingthe  BM25 algorithm.

Uses forked version of lucene postings highlighter to support:
- per value discrete highlighting for fields that have multiple values, needed when number_of_fragments=0 since we want to return a snippet per value
- manually passing in query terms to avoid calling extract terms multiple times, since we use a different highlighter instance per doc/field, but the query is always the same

The lucene postings highlighter api is  quite different compared to the existing highlighters api, the main difference being that it allows to highlight multiple fields in multiple docs with a single call, ensuring sequential IO.
The way it is introduced in elasticsearch in this first round is a compromise trying not to change the current highlight api, which works per document, per field. The main disadvantage is that we lose the sequential IO, but we can always refactor the highlight api to work with multiple documents.

Supports pre_tag, post_tag, number_of_fragments (0 highlights the whole field), require_field_match, order by score and and html encoding.