[Discovery] lightweight minimum master node recovery
don't perform full recovery when minimum master nodes are not met, keep the state around and use it once elected as master
[Internal] make no master lock an instance var so it can be configured
[Discovery] add rejoin on master gone flag, defaults to false

defaults to false since there is still work left to properly make it work
[Discovery] Make noMasterBlock configurable and added simple test that shows reads do execute (partially) when m_m_n isn't met
[Discovery] Enable `discovery.zen.rejoin_on_master_gone` setting in DiscoveryWithNetworkFailuresTests only.
[Discovery] Changed the default for the 'rejoin_on_master_gone' option from false to true in zen discovery.

Added AwaitFix for the FullRollingRestartTests.
[Discovery] If available newly elected master node should take over previous known nodes.
[Discovery] Eagerly clean the routing table of shards that exist on nodes that are not in the latestDiscoNodes list.

Only the previous master node has been removed, so only shards allocated to that node will get failed.
This would have happened anyhow on later on when AllocationService#reroute is invoked (for example when a cluster setting changes or another cluster event),
but by cleaning the routing table pro-actively, the stale routing table is fixed sooner and therefor the shards
that are not accessible anyhow (because the node these shards were on has left the cluster) will get re-assigned sooner.
Updated to use ClusterBlocks new constructor signature

Introduced with: 11a3201a092ed6c5d31516ae4b30dbb618ba348c
[Internal] Do not execute cluster state changes if current node is no longer master

When a node steps down from being a master (because, for example, min_master_node is breached), it may still have
cluster state update tasks queued up. Most (but not all) are tasks that should no longer be executed as the node
no longer has authority to do so. Other cluster states updates, like electing the current node as master, should be
executed even if the current node is no longer master.

This commit make sure that, by default, `ClusterStateUpdateTask` is not executed if the node is no longer master. Tasks
that should run on non masters are changed to implement a new interface called `ClusterStateNonMasterUpdateTask`

Closes #6230
[TEST] It may take a little bit before the unlucky node deals with the fact the master left
[TEST] Added test that verifies data integrity during and after a simulated network split.
[TEST] Make sure there no initializing shards when network partition is simulated
[TEST] Added test that exposes a shard consistency problem when isolated node(s) rejoin the cluster after network segmentation and when the elected master node ended up on the lesser side of the network segmentation.
[Discovery] Removed METADATA block
[Discovery] Made 'discovery.zen.rejoin_on_master_gone' setting updatable at runtime.
[Discovery] do not use versions to optimize cluster state copying for a first update from a new master

We have an optimization which compares routing/meta data version of cluster states and tries to reuse the current object if the versions are equal. This can cause rare failures during recovery from a minimum_master_node breach when using the "new light rejoin" mechanism and simulated network disconnects. This happens where the current master updates it's state, doesn't manage to broadcast it to other nodes due to the disconnect and then steps down. The new master will start with a previous version and continue to update it. When the old master rejoins, the versions of it's state can equal but the content is different.

Also improved DiscoveryWithNetworkFailuresTests to simulate this failure (and other improvements)

Closes #6466
[TEST] Remove 'index.routing.allocation.total_shards_per_node' setting in data consistency test
[Test]  testIsolateMasterAndVerifyClusterStateConsensus didn't wait on initializing shards before comparing cluster states
[Discovery] Change (Master|Nodes)FaultDetection's connect_on_network_disconnect default to false

The previous default was true, which means that after a node disconnected event we try to connect to it as an extra validation. This can result in slow detection of network partitions if the extra reconnect times out before failure.

Also added tests to verify the settings' behaviour