[TEST] use CFS consistently to not trigger single segment merge without force flag
Internal: made original indices optional for broadcast delete and delete by query shard requests

Shard requests like broadcast delete and delete by query, that needs to be executed on primary and all replicas, get read and written out to the transport on the same node. That means that if we add some field version checks are not enough to maintain bw comp since a newer node that holds the primary might receive the request from an older node, that didn't provide the field. Yet, when writing the request out again to a newer node that holds the replica, we do try and serialize the field although it's missing. The newer fields just needs to be set to optional in these cases, in addition to the version checks.

Re-enabled testDeleteByQuery and testDeleteRoutingRequired bw comp tests since this was the cause of their failures.

Closes #7406
[Tests] NoMasterNodeTests make timeout checks less sensitive

Also remove catching of MasterNotDiscoveredException in bulk operation it is only set on a per item basis
lightweight minimum master node recovery
don't perform full recovery when minimum master nodes are not met, keep the state around and use it once elected as master
make no master lock an instance var so it can be configured
add rejoin on master gone flag, defaults to false
defaults to false since there is still work left to properly make it work
Make noMasterBlock configurable and added simple test that shows reads do execute (partially) when m_m_n isn't met
Enable `discovery.zen.rejoin_on_master_gone` setting in DiscoveryWithNetworkFailuresTests only.
Changed the default for the 'rejoin_on_master_gone' option from false to true in zen discovery.
Added AwaitFix for the FullRollingRestartTests.
If available newly elected master node should take over previous known nodes.
Fixed compile error
Do not start a recovery process if the primary shard is currently allocated on a node which is not part of the cluster state

If a source node disconnect during recover, the target node will respond by canceling the recovery. Typically the master will respond by removing the disconnected node from the cluster state, promoting another shard to become primary. This is sent it to all nodes and the target node will start recovering from the new primary. However, if the drop of a node caused the node count to go bellow min_master_node, the master will step down and will not promote shard immediately. When a new master is elected we may publish a new cluster state (who's point is to notify of a new master) which is not yet updated. This caused the node to start a recovery to a non existent node. Before we aborted the recovery without cleaning up the shard, causing subsequent correct cluster states to be ignored. We should not start the recovery process but wait for another cluster state to come in.
RoutingService background reroute task should check if node is still master when its run

It may be that between the moment the task was added to the queue the current node stepped down from being master (due to minimum_master_nodes settings).
Eagerly clean the routing table of shards that exist on nodes that are not in the latestDiscoNodes list.

Only the previous master node has been removed, so only shards allocated to that node will get failed.
This would have happened anyhow on later on when AllocationService#reroute is invoked (for example when a cluster setting changes or another cluster event),
but by cleaning the routing table pro-actively, the stale routing table is fixed sooner and therefor the shards
that are not accessible anyhow (because the node these shards were on has left the cluster) will get re-assigned sooner.
Use just AllocationService#reroute to eagerly clean the routing table of nodes that don't exist instead of using AllocationService#applyFailedShards directly.
Updated to use ClusterBlocks new constructor signature

Introduced with: 11a3201a092ed6c5d31516ae4b30dbb618ba348c
Do not execute cluster state changes if current node is no longer master

When a node steps down from being a master (because, for example, min_master_node is breached), it may still have
cluster state update tasks queued up. Most (but not all) are tasks that should no longer be executed as the node
no longer has authority to do so. Other cluster states updates, like electing the current node as master, should be
executed even if the current node is no longer master.

This commit make sure that, by default, `ClusterStateUpdateTask` is not executed if the node is no longer master. Tasks
that should run on non masters are changed to implement a new interface called `ClusterStateNonMasterUpdateTask`

Closes #6230
[TEST] It may take a little bit before the unlucky node deals with the fact the master left
[TEST] Use prettyprint
[TEST] Added test that verifies data integrity during and after a simulated network split.