Search on a shard group while relocation final flip happens might fail

At the final stage of a relocation, during the final flip of the states, a search request might hit a node that would then execute it on a shard that has already relocated.

For this, we need to execute broadcast and search operations against initializing shards as well, but only as a last resort. The operation will be rejected if not applicable (i.e. IndexShard#searcher() checked for read allowed).

Note, this requires careful though about which failures we send back. If we try and initializing shard and it fails, its failure should not override an actual failure of an active shard.

Also, removed an atomic integer used in broadcast request and use a similar shard index trick we now have in our search execution.

closes #3427
improve test to wait for nodes before getting the local node id
less agreesive timeout to catch it on the pending check
Search on a shard group while relocation final flip happens might fail

make sure relocation shards add their corresponding initializing shard routing when search across initializing shards

also, make shardFailures lazy again

closes #3427
Allow bin/plugin to set -D JVM parameters

Currently the bin/plugin command did not allow one to set jvm parameters
for startup. Usually this parameters are not needed (no need to configure
heap sizes for such a short running process), but one could not set the
configuration path. And that one is important for plugins in order find
out, where the plugin directory is.

This is especially problematic when elasticsearch is installed as
debian/rpm package, because the configuration file is not placed in the
same directory structure the plugin shell script is put.

This pull request allows to call bin/plugin like this

bin/plugin -Des.default.config=/etc/elasticsearch/elasticsearch.yml -install mobz/elasticsearch-head

As a last small improvement, the PluginManager now outputs the directort
the plugin was installed to in order to avoid confusion.

Closes #3304
Search on a shard group while relocation final flip happens might fail

single shard read operations should have the same override exception logic as search and broadcast

relates to #3427