[TEST] fixed scriptedMetricTests

The tests were failing because there was a shard which didn't get any documents and the tests assumed all shards had documents. This commit fixes this assumption
[docs] fix typo in getting-started

Closes #8354
Discovery: a more lenient wait joinThread when stopping

When a node stops, we cancel any ongoing join process. With #8327, we improved this logic and wait for it to complete before shutting down the node. In our tests we typically shutdown an entire cluster at once, which makes it very likely for nodes to be joining while shutting down. This introduces a race condition where the joinThread.interrupt can happen before the thread starts waiting on pings which causes shutdown logic to be slow. This commits improves by repeatedly trying to stop the thread in smaller waits.

Another side effect of the change is that we are now more likely to ping ourselves while shutting down, we results in an ugly warn level log. We now log all remote exception during pings at a debug level.

Closes #8359
Revert "Discovery: a more lenient wait joinThread when stopping"

This reverts commit 83d9dab79882700fb21cb496282bafdb1d28a5ca.
Refactor shard recovery from anonymous class to ShardRecoveryHandler

Previously the bulk of our shard recovery code was in a 300-line
anonymous class in `RecoverySource`. This made it difficult to find and
more difficult to read.

This factors out that code into a `ShardRecoveryHandler` class, adding
javadocs for each function and phase of the recovery, as well as
comments explaining some of the more esoteric functions performed during
recovery.

It's hoped that this will help more people understand Elasticsearch's
recovery procedure.

No *major* functionality has changed, only typo corrections, some minor
allocation improvements and logging clarification changes.