[Engine] add a best effort waiting for ongoing recoveries to cancel on close

Currently one can close the engine while there are still on going recoveries. This is not a problem because the engine is close in tandem with the shard it belongs to, which in turn cancels the recoveries. This does cause some issues in our tests as we check the no resources were left behind after an index was deleted, which trips if the recoveries are not canceled.
the 30s should be reduced to 10s and be configurable. This may be called during cluster state publishing which times out after 30s by default.