Gateway: Prefer recovering the state file that uses the latest format.

Currently MetaDataStateFormat loads the first available state file that has
the latest version. In case several files are available and some of them use
the new format while other ones use the legacy format, it should also prefer
the new format. This is typically useful when we upgrade the metadata when
recovering from the gateway: we might write the upgraded state with the new
format while the previous state used the legacy format, so we end up with
two files having the same version but using different formats.

Close #8343
Tests: Fix GroovyScriptTests to not depend on the way documents are routed to shards.