Backport `ShardRecoveryHandler`refactorings to 1.x

this is a backport of several commits and since it's a biggy I think we should do some review cycles here.
Should this await be cancelable?