[RECOVERY] Allow to cancle recovery sources when shards are closed

Today recovery sources are not cancled if a shard is closed. The recovery target
is already cancled when shards are closed but we should also cleanup and cancel
the sources side since it holds on to shard locks / references until it's closed.
We should catch any exceptions during the cancel and log them so we can continue to cancel any other handlers? Otherwise the first exception will cause us to bail