Clear worker references for get_shard_replacements

This function relies on record equality. If a request passes workers
that have references defined it would return all copies of the shard
range incorrectly.

BugzId: 28992