Simplify MoveAdjacentTo logic.

Extracted from #4469, which I plan on reopening after the feature freeze is over. Fixes #5445.