Simplify MoveAdjacentTo logic. Fixes #5445.