Only set it to not switch if the priorities aren't equal.