Ensure we never double count time changes.