Ensure we only set the color when we need to