Prevent unnecessary color space conversions, regression in #4163