Prevent unnecessary colorspace conversions (regression in #4163)