blending: fix clamping in blend mode coloradjustment

this should fix #9882