quantize luminance in cartoon filter