Reduced banding caused by dithering error clamping