swscale: set dither to a specific value for rgb/bgr8 output

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
swscale: improve dither checks

Bssed on patch by Øyvind Kolås <pippin@gimp.org>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>