swscale/swscale: fix integer overflow

Should fix fate failure with clang ftrapv

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>