avcodec/roqvideoenc: check dimensions against maximum

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>