avcodec/roqvideoenc: fix infinite lambda increasing loop

The threshold was choosen so that no further size decrease happened with larger lambda
with the test video.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>