avformat/rtpdec_h261: Fix sanity checks

Fixes mbap and quant

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>