avformat/avidec: optimize probe

about 2x as fast

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
avformat/aea: reduce false positives in probing

Fixes probetest failure

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>