avformat/swfdec: Check frame size rectangle in probe()

fixes probetest failure

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>