avformat/smacker: check for format mismatch more completely

Fixes use of uninitialized memory
Fixes: msan_uninit-mem_7f6e83322950_9769_wetlogo.smk
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>