avcodec/mace: check buffer size

Fixes use of uninitialized memory
Fixes: msan_uninit-mem_7f9d2775e090_8782_surge-2-8-MAC3.mov
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>