vmd: use the PALETTE_COUNT constant uniformly

While at it drop useless parentheses.
vmd: return meaningful errors

CC: libav-stable@libav.org
vmd: refactor the inner decode loop

Simplify a little, assume empty frames are acceptable and
do not pointlessly reinit the bytestream2 contexts using
possibly wrong size values.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org