vmd: use the PALETTE_COUNT constant uniformly

While at it drop useless parentheses.
(cherry picked from commit 91a6944e56236234f0a7ba162404665753cbcb51)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
vmd: refactor the inner decode loop

Simplify a little, assume empty frames are acceptable and
do not pointlessly reinit the bytestream2 contexts using
possibly wrong size values.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 676da248cad49debc40720baa13214f0b94dcc71)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

Conflicts:
	libavcodec/vmdav.c