vmdav: Try to fix unpack_rle()

This fixes out of array accesses
The code prior to this commit could not have worked, thus obviously
was untested. I was also not able to find a valid sample that uses this
code.
This fix is thus only based on the description of the format

If someone has a sample that uses unpack_rle(), please mail me.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>