vmnc: K&R formatting cosmetics

Signed-off-by: Diego Biurrun <diego@biurrun.de>
vmnc: Port to bytestream2

Fix some buffer overreads.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
vmnc: Check the cursor dimensions

And manage the reallocation failure path.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
vmnc: Use meaningful return values
jpeg2000: Check block length

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org