shorten: Extend fixed_coeffs to properly support pred_order 0

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
build: Require yasm 0.8.0 or higher

This is in preparation for removing a .rodata kludge
which was only required for older YASM versions.

The movbe instruction was introduced in 0.8.0, which already
had AVX, which was introduced in 0.7.0, and NASM introduced
movbe in 2.0.3, which is the same version which introduced
AVX support.

Also, make the failure message more accurate.

Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
x86inc: Remove .rodata kludges

The Mach-O bug was fixed in yasm 0.8.0 and we don't
support versions that old anymore.

Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
vmnc: K&R formatting cosmetics

Signed-off-by: Diego Biurrun <diego@biurrun.de>
vmnc: Port to bytestream2

Fix some buffer overreads.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
vmnc: Check the cursor dimensions

And manage the reallocation failure path.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
vmnc: Use meaningful return values
jpeg2000: Check block length

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org