x86inc: activate REP_RET automatically

Now RET checks whether it immediately follows a branch, so the
programmer dosen't have to keep track of that condition. REP_RET
is still needed manually when it's a branch target, but that's
much rarer.

The implementation involves lots of spurious labels, but that's OK
because we strip them.

Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
build: Strip spurious labels

The implementation of 25cb0c1a involves lots of spurious labels.

The effect of keeping those labels around is making debugging harder.
Those labels are meaningless, and complicate the disassembly. Also,
gdb can't tell the difference between them and function entry points.

This new strip command is irrelevant to any usage of Libav that would
have used the old fully stripped version, because the old one was for
non-debug use.

Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
x86inc: Set ELF hidden visibility for global constants

Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
x86inc: Use SSE instead of SSE2 for copying data

Reduces code size because movaps/movups is one byte
shorter than movdqa/movdqu.

Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
x86inc: fix some corner cases of SWAP

SWAP with >=3 named (rather than numbered) args
PERMUTE followed by SWAP with 2 named args
used to produce the wrong permutation

Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
x86inc: create xm# and ym#, analagous to m#

For when we want to mix simd sizes within one function.

Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
x86inc: Utilize the shadow space on 64-bit Windows

Store XMM6 and XMM7 in the shadow space in functions that
clobbers them. This way we don't have to adjust the stack
pointer as often, reducing the number of instructions as
well as code size.

Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
x86inc: Check for __OUTPUT_FORMAT__ having a value of "x64"

This is also a valid value for WIN64.

Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
x86inc: various minor backports from x264

Small backports that sneaked into other asm commits in x264.

Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
x86inc: remove misaligned cpu flag

Prevents a crash if the misaligned exception mask bit is
cleared for some reason.

Misaligned SSE functions are only used on AMD Phenom CPUs
and the benefit is miniscule. They also require modifying
the MXCSR control register and by removing those functions
we can get rid of that complexity altogether.

Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
x86: h264_idct: Update comments to match 8/10-bit depth optimization split
shorten: Extend fixed_coeffs to properly support pred_order 0

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
build: Require yasm 0.8.0 or higher

This is in preparation for removing a .rodata kludge
which was only required for older YASM versions.

The movbe instruction was introduced in 0.8.0, which already
had AVX, which was introduced in 0.7.0, and NASM introduced
movbe in 2.0.3, which is the same version which introduced
AVX support.

Also, make the failure message more accurate.

Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
x86inc: Remove .rodata kludges

The Mach-O bug was fixed in yasm 0.8.0 and we don't
support versions that old anymore.

Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
vmnc: K&R formatting cosmetics

Signed-off-by: Diego Biurrun <diego@biurrun.de>
vmnc: Port to bytestream2

Fix some buffer overreads.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
vmnc: Check the cursor dimensions

And manage the reallocation failure path.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
vmnc: Use meaningful return values
jpeg2000: Check block length

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
avi: properly fail if the dv demuxer is missing

CC: libav-stable@libav.org