hevc: check that VPS referenced from SPS exists

This matches how its done for SPS/PPS.
Fixes null pointer dereference.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
hevc: Fix modulo operations

Keep qp fields within the range.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
hevc: Use uint64 to check for tile dimensions

And use unsigned datatypes.

Otherwise it would overflow.

Sample-Id: 00001315-google
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
hevc: clip pixels when transquant bypass is used

The reconstructed picture should always be clipped (see section 8.6.5),
previously we did not clip coding units where
cu_transquant_bypass_flag == 1.

Sample-Id: 00001325-google
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
hevc: Remove useless clip

The src buffer should only contain values in the interval
[0, (1 << BIT_DEPTH) - 1].

Since shift = (BIT_DEPTH - 5), src[x] >> shift must be in
the interval [0, 31], so no clip is needed.

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
hevc: Prevent some integer overflows

get_ue_golomb_long() returns an unsigned.

Sample-Id: 00001541-google
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
hevc: Bound check slice_qp

The T-REC-H.265-2013044 page 79 states it has to be in the range
[-s->sps->qp_bd_offset, 51].

Sample-Id: 00001386-google
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
hevc: Bound check cu_qp_delta

The T-REC-H.265-2013044 page 91 states it has to be in the range
[-(26 + s->sps->qp_bd_offset / 2), (25 + s->sps->qp_bd_offset / 2)].
dxva2: Retry IDirectXVideoDecoder_BeginFrame()

If the function returns E_PENDING retry for a fixed number of times.

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
dxva2: Log errors verbosely

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
dxva2: Add missing #includes
dxva2: Use correct printf format strings
dxtory: correctly handle YUV slices with average odd height
dxtory: change error code for unexpected slice configuration
lavc: do not force the emu edge flag

The default get_buffer2() implementation (and possibly some
user ones) does not allocate edges when this flag is set, which may
expose bugs in some decoders. Until the 10 release is out, it is safer
to remove this part.
swscale: fix stride used in planarToNv12Wrapper()

Signed-off-by: Anton Khirnov <anton@khirnov.net>
swscale: add nv12/nv21->yuv420 converter

Signed-off-by: Anton Khirnov <anton@khirnov.net>
swscale/x86/rgb2rgb: add support for AVX

This does not yet include any actual AVX code

Signed-off-by: Anton Khirnov <anton@khirnov.net>
rgb2rgb_template: add MMX/SSE2/AVX-optimized deinterleaveBytes

Signed-off-by: Anton Khirnov <anton@khirnov.net>
configure: Set default HOSTCFLAGS/HOSTCPPFLAGS after compiler detection

This prevents the default HOSTCFLAGS from getting clobbered by flags
passed to configure on the command line.