hevc: check that VPS referenced from SPS exists

This matches how its done for SPS/PPS.
Fixes null pointer dereference.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
hevc: Fix modulo operations

Keep qp fields within the range.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
hevc: Use uint64 to check for tile dimensions

And use unsigned datatypes.

Otherwise it would overflow.

Sample-Id: 00001315-google
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
hevc: clip pixels when transquant bypass is used

The reconstructed picture should always be clipped (see section 8.6.5),
previously we did not clip coding units where
cu_transquant_bypass_flag == 1.

Sample-Id: 00001325-google
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
hevc: Remove useless clip

The src buffer should only contain values in the interval
[0, (1 << BIT_DEPTH) - 1].

Since shift = (BIT_DEPTH - 5), src[x] >> shift must be in
the interval [0, 31], so no clip is needed.

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
hevc: Prevent some integer overflows

get_ue_golomb_long() returns an unsigned.

Sample-Id: 00001541-google
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
hevc: Bound check slice_qp

The T-REC-H.265-2013044 page 79 states it has to be in the range
[-s->sps->qp_bd_offset, 51].

Sample-Id: 00001386-google
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
hevc: Bound check cu_qp_delta

The T-REC-H.265-2013044 page 91 states it has to be in the range
[-(26 + s->sps->qp_bd_offset / 2), (25 + s->sps->qp_bd_offset / 2)].
dxva2: Retry IDirectXVideoDecoder_BeginFrame()

If the function returns E_PENDING retry for a fixed number of times.

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
dxva2: Log errors verbosely

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Merge commit 'a7a07cc98ac548297b5b0628cb81280e11952e3f'

* commit 'a7a07cc98ac548297b5b0628cb81280e11952e3f':
  hevc: check that VPS referenced from SPS exists

Conflicts:
	libavcodec/hevc_ps.c

See: d66bab0a69ac1860e78dd951ad8db1a507e75642
Merged-by: Michael Niedermayer <michaelni@gmx.at>
Merge commit 'd5c15ebeaf1914ea5e3e0599d4316d7c4cf74434'

* commit 'd5c15ebeaf1914ea5e3e0599d4316d7c4cf74434':
  hevc: Fix modulo operations

Conflicts:
	libavcodec/hevc.c
	libavcodec/hevc_filter.c

See: 4ced5d7780fea2ea49444d6686d26f26b3a2160f
Merged-by: Michael Niedermayer <michaelni@gmx.at>
Merge commit 'b37e796082b2d787aff3cd5631bb89c4fd374708'

* commit 'b37e796082b2d787aff3cd5631bb89c4fd374708':
  hevc: Use uint64 to check for tile dimensions

Merged-by: Michael Niedermayer <michaelni@gmx.at>
Merge commit 'a246d06fe0dc6c2ea65e95327624b4537ff9bd0d'

* commit 'a246d06fe0dc6c2ea65e95327624b4537ff9bd0d':
  hevc: clip pixels when transquant bypass is used

See: c9fe0caf7a1abde7ca0b1a359f551103064867b1
Merged-by: Michael Niedermayer <michaelni@gmx.at>
Merge commit 'faf03ecba03155bb1f5416713bd01da043863b43'

* commit 'faf03ecba03155bb1f5416713bd01da043863b43':
  hevc: Remove useless clip

See: b00a8b4d194f1bf23343f3f42138affa1fe26641
Merged-by: Michael Niedermayer <michaelni@gmx.at>
Merge commit '838740e6420538ad45982da6b1d3aa3ae91307f5'

* commit '838740e6420538ad45982da6b1d3aa3ae91307f5':
  hevc: Prevent some integer overflows

Merged-by: Michael Niedermayer <michaelni@gmx.at>
Merge commit '0d999333f96a34903448579bf13a3209deaee9da'

* commit '0d999333f96a34903448579bf13a3209deaee9da':
  hevc: Bound check slice_qp

Conflicts:
	libavcodec/hevc.c

See: aead772b5814142b0e530804486ff7970ecd9eef
Merged-by: Michael Niedermayer <michaelni@gmx.at>
Merge commit 'e22ebd04bcab7f86548794556c28ecca46d9c2ac'

* commit 'e22ebd04bcab7f86548794556c28ecca46d9c2ac':
  hevc: Bound check cu_qp_delta

Conflicts:
	libavcodec/hevc.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
Merge commit '5b2b23f2d69e05c5fcd1c933e383fe60e185574d'

* commit '5b2b23f2d69e05c5fcd1c933e383fe60e185574d':
  dxva2: Retry IDirectXVideoDecoder_BeginFrame()

This also adds a missig #include to prevent breaking building

Merged-by: Michael Niedermayer <michaelni@gmx.at>
Merge remote-tracking branch 'qatar/master'

* qatar/master:
  dxva2: Log errors verbosely

Merged-by: Michael Niedermayer <michaelni@gmx.at>