hevc: check that VPS referenced from SPS exists

This matches how its done for SPS/PPS.
Fixes null pointer dereference.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
hevc: Fix modulo operations

Keep qp fields within the range.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
hevc: Use uint64 to check for tile dimensions

And use unsigned datatypes.

Otherwise it would overflow.

Sample-Id: 00001315-google
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
hevc: clip pixels when transquant bypass is used

The reconstructed picture should always be clipped (see section 8.6.5),
previously we did not clip coding units where
cu_transquant_bypass_flag == 1.

Sample-Id: 00001325-google
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
hevc: Remove useless clip

The src buffer should only contain values in the interval
[0, (1 << BIT_DEPTH) - 1].

Since shift = (BIT_DEPTH - 5), src[x] >> shift must be in
the interval [0, 31], so no clip is needed.

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
hevc: Prevent some integer overflows

get_ue_golomb_long() returns an unsigned.

Sample-Id: 00001541-google
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
hevc: Bound check slice_qp

The T-REC-H.265-2013044 page 79 states it has to be in the range
[-s->sps->qp_bd_offset, 51].

Sample-Id: 00001386-google
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
hevc: Bound check cu_qp_delta

The T-REC-H.265-2013044 page 91 states it has to be in the range
[-(26 + s->sps->qp_bd_offset / 2), (25 + s->sps->qp_bd_offset / 2)].