avformat/mxfdec: detect loops during header parsing

The header parser uses forward and backward parsing, making the
bulletproof prevention of loops difficult, thus this simple
detection code.
If someone improves the forward/backward parsing so it cannot loop
then this commit should be reverted

Fixes Ticket3278

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 1c010fd035c1a14dc73827b84f21f593e969a5d6)
Use the h264 parser when decoding VSSH in avi.

Fixes ticket #3261 visually.

Analyzed-by: Michael Doilnitsyn
(cherry picked from commit 94cf4f8bac12c58e30ce3b5d72cf5898baafe9a8)
avcodec/msvideo1enc: fix SKIPS_MAX

Fixes Ticket3270

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit fb8f5d0510619cea2204246631f1c0dcd994ee25)
Fix libopenjpeg colour range adjust for 8<bpp<16.

Fixes ticket #3284.

Reviewed-by: Michael Bradshaw
(cherry picked from commit 8298b54179c92fc3293ea312c4fcf153917bca0a)
vdpau: restore compatibility with deprecated fields in AVVDPAUContext

Fixes ticket #3133.

Signed-off-by: Hendrik Leppkes <h.leppkes@gmail.com>
Tested-by: EricV
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 23bc1351ad7db698da9348e23ee63ec7300a881d)
avcodec/h264: fix code that blindly dereferences NULL DPB

Fixes mixed flushing and decoding NULL packets
Found-by: wm4

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit d9339ab55373b12f078a3e3f1e294d8ff78652dd)

Conflicts:

	libavcodec/h264.c
avformat/rmdec: when reading audio blocks, dont leave holes when reading fails

The fate test is changed because the reference file depends on the use of
non cleared data at the very
end. Alternatively we could upload a new reference file, though that would
then have to be changed every time the handling of a truncated frame changes
or theres a change to error concealment, each time adding a new file ...

Fixes use of uninitialized memory
Fixed: msan_uninit-mem_7f3c02b81363_2787_RLG2_19.rm
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 77d2a1ca595ebe082d35c4b624ac9a9145991494)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
avformat/mxfdec: check avio_read(UID) result

Fixes use of uninitialized memory
Fixes: msan_uninit-mem_7fc9ba2fd98e_82_02785736.mxf
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 4162ceea93684f3cd656dc21d30903e102a44e73)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
avformat/flvdec: initialize context before reading from it

Fixes use of uninitialized memory
Fixes: msan_uninit-mem_7f9b8387069e_5377_flv_with_pcm_s16be_audio_track.flv
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 396ddcf22d55fa7e735d69eed22a4a4b1649b73c)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
avformat/matroskadec: check generic audio deinterleaver sub_packet_size against frame_size

Fixes use of uninitialized memory
Fixes: msan_uninit-mem_7f67d052a530_7517_nosound.mkv
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit a1ed1c2193483849df689b105bec0d26c2497999)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
avcodec/wmalosslessdec: Pass on error code from decode_tilehdr()

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 03fff09b32171e0c76d104c02ebf578c7f4fe21d)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
avcodec/wmalosslessdec: deallocate uninitialized frame on decode_tilehdr() failure

Fixes use of uninitialized memory
partly fixes: msan_uninit-mem_7f7834b6a530_6473_luckynight-partial.wma
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit ae3856dcaf9c5ef339969c95a72bcaf7c4bba9ec)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
avcodec/wmalosslessdec: shrink output on error so no uninitialized data is returned

Fixes use of uninitialized memory
partly fixes: msan_uninit-mem_7f7834b6a530_6473_luckynight-partial.wma
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 6b18a6839b43ea78e70cd3e35f781d1c955bda73)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
avformat/pjsdec: dont increase pointer when its already at the end in read_ts()

Fixes use of uninitialized memory
Fixes: msan_uninit-mem_7f91f2de7764_2649_PJS_capability_tester.pjs
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit b84a7330af41cec93384bf59ed68c67b09d105cd)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
avcodec/alac: only set *got_frame_ptr when all channels have been decoded

Fixes use of uninitialized memory
Fixes: msan_uninit-mem_7f8b64436530_7895_quicktime_newcodec_applelosslessaudiocodec.m4a
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit e11983bda073f8c63f60509ee753da9fba20ed10)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
avcodec/hevc: use av_mallocz() for allocating tab_ipm

Fixes use of uninitialized memory and out of stack array read
Fixes: signal_sigsegv_ecc526_7846_WPP_C_ericsson_MAIN_2.bit
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 0999f1613bc48ed9d6578a3ad7bcd17610e07fbf)

Conflicts:

	libavcodec/hevc.c
hevc: Clip the pixel before shifting

Prevent an out of array bound read.

Fixes: asan_stack-oob_eae8e3_7333_WPP_B_ericsson_MAIN10_2.bit
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 5856bca360c5bc3e340a357d91b1f993c80a7bea)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
hevc: clip pixels when transquant bypass is used

Fixes: asan_stack-oob_eae8e3_7333_WPP_B_ericsson_MAIN10_2.bit
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind

This is a more proper fix than 5856bca360c5bc3e340a357d91b1f993c80a7bea

The reconstructed picture should always be clipped (see section 8.6.5),
previously we did not clip coding units where
cu_transquant_bypass_flag == 1

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit c9fe0caf7a1abde7ca0b1a359f551103064867b1)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
hevc: remove useless clip in FUNC(sao_band_filter)()

The src buffer should only contain values in the interval
[0, (1 << BIT_DEPTH) - 1]. Since shift = (BIT_DEPTH - 5), src[x] >> shift
must be in the interval [0, 31], so no clip is needed.

This removes the code that was changed in 5856bca360c5bc3e340a357d91b1f993c80a7bea
as the clip that was repositioned in that commit is removed

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit b00a8b4d194f1bf23343f3f42138affa1fe26641)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
hevc: Reject impossible dependent tile

The tile 0 cannot depend on a previous one.
Prevent an out of array bound load in ff_hevc_cabac_init().

Fixes: asan_heap-oob_e3a924_1630_DBLK_A_MAIN10_VIXS_2.bit
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind

Reviewed-by: Guillaume Martres <smarter@ubuntu.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 48a5b155433ed7af20fb0a5c20ca131958727727)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>