vc1dec: add flush function for WMV9 and VC-1 decoders

CC: libav-stable@libav.org
(cherry picked from commit 4dc8c8386eef942dba35c4f2fb3210e22b511a5b)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 02b72394627933dc8ce26445231a69f00dba491b)

Conflicts:
	libavcodec/vc1dec.c

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 0173a7966b331105158a88f96b9afcc431d2fef8)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit aa4121276777b20eaaa83bf9bd544b00748c865c)

Conflicts:
	libavcodec/vc1dec.c

Signed-off-by: Anton Khirnov <anton@khirnov.net>
nuv: check RTjpeg header for validity

CC: libav-stable@libav.org
(cherry picked from commit 859a579e9bbf47fae2e09494c43bcf813dcb2fad)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 6704522ca9dd32c858ee474492be568c386910f9)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit f31170d4e7f9671e019315391160d454b18d7296)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 459feb7cce03af7154c098171fc9d36fc9d472f6)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
imgconvert: avoid undefined left shift in avcodec_find_best_pix_fmt

CC: libav-stable@libav.org
(cherry picked from commit 39bb27bf79bc4c2d8beaed637a14176264cb1916)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 7a7229b52d1900279041991fadbd29b27e8dfe95)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 8812b5f164109553f009ce385e17a1af16b6ea53)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit fd7426ed898533bed98e6b472ff5f5c8e47f2eb5)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
mpegvideo: Don't use ff_mspel_motion() for vc1

Using ff_mspel_motion assumes that s (a MpegEncContext
poiinter) really is a Wmv2Context.

This fixes crashes in error resilience on vc1/wmv3 videos.

CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 18f2d5cb9c48d06895960f37467576725c9dc2d1)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit da0c457663479bc1828918e1bb3e4a5e4de0d557)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 899d95efe12f1e250b361837c1c8c06df9ac9b86)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit c82ae85a8a78a98f7c7fea68d24a4ac0ca74d01f)

Conflicts:
	libavcodec/mpegvideo_common.h

Signed-off-by: Anton Khirnov <anton@khirnov.net>
lavfi: avfilter_merge_formats: handle case where inputs are same

This fixes a double-free crash if lists are the same due to the two
merge_ref() calls at the end of the (useless) merging that happens.

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 11b6a82412bcd372adf694a26d83b07d337e1325)

Conflicts:

	libavfilter/formats.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit e5f4e249422834f727bcd432b73af971277f1371)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit b6c5848a1f8fc2755ea70d325acaddae9fac45ab)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit a4e277312cacfb78ef7583ed0b4fe4ccf5a0bcb1)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
Merge remote-tracking branch 'qatar/release/0.5' into release/0.5

* qatar/release/0.5:
  lavfi: avfilter_merge_formats: handle case where inputs are same
  mpegvideo: Don't use ff_mspel_motion() for vc1
  imgconvert: avoid undefined left shift in avcodec_find_best_pix_fmt
  nuv: check RTjpeg header for validity
  vc1dec: add flush function for WMV9 and VC-1 decoders

Merged-by: Michael Niedermayer <michaelni@gmx.at>
huffyuvdec: Check init_vlc() return codes.

Prevents out of array writes

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f67a0d115254461649470452058fa3c28c0df294)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 95ab8d33e1a680f30a5a9605175112008ab81afc)

Conflicts:

	libavcodec/huffyuv.c
(cherry picked from commit 277def59fce10d91e3113e5c0f63e22bc4abfa88)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
huffyuvdec: Skip len==0 cases

Fixes vlc decoding for hypothetical files that would contain such cases.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 0dfc01c2bbf4b71bb56201bc4a393321e15d1b31)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 5ff41ffeb4cb9ea6df49757dc859619dc3d3ab4f)

Conflicts:

	libavcodec/huffyuv.c
(cherry picked from commit 9bc70fe1ae50fd2faa0b9429d47cfbda01a92ebc)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Bump version number for the 0.5.10 release
bmpdec: only initialize palette for pal8.

Gray8 is not considered to be paletted, so this would cause an invalid
write.

Fixes bug 367.

CC: libav-stable@libav.org
(cherry picked from commit 8b78c2969a5b7dca939d93bf525aa2bcd737b5d9)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
lavfi: avfilter_merge_formats: handle case where inputs are same

This fixes a double-free crash if lists are the same due to the two
merge_ref() calls at the end of the (useless) merging that happens.

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 11b6a82412bcd372adf694a26d83b07d337e1325)

Conflicts:

	libavfilter/formats.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
avsdec: Set dimensions instead of relying on the demuxer.

The decode function assumes that the video will have those dimensions.

Fixes CVE-2012-2801

CC:libav-stable@libav.org

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 85f477935cd6b34e6ec2716b20e15ce748277a89)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
bytestream: add a new set of bytestream functions with overread checking

Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>
avidec: use actually read size instead of requested size

Fixes CVE-2012-2788
(cherry picked from commit 0af49a63c7f87876486ab09482d5b26b95abce60)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
cavsdec: check for changing w/h.

Our decoder does not support changing w/h.

Fixes CVE-2012-2777 and CVE-2012-2784.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit c20a69630619d14ae92c5541d52c579d7c8f3e94)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
avidec: return 0, not packet size from read_packet().

(cherry picked from commit eeade678f0a2bac127aeed2fb68d8717a6463420)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
yuv4mpeg: return proper error codes.

Fixes Bug 373.

CC:libav-stable@libav.org
(cherry picked from commit d3a72becc6371563185a509b94f5daf32ddbb485)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
h264: avoid stuck buffer pointer in decode_nal_units

When decode_nal_units() previously encountered a NAL_END_SEQUENCE,
and there are some junk bytes left in the input buffer, but no start codes,
buf_index gets stuck 3 bytes before the end of the buffer.

This can trigger an infinite loop in the caller code, eg. in
try_decode_trame(), as avcodec_decode_video() then keeps returning zeroes,
with 3 bytes of the input packet still available.

With this change, the remaining bytes are skipped so the whole packet gets
consumed.

CC:libav-stable@libav.org

Signed-off-by: Jindřich Makovička <makovick@gmail.com>
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 1a8c6917f68f7378465e18f7615762bfd22704c2)

Conflicts:

	libavcodec/h264.c
mpegaudiodec: fix short_start calculation

The value should be always 3, as it follows from the specification.

Fix a stack buffer overflow in exponents_from_scale_factors as reported
by asan. Thanks to Dale Curtis for the sample vector.
(cherry picked from commit 97cfa55eea39cef30abe14682c56c1e4e7f6f10d)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
tiffenc: Check av_malloc() results.

(cherry picked from commit b92dfb56d4582633571db18c3d904f8602eaa2a6)

Conflicts:

	libavcodec/tiffenc.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>