rtmp: fix multiple broken overflow checks

Sanity checks like `data + size >= data_end || data + size < data' are
broken, because `data + size < data' assumes pointer overflow, which is
undefined behavior in C.  Many compilers such as gcc/clang optimize such
checks away.

Use `size < 0 || size >= data_end - data' instead.

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 902cfe2f74d777a7dc20ac68f2393b9f84b790c1)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
rtmp: fix buffer overflows in ff_amf_tag_contents()

A negative `size' will bypass FFMIN().  In the subsequent memcpy() call,
`size' will be considered as a large positive value, leading to a buffer
overflow.

Change the type of `size' to unsigned int to avoid buffer overflow, and
simplify overflow checks accordingly.

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 4e692374f7962ea358c329de38c380103f8991b6)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
rv34: error out on size changes with frame threading

(cherry picked from commit cb7190cd2c691fd93e4d3664f3fce6c19ee001dd)

Fixes: CVE-2012-2772 (according to Ubuntu)
indeo5dec: Make sure we have had a valid gop header.

This prevents decoding happening on a half initialized context.

Fixes CVE-2012-2779

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 891918431db628db17885ed947ee387b29826a64)

Conflicts:

	libavcodec/ivi_common.c
	libavcodec/ivi_common.h
dfa: improve boundary checks in decode_dds1()

Fixes CVE-2012-2798

CC:libav-stable@libav.org
(cherry picked from commit d05f72c75445969cd7bdb1d860635c9880c67fb6)

Conflicts:

	libavcodec/dfa.c
indeo4/5: check empty tile size in decode_mb_info().

This prevents writing into a too small array if some parameters changed
without the tile being reallocated.

Based on a patch by Michael Niedermayer <michaelni@gmx.at>

Fixes CVE-2012-2800

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit ae3da0ae5550053583a6f281ea7fd940497ea0d1)

Conflicts:

	libavcodec/ivi_common.c
mpeg12: do not decode extradata more than once.

Fixes CVE-2012-2803.

(cherry picked from commit 582368626188c070d4300913c6da5efa4c24cfb2)

Conflicts:

	libavcodec/mpeg12.c
	libavcodec/mpeg12.h
Update changelog for 0.7.7 release
huffyuvdec: Check init_vlc() return codes.

Prevents out of array writes

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f67a0d115254461649470452058fa3c28c0df294)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 95ab8d33e1a680f30a5a9605175112008ab81afc)

Conflicts:

	libavcodec/huffyuv.c
(cherry picked from commit 277def59fce10d91e3113e5c0f63e22bc4abfa88)

Conflicts:

	libavcodec/huffyuv.c
(cherry picked from commit adf022f458d75e2c8041262e1906a249366ad518)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
huffyuvdec: Skip len==0 cases

Fixes vlc decoding for hypothetical files that would contain such cases.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 0dfc01c2bbf4b71bb56201bc4a393321e15d1b31)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 5ff41ffeb4cb9ea6df49757dc859619dc3d3ab4f)

Conflicts:

	libavcodec/huffyuv.c
(cherry picked from commit 9bc70fe1ae50fd2faa0b9429d47cfbda01a92ebc)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Merge remote-tracking branch 'qatar/release/0.7' into release/0.8

* qatar/release/0.7:
  Update changelog for 0.7.7 release
  mpeg12: do not decode extradata more than once.
  indeo4/5: check empty tile size in decode_mb_info().
  dfa: improve boundary checks in decode_dds1()
  indeo5dec: Make sure we have had a valid gop header.
  rv34: error out on size changes with frame threading

Conflicts:
	Changelog

Merged-by: Michael Niedermayer <michaelni@gmx.at>