rtpenc: fix overflow checking in avc_mp4_find_startcode()

The check `start + res < start' is broken since pointer overflow is
undefined behavior in C.  Many compilers such as gcc/clang optimize
away this check.

Use `res > end - start' instead.  Also change `res' to unsigned int
to avoid signed left-shift overflow.

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 2f014567cfd63e58156f60666f1a61ba147276ab)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
rtmp: fix multiple broken overflow checks

Sanity checks like `data + size >= data_end || data + size < data' are
broken, because `data + size < data' assumes pointer overflow, which is
undefined behavior in C.  Many compilers such as gcc/clang optimize such
checks away.

Use `size < 0 || size >= data_end - data' instead.

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 902cfe2f74d777a7dc20ac68f2393b9f84b790c1)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
rtmp: fix buffer overflows in ff_amf_tag_contents()

A negative `size' will bypass FFMIN().  In the subsequent memcpy() call,
`size' will be considered as a large positive value, leading to a buffer
overflow.

Change the type of `size' to unsigned int to avoid buffer overflow, and
simplify overflow checks accordingly.

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 4e692374f7962ea358c329de38c380103f8991b6)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
huffyuvdec: Check init_vlc() return codes.

Prevents out of array writes

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f67a0d115254461649470452058fa3c28c0df294)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 95ab8d33e1a680f30a5a9605175112008ab81afc)

Conflicts:

	libavcodec/huffyuv.c
huffyuvdec: Skip len==0 cases

Fixes vlc decoding for hypothetical files that would contain such cases.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 0dfc01c2bbf4b71bb56201bc4a393321e15d1b31)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 5ff41ffeb4cb9ea6df49757dc859619dc3d3ab4f)

Conflicts:

	libavcodec/huffyuv.c
movenc: hotfix, dont store fiel for h264 / mpeg4-asp / dnxhd

Other software does not store it in this case, and the information
is provided by the codec stream

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 405cc0d9052079307b2b4188c396ae30c8e9108b)

Conflicts:

	tests/ref/lavf/mov
	tests/ref/seek/lavf_mov
	tests/ref/vsynth/vsynth1-dnxhd-1080i
	tests/ref/vsynth/vsynth1-mpeg4
	tests/ref/vsynth/vsynth2-dnxhd-1080i
	tests/ref/vsynth/vsynth2-mpeg4
configure: Make warnings from -Wreturn-type fatal errors

These warnings have no false positives and point to serious bugs.
(cherry picked from commit 99853cb8d4237b810b2fffb4a34f66fd0064ef72)

Conflicts:

	configure

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
doc/APIchanges: fix odd .01 versions

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 9f16cb9e50a5a196af9244dc7d33ed193227528a)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
apichanges: fix date

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit ad6802f975a91bf6757fe3729ef8c6f10e6796b7)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
apichanges: Use , instead of / to seperate multiple hashes

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 33d6330652c088dadde163da569b1a2f6c7603c0)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
apichanges: fix 2 wrong hashes

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 2f3bc5122822687dc388f7352c92cf6db456cf7c)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
doc/APIchanges: List merge commit hashes and version numbers

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
mp3dec: Fix VBR bit rate parsing

When parsing the Xing/Info tag, don't set the bit rate if it's an Info tag.

When parsing the stream, don't override the bit rate if it's already set,
otherwise calculate the mean bit rate from parsed frames. This way, the bit
rate will be set correctly both for CBR and VBR streams.

Signed-off-by: Alexander Kojevnikov <alexander@kojevnikov.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 29d8cd265a536063420afe78375b2176a9e1abc5)

Conflicts:

	tests/ref/lavf-fate/mp3

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Merge remote-tracking branch 'jamrial/release/0.11' into release/0.11

* jamrial/release/0.11:
  lavc/bink: Chech for malloc failure

Merged-by: Michael Niedermayer <michaelni@gmx.at>