rtmp: fix multiple broken overflow checks

Sanity checks like `data + size >= data_end || data + size < data' are
broken, because `data + size < data' assumes pointer overflow, which is
undefined behavior in C.  Many compilers such as gcc/clang optimize such
checks away.

Use `size < 0 || size >= data_end - data' instead.

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Martin Storsjö <martin@martin.st>
rtmp: fix buffer overflows in ff_amf_tag_contents()

A negative `size' will bypass FFMIN().  In the subsequent memcpy() call,
`size' will be considered as a large positive value, leading to a buffer
overflow.

Change the type of `size' to unsigned int to avoid buffer overflow, and
simplify overflow checks accordingly. Also change a literal buffer
size to use sizeof, and limit the amount of data copied in another
memcpy call as well.

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Martin Storsjö <martin@martin.st>
rtpenc: fix overflow checking in avc_mp4_find_startcode()

The check `start + res < start' is broken since pointer overflow is
undefined behavior in C.  Many compilers such as gcc/clang optimize
away this check.

Use `res > end - start' instead.  Also change `res' to unsigned int
to avoid signed left-shift overflow.

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Martin Storsjö <martin@martin.st>