rtmp: Support AMF_DATA_TYPE_MIXEDARRAY

And fix the AMF_DATA_TYPE_ARRAY parsing while at it.

A MIXEDARRAY type, as the ARRAY, store the number of elements in
an uint32 before the list. The ARRAY is strict and does not have
an OBJECT terminator, MIXEDARRAY behaves like an OBJECT type and
a different than stated number of element can be present.
rtmp: Refactor get_packet
rtmp: Rewrite embedded flv handling

Use update_offset() as done for rtmp audio, video and notifications and
read update and write the fields instead of replacing them in the rtmp
packet and then memcpying it to the output buffer.
rtmp: Replace a magic number with a macro

11 is the RTMP header size.
flv: Make onTextData parsing robust

Certain streaming servers do not preserve the order of the fields.
tls: Do not abort on non-fatal TLS alerts with gnutls

Signed-off-by: Martin Storsjö <martin@martin.st>
movenc: Set mov->mode earlier in mov_write_header

This allows simplifying some early checks that depend on the
muxer mode.

Signed-off-by: Martin Storsjö <martin@martin.st>
movenc: Set all implicit flags immediately in mov_write_header

This makes sure other sanity checks for conflicting options
can work properly, e.g. for the conflict between the faststart
flag when using the ismv mode.

Signed-off-by: Martin Storsjö <martin@martin.st>
movenc: Use keyframes as default fragmentation point in ismv

Fragmenting blindly to a certain duration isn't a good choice
if one should be able to switch between different qualities,
therefore default to keyframes instead.

Signed-off-by: Martin Storsjö <martin@martin.st>
wnv1: Make sure the input packet is large enough

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
vc1dec: Fix leaks in ff_vc1_decode_init_alloc_tables on errors

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
vc1dec: Undo mpegvideo initialization if unable to allocate tables

Previously, s->context_initialized was left set to 1
if ff_vc1_decode_init_alloc_tables failed, skipping the
initialization completely on the next decode call.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
vc1dec: Make sure last_picture is initialized in vc1_decode_skip_blocks

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
vc1dec: Don't decode slices when the latest slice header failed to decode

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
lavf: Avoid setting avg_frame_rate if delta_dts is negative

This avoids setting avg_frame_rate to invalid (negative)
values.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
electronicarts: Check packet sizes before reading

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
vc1dec: Check the error handling flags on slice/field header decode errors

Signed-off-by: Martin Storsjö <martin@martin.st>
movenc: Return a proper error code for invalid combinations

Signed-off-by: Martin Storsjö <martin@martin.st>
movenchint: Clear size and len if the realloc failed

Previously these were left in an inconsistent state.

Pointed out by Michael Niedermayer.

Signed-off-by: Martin Storsjö <martin@martin.st>
h263dec: Remove a hack that can cause infinite loops

The actual usefulness of the hack is not known, and it does cause
infinite loops with some broken input files.

CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>