Commit message: stream_out: rtp: don't use VLA for user controlled data

It should fix a possible invalid memory access

When streaming ogg-files via rtp, an ogg-file can trigger an invalid
write access using an overly long 'configuration' string.

The original code attemps to allocate space to hold the string on the stack
and hence, cannot verify if allocation succeeds. Instead, we now allocate the
buffer on the heap and return if allocation fails.

In detail, rtp_packetize_xiph_config allocates a buffer on the stack at (1) where
the size depends on the local variable 'len'. The variable 'len' is
calculated at (0) to be the length of a string contained in a specially
crafted Ogg Vorbis file, and therefore, it is attacker-controlled.

Signed-off-by: Jean-Baptiste Kempf <jb@videolan.org>
CVE: CVE-2014-9630
CWE: CWE-787
Vulnerability description: The rtp_packetize_xiph_config function in modules/stream_out/rtpfmt.c in VideoLAN VLC media player before 2.1.6 uses a stack-allocation approach with a size determined by arbitrary input data, which allows remote attackers to cause a denial of service (memory corruption) or possibly have unspecified other impact via a crafted length value.

Project: vlc
File: rtpfmt.c

int rtp_packetize_xiph_config( sout_stream_id_sys_t *id, const char *fmtp,
                               int64_t i_pts )
{
    if (fmtp == NULL)
        return VLC_EGENERIC;

    /* extract base64 configuration from fmtp */
    char *start = strstr(fmtp, "configuration=");
    assert(start != NULL);
    start += sizeof("configuration=") - 1;
    char *end = strchr(start, ';');
    assert(end != NULL);
    size_t len = end - start;
    char b64[len + 1];
    memcpy(b64, start, len);
    b64[len] = '\0';

    int     i_max   = rtp_mtu (id) - 6; /* payload max in one packet */

    uint8_t *p_orig, *p_data;
    int i_data;

    i_data = vlc_b64_decode_binary(&p_orig, b64);
    if (i_data <= 9)
    {
        free(p_orig);
        return VLC_EGENERIC;
    }
    p_data = p_orig + 9;
    i_data -= 9;

    int i_count = ( i_data + i_max - 1 ) / i_max;

    for( int i = 0; i < i_count; i++ )
    {
        int           i_payload = __MIN( i_max, i_data );
        block_t *out = block_Alloc( 18 + i_payload );

        unsigned fragtype, numpkts;
        if (i_count == 1)
        {
            fragtype = 0;
            numpkts = 1;
        }
        else
        {
            numpkts = 0;
            if (i == 0)
                fragtype = 1;
            else if (i == i_count - 1)
                fragtype = 3;
            else
                fragtype = 2;
        }
        /* Ident:24, Fragment type:2, Vorbis/Theora Data Type:2, # of pkts:4 */
        uint32_t header = ((XIPH_IDENT & 0xffffff) << 8) |
                          (fragtype << 6) | (1 << 4) | numpkts;

        /* rtp common header */
        rtp_packetize_common( id, out, 0, i_pts );

        SetDWBE( out->p_buffer + 12, header);
        SetWBE( out->p_buffer + 16, i_payload);
        memcpy( &out->p_buffer[18], p_data, i_payload );

        out->i_dts    = i_pts;

        rtp_packetize_send( id, out );

        p_data += i_payload;
        i_data -= i_payload;
    }

    free(p_orig);

    return VLC_SUCCESS;
}