shorten: Use separate pointers for the allocated memory for decoded samples.

Fixes invalid free() if any of the buffers are not allocated due to either
not decoding a header or an error prior to allocating all buffers.

Fixes CVE-2012-0858
CC: libav-stable@libav.org

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>
(cherry picked from commit 204cb29b3c84a74cbcd059d353c70c8bdc567d98)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
golomb: avoid infinite loop on all-zero input (or end of buffer).

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit c6643fddba73560f26f90d327c84d8832222a720)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
get_bits: add HAVE_BITS_REMAINING macro.

(cherry picked from commit b44b41633f110e9d938165e0f79c9d32191fc135)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
golomb: use HAVE_BITS_REMAINING() macro to prevent infloop on EOF.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 46b3fbc30b7aaf7fdd52391734cfd6d93af8720a)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
flac: fix infinite loops on all-zero input or end-of-stream.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 52e4018be47697a60f4f18f83551766df31f5adf)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
rv20: prevent calling ff_h263_decode_mba() with unset height/width

Prevents a crash of VLC during playback of a invalid matroska file,
found by John Villamil <johnv@matasano.com>.

CC: libav-stable@libav.org
(cherry picked from commit c3e10ae4127c998b809066926a410f40ebd47593)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
wma: Clip WMA1 and WMA2 frame length to 11 bits.

The MDCT buffers in the decoder are only sized for up to 11 bits. The
reverse engineered documentation for WMA1/2 headers say that that for
all samplerates above 32kHz 11 bits are used. 12 and 13 bit support
were added for WMAPro. I was unable to make any Microsoft tools generate
a test file at a samplerate above 48kHz.

Discovered by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind

CC: libav-stable@libav.org
(cherry picked from commit d78bb1a4b2a3a415b68e4e6dd448779eccec64e3)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
aac: fix infinite loop on end-of-frame with sequence of 1-bits.

Based-on-work-by: Ronald S. Bultje <rsbultje@gmail.com>
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 1cd9a6154bc1ac1193c703cea980ed21c3e53792)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
matroskadec: Pad AAC extradata.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind

CC: libav-stable@libav.org
(cherry picked from commit d2ee8c17793201ce969afd1f433ba1580c143cd2)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
matroska: don't overwrite string values until read/alloc was succesful.

This prevents certain tags with a default value assigned to them (as per
the EBML syntax elements) from ever being assigned a NULL value. Other
parts of the code rely on these being non-NULL (i.e. they don't check for
NULL before e.g. using the string in strcmp() or similar), and thus in
effect this prevents crashes when reading of such specific tags fails,
either because of low memory or because of targeted file corruption.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit cd40c31ee9ad2cca6f3635950b002fd46be07e98)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
lavf: prevent infinite loops while flushing in avformat_find_stream_info

If no data was seen for a stream decoder are returning 0 when fed with
empty packets for flushing. We can stop flushing when the decoder does
not return delayed delayed frames anymore. Changes try_decode_frame()
return value to got_picture or negative error.

CC: libav-stable@libav.org
(cherry picked from commit b3461c29c1aee7d62eeb02a59d46593c60362679)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
smacker: Sanity check huffman tables found in the headers.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind

CC: libav-stable@libav.org
(cherry picked from commit 9adf25c1cf78dbf1d71bf386c49dc74cb8a60df0)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
vc1: prevent null pointer dereference on broken files

CC: libav-stable@libav.org
(cherry picked from commit 510ef04a461b3b54a762c6141ad880cbed85981f)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
tta: error out if samplerate is zero.

Prevents a division by zero later on.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 7416d610362807848236ceff1bc6740dbc82842d)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
swscale: enforce a minimum filtersize.

At very small dimensions, this calculation could lead to zero-sized
filters, which leads to uninitialized output, zero-sized allocations,
loop overflows in SIMD that uses do{..}while(i++<filtersize); instead
of for(i=0;i<filtersize;i++){..} and several other similar failures.
Therefore, require a minimum filtersize of 1.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit dae2ce361a2b5fd9be1d43e5e8c00bdbc5f03e3d)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
swscale: fix overflows in filterPos[] calculation for large sizes.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 19a65b5be47944c607a9e979edb098924d95f2e4)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
swscale: take first/lastline over/underflows into account for MMX.

Fixes crashes for extremely large resizes (several 100-fold).

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 1d8c4af396b6ed84c84b5ebf0bf1163c4a7a3017)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
vc1: prevent using last_frame as a reference for I/P first frame.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit ae591aeea58d64399b8281be31dacec0de85ae04)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
cook: prevent div-by-zero if channels is zero.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 941fc1ea1ed7f7d99a8b9e2607b41f2f2820394a)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
als: prevent infinite loop in zero_remaining().

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit af468015d972c0dec5c8c37b2685ffa5cbe4ae87)

Signed-off-by: Anton Khirnov <anton@khirnov.net>