dsputil/pngdsp: fix signed/unsigned type in end comparison

Fixes out of array accesses and integer overflows.

(cherry picked from commit d1916d13e28b87f4b1b214231149e12e1d536b4b)
Adresses: CVE-2013-7010, CVE-2013-7014

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
flashsv: Check diff_start diff_height values

Fix out of array accesses.

Found-by: ami_stuff
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Adresses: CVE-2013-7015
(cherry picked from commit 57070b1468edc6ac8cb3696c817f3c943975d4c1)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
vmnc: Port to bytestream2

Fix some buffer overreads.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
cavsdec: check ff_get_buffer() return value

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
vmnc: Check the cursor dimensions

And manage the reallocation failure path.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 5e992a4682d2c09eed3839c6cacf70db3b65c2f4)
h263: Check init_get_bits return value

And use init_get_bits8 to check for integer overflows while at it.

CC: libav-stable@libav.org
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
avi: DV in AVI must be considered single stream

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
h264_cavlc: check the size of the intra PCM data.

Fixes invalid reads.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
vmnc: K&R formatting cosmetics

Signed-off-by: Diego Biurrun <diego@biurrun.de>
cavs: Check for negative cbp

Sample-Id: 00000647-google
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
segafilm: fix leaks if reading the header fails

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
(cherry picked from commit 6892d145a0c80249bd61ee7dd31ec851c5076bcd)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
mov: Free an earlier allocated array if allocating a new one

It could probably also be considered an error if the pointer isn't
null at this point, but then we might risk rejecting some
slightly broken files that we might have handled so far.

Sample-Id: 00000496-google
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 2620df13104ddaa136158eb6bb1195adbf9d7692)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
mov: Free intermediate arrays in the normal cleanup function

These arrays are normally freed at the end of mov_read_trak,
but make sure they're freed in case mov_read_trak returned
early (due to errors) or in case the atoms that allocate arrays
are encountered at some other point than within a trak (which
we don't have checks against).

Sample-Id: 00000496-google
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit d51f09962d5b4bc999fb70c040f330dd1873212e)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
h264: reset num_reorder_frames if it is invalid

An invalid VUI is not considered a fatal error, so the SPS containing it
may still be used. Leaving an invalid value of num_reorder_frames there
can result in writing over the bounds of H264Context.delayed_pic.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
(cherry picked from commit 9ecabd7892ff073ae60ded3fc0a1290f5914ed5c)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>

Conflicts:
	libavcodec/h264_ps.c
vc1: Always reset numref when parsing a new frame header.

Fixes an issue where the B-frame coding mode switches from interlaced
fields to interlaced frames, causing incorrect decisions in the motion
compensation code and resulting in visual artifacts.

CC: libav-stable@libav.org
Signed-off-by: Tim Walker <tdskywalker@gmail.com>
(cherry picked from commit dd2d0039b6405dc724e4fef0d5b8f49530eea3aa)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
mathematics: remove asserts from av_rescale_rnd()

It is a public function, it must not assert on its parameters.

(cherry picked from commit 94a417acc05cc5151b473abc0bf51fad26f8c5a0)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
oggparseogm: check timing variables

Fixes a potential divide by zero.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
(cherry picked from commit 75647dea6f7db79b409bad66a119f5c73da730f3)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
h264: check that an IDR NAL only contains I slices

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
(cherry picked from commit 8b2e5e42bb9d6a59ede5af2e6df4aaf7750d1195)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
Update Changelog for 9.11
Merge commit 'f1476459b7013d306eb911573f1dc81e74ccd082' into release/1.1

* commit 'f1476459b7013d306eb911573f1dc81e74ccd082':
  vmnc: K&R formatting cosmetics
  flashsv: Check diff_start diff_height values
  dsputil/pngdsp: fix signed/unsigned type in end comparison

Conflicts:
	libavcodec/dsputil.c
	libavcodec/flashsv.c
	libavcodec/vmnc.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>