lavf: simplify handling of offset in av_probe_input_buffer()

(cherry picked from commit c1868e7ee7b07b40a0fe15f50df89fe499a01a50)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
lavf: use a fixed width type

It's shorter and more consistent with the rest of the code.

(cherry picked from commit 8b76362836f3c373c3aadc544522edcbef16dd5f)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
lavf: make av_probe_input_buffer more robust

Always use the actually read size as the offset instead of making
possibly invalid assumptions.

Addresses: CVE-2012-6618

(cherry picked from commit 2115a3597457231a6e5c0527fe0ff8550f64b733)

Conflicts:
	libavformat/utils.c

Signed-off-by: Anton Khirnov <anton@khirnov.net>
dsputil/pngdsp: fix signed/unsigned type in end comparison

Fixes out of array accesses and integer overflows.

(cherry picked from commit d1916d13e28b87f4b1b214231149e12e1d536b4b)
Adresses: CVE-2013-7010, CVE-2013-7014

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
flashsv: Check diff_start diff_height values

Fix out of array accesses.

Found-by: ami_stuff
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Adresses: CVE-2013-7015
(cherry picked from commit 57070b1468edc6ac8cb3696c817f3c943975d4c1)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
vmnc: Port to bytestream2

Fix some buffer overreads.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
cavsdec: check ff_get_buffer() return value

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
vmnc: Check the cursor dimensions

And manage the reallocation failure path.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 5e992a4682d2c09eed3839c6cacf70db3b65c2f4)
h263: Check init_get_bits return value

And use init_get_bits8 to check for integer overflows while at it.

CC: libav-stable@libav.org
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
avi: DV in AVI must be considered single stream

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
h264_cavlc: check the size of the intra PCM data.

Fixes invalid reads.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
vmnc: K&R formatting cosmetics

Signed-off-by: Diego Biurrun <diego@biurrun.de>
cavs: Check for negative cbp

Sample-Id: 00000647-google
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
segafilm: fix leaks if reading the header fails

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
(cherry picked from commit 6892d145a0c80249bd61ee7dd31ec851c5076bcd)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
mov: Free an earlier allocated array if allocating a new one

It could probably also be considered an error if the pointer isn't
null at this point, but then we might risk rejecting some
slightly broken files that we might have handled so far.

Sample-Id: 00000496-google
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 2620df13104ddaa136158eb6bb1195adbf9d7692)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
mov: Free intermediate arrays in the normal cleanup function

These arrays are normally freed at the end of mov_read_trak,
but make sure they're freed in case mov_read_trak returned
early (due to errors) or in case the atoms that allocate arrays
are encountered at some other point than within a trak (which
we don't have checks against).

Sample-Id: 00000496-google
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit d51f09962d5b4bc999fb70c040f330dd1873212e)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
h264: reset num_reorder_frames if it is invalid

An invalid VUI is not considered a fatal error, so the SPS containing it
may still be used. Leaving an invalid value of num_reorder_frames there
can result in writing over the bounds of H264Context.delayed_pic.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
(cherry picked from commit 9ecabd7892ff073ae60ded3fc0a1290f5914ed5c)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>

Conflicts:
	libavcodec/h264_ps.c
vc1: Always reset numref when parsing a new frame header.

Fixes an issue where the B-frame coding mode switches from interlaced
fields to interlaced frames, causing incorrect decisions in the motion
compensation code and resulting in visual artifacts.

CC: libav-stable@libav.org
Signed-off-by: Tim Walker <tdskywalker@gmail.com>
(cherry picked from commit dd2d0039b6405dc724e4fef0d5b8f49530eea3aa)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
mathematics: remove asserts from av_rescale_rnd()

It is a public function, it must not assert on its parameters.

(cherry picked from commit 94a417acc05cc5151b473abc0bf51fad26f8c5a0)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
oggparseogm: check timing variables

Fixes a potential divide by zero.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
(cherry picked from commit 75647dea6f7db79b409bad66a119f5c73da730f3)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>