atrac3: avoid oversized shifting in decode_bytes()

When `off' is 0, `0x537F6103 << 32' in the following expression invokes
undefined behavior, the result of which is not necessarily 0.

    (0x537F6103 >> (off * 8)) | (0x537F6103 << (32 - (off * 8)))

Avoid oversized shifting.

CC: libav-stable@libav.org

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
flacdec: simplify bounds checking in flac_probe()

Simplify `p->buf > p->buf + p->buf_size - 4' as `p->buf_size < 4'.
Avoid a possible out-of-bounds pointer, which is undefined behavior
in C.

CC: libav-stable@libav.org

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
lzo: fix overflow checking in copy_backptr()

The check `src > dst' in the form `&c->out[-back] > c->out' invokes
pointer overflow, which is undefined behavior in C.

Remove the check.  Also replace `&c->out[-back] < c->out_start' with
a safe form `c->out - c->out_start < back' to avoid overflow.

CC: libav-stable@libav.org

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
lavf: sanity check size in av_get/append_packet().

To avoid allocating ridiculous amounts of memory for corrupted files,
read the input in chunks limited to filesize or an arbitrary large
amount when that is not known (chosen to be 50M).
lavfi: use designated initializers in avfilter_class
avfiltergraph: replace AVFilterGraph.filter_count with nb_filters

This is more consistent with the naming in the rest of Libav.
avfiltergraph: use sizeof(var) instead of sizeof(type)
filters.texi: restore mistakenly removed section name for noformat
AVOption: remove an unused function parameter.
AVOptions: fix using named constants with child contexts.

The named constant needs to be searched for in the same object on which
the option is set, i.e. target_obj.
vf_showinfo: remove its useless init function

Filter private data is memset to 0 so there is no point in explicitly
initing anything to 0.
asrc_anullsrc: do not set samplerate and channel layout explicitly

They are auto-negotiated.
asrc_anullsrc: return EOF, not -1
af_join: do not leak input frames.
mpeg12: reduce hwaccel-related code duplication.

Signed-off-by: Reimar Döffinger <Reimar.Doeffinger@gmx.de>
Make ff_win32_open more robust.

- Make MultiByteToWideChar fail when it encounters invalid encoding.
  Without this, invalid characters might just be skipped
- When MultiByteToWideChar fails, assume the file name is in CP_ACP
  and open it via normal open function, even when the file will be
  written
- When malloc fails return error instead of crashing

Signed-off-by: Reimar Döffinger <Reimar.Doeffinger@gmx.de>
vc1testenc: give muxer same name as demuxer

Otherwise ffmpeg -formats claims that we only support demuxing
of that format.
To keep compatibility the struct could be duplicated instead,
but this seems almost like overkill for such a rare format.

Signed-off-by: Reimar Döffinger <Reimar.Doeffinger@gmx.de>
Add raw VC-1 muxer to match demuxer.

This is admittedly kind of pointless since usually -f image2pipe
can be used for the purpose, but this is more user-friendly.

Signed-off-by: Reimar Döffinger <Reimar.Doeffinger@gmx.de>
Merge commit 'ca6c3f2c53be70aa3c38e8f1292809db89ea1ba6'

* commit 'ca6c3f2c53be70aa3c38e8f1292809db89ea1ba6':
  lzo: fix overflow checking in copy_backptr()
  flacdec: simplify bounds checking in flac_probe()
  atrac3: avoid oversized shifting in decode_bytes()

Conflicts:
	libavformat/flacdec.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
Merge commit 'aa3c77998404cc32233cb76e961ca27db8565459'

* commit 'aa3c77998404cc32233cb76e961ca27db8565459':
  lavf: sanity check size in av_get/append_packet().

Conflicts:
	libavformat/utils.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>