atrac3: avoid oversized shifting in decode_bytes()

When `off' is 0, `0x537F6103 << 32' in the following expression invokes
undefined behavior, the result of which is not necessarily 0.

    (0x537F6103 >> (off * 8)) | (0x537F6103 << (32 - (off * 8)))

Avoid oversized shifting.

CC: libav-stable@libav.org

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
flacdec: simplify bounds checking in flac_probe()

Simplify `p->buf > p->buf + p->buf_size - 4' as `p->buf_size < 4'.
Avoid a possible out-of-bounds pointer, which is undefined behavior
in C.

CC: libav-stable@libav.org

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
lzo: fix overflow checking in copy_backptr()

The check `src > dst' in the form `&c->out[-back] > c->out' invokes
pointer overflow, which is undefined behavior in C.

Remove the check.  Also replace `&c->out[-back] < c->out_start' with
a safe form `c->out - c->out_start < back' to avoid overflow.

CC: libav-stable@libav.org

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
mpeg12: reduce hwaccel-related code duplication.

Signed-off-by: Reimar Döffinger <Reimar.Doeffinger@gmx.de>
Make ff_win32_open more robust.

- Make MultiByteToWideChar fail when it encounters invalid encoding.
  Without this, invalid characters might just be skipped
- When MultiByteToWideChar fails, assume the file name is in CP_ACP
  and open it via normal open function, even when the file will be
  written
- When malloc fails return error instead of crashing

Signed-off-by: Reimar Döffinger <Reimar.Doeffinger@gmx.de>
vc1testenc: give muxer same name as demuxer

Otherwise ffmpeg -formats claims that we only support demuxing
of that format.
To keep compatibility the struct could be duplicated instead,
but this seems almost like overkill for such a rare format.

Signed-off-by: Reimar Döffinger <Reimar.Doeffinger@gmx.de>
Add raw VC-1 muxer to match demuxer.

This is admittedly kind of pointless since usually -f image2pipe
can be used for the purpose, but this is more user-friendly.

Signed-off-by: Reimar Döffinger <Reimar.Doeffinger@gmx.de>
Merge commit 'ca6c3f2c53be70aa3c38e8f1292809db89ea1ba6'

* commit 'ca6c3f2c53be70aa3c38e8f1292809db89ea1ba6':
  lzo: fix overflow checking in copy_backptr()
  flacdec: simplify bounds checking in flac_probe()
  atrac3: avoid oversized shifting in decode_bytes()

Conflicts:
	libavformat/flacdec.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
lavfi: jfmeter filter

Signed-off-by: Paul B Mahol <onemda@gmail.com>