demux_lavf: refactor format probing hack

Restructure it a bit, so we can use the format hack list even if no mime
type applies. Shouldn't change anything functionally yet. Preparation
for the next commit.
demux_lavf: add hack to workaround too unreliable mp3 detection

This generally affects mp3 files that don't have any (or many) mp3
frames in the first 2 MB. 2 MB is the maximum probe size, and
libavformat returns a low probescore even if we give it the full 2 MB.
Trying to probe a larger buffer (or even the full file) doesn't work for
mysterious reasons.

The workaround consists in accepting a very weak probescore if the
format is detected as mp3 and we probed already 2 MB.
sub: fix crash with certain uses of --vf=sub

If, for some reason, the subtitle renderer attempts to render a
subtitle before SD_CTRL_SET_VIDEO_PARAMS was called, it passed a
value calculated from invalid values. This can happen with --vf=sub
and --start. The crash happens if 1. there was a subtitle packet that
falls into the timestamp of the rendered video frame, 2. the playloop
hasn't informed the subtitle decoder about the video resolution yet
(normally unneeded, because that is used for weird corner cases only,
so this code is a bit fuzzy), and 3. something actually requests a
frame to be drawn from the subtitle renderer, like with vf_sub.

The actual crash was due to passing NaN as pixel aspect to libass,
which then created glyphs with ridiculous sizes, involving a few
integer overflows and unchecked mallocs.

The sd_lavc.c and sd_spu.c cases probably don't crash, but I'm not
sure, and it's better fix them anyway.

Not bothering with sd_spu.c, this crap is for compatibility and will
be removed soon.

Note that this would have been no problem, had the code checked whether
SD_CTRL_SET_VIDEO_PARAMS was actually called. This commit adds such a
check (although it basically checks after using the parameters).

Regression since 49caa0a7 and 633fde4a.
w32: don't disable the error reporting dialog

Windows users expect this when a program crashes. Without it, the
program just disappears. Also change the SetErrorMode call to use macros
instead of a hardcoded constant.
w32: enable heap corruption detection

Enable the terminate-on-corruption feature. This is recommended for new
Windows applications and shouldn't cause a performance hit. It actually
shouldn't change anything for 64-bit builds, since Win64 has this
switched on by default.

See:
http://blogs.msdn.com/b/michael_howard/archive/2008/02/18/faq-about-heapsetinformation-in-windows-vista-and-heap-based-buffer-overruns.aspx
w32: use safe DLL search paths everywhere

Windows applications that use LoadLibrary are vulnerable to DLL
preloading attacks if a malicious DLL with the same name as a system DLL
is placed in the current directory. mpv had some code to avoid this in
ao_wasapi.c. This commit just moves it to main.c, since there's no
reason it can't be used process-wide.

This change can affect how plugins are loaded in AviSynth, but it
shouldn't be a problem since MPC-HC also does this and it's a very
popular AviSynth client.
wayland/shm: remove resize boilerplate

The problem seems to have solved itself. I guess the previous changes to
resizing and commit ba101ab made this possible. Consider me happy for removing
that crap.
wayland/shm: fix memory leak
wayland/shm: RGB888 as default, change options

RGB565 is one of the fastest and most supported formats on low end consumer
devices, but ffmpeg spams warning when using it. Make it opt-in instead of
opt-out.
sd_lavc: skip 0 sized sub-bitmaps

Not everything in the OSD path handles 0x0 sized sub-bitmaps well. At
least the code implementing --sub-gray had severe problems with it.

Fix this by skipping such bitmaps.
mp_image: reject too large image sizes

Larger sizes can introduce overflows, depending on the image format. In
the worst case, something larger than 16000x16000 with 8 bytes per pixel
will overflow 31 bits.

Maybe there should be a proper failure path instead of a hard crash, but
not yet. I imagine anything that sets a higher image size than a known
working size should be forced to call a function to check the size (much
like in ffmpeg/libavutil).
msg: don't clear the status line if new and previous status was empty

This avoids stray newlines when:

1. Some (non-status line) text was output
2. Then an empty status line is output

According to the logic, 2. should print an empty line to show the blank
status line. Don't do that, and instead output nothing in this case.

This caused problems with mpv_identify.sh, and also looked ugly when
using --quiet.
vo_wayland: silence shadowing warning

No real problem.