demux_mkv: add some overflow checks etc.

Some of these might be security relevant.

The RealAudio code was especially bad. I'm not sure if all RealAudio
stuff still plays correctly; I didn't have that many samples for
testing. Some checks might be unnecessary or overcomplicated compared
to the (obfuscated) nature of the code.

CC: @mpv-player/stable
demux_mkv: cosmetics