demux_mkv: check file type without actually reading data

Do a minimal check on data read with stream_peek(). This could help with
probing from unseekable streams in some situations. (We could check the
entire EBML and Matroska headers, but probably not worth the trouble. We
could also seek back to the start, which demux.c doesn't do, but which
would work usually - also not worth the trouble.)
lua: subprocess: remove minor code duplication

Now that the code for stderr and stdout does exactly the same things,
and the specialization is in the callbacks, this is blatantly
duplicated.

Also, define a typedef for those callbacks to reduce the verbosity.