subtitles: Fix some memory leaks when loading ext subs

Also hexdump first 64 bytes of subtitle file if we can't detect format

refs #1835