common: fix seek relative end of file offset handling

fseeko() etc. require negative offsets if seeking relative to the end of
the file should result in the file pointer being inside the
file. Therefore MKVToolNix' APIs require that, too, but some
calculations treated it as if it were the other war around.

Fixes #1035.