libavformat: Check mkdir return error codes

Previously, the returned error codes were intentionally ignored
(see fadd3a68213), to avoid aborting if the directory already
existed. If the mkdir actually failed, this was caught when
opening files within the directory fails anyway.

By handling the error code here (but explicitly ignoring EEXIST),
the error messages and return codes in these cases are more
appropriate and less confusing.

Signed-off-by: Martin Storsjö <martin@martin.st>
ismindex: Avoid writing ismf files if no base name has been specified

Previously, this could create files named "(null).ismf", if the -ismf
parameter is specified (before an input file name), but without
specifying any base name.

Signed-off-by: Martin Storsjö <martin@martin.st>