avformat/hdsenc: check mkdir() return code

This also returns failure if the mkdir failure is not due to an already existing
path.

Fixed CID1135749

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>