cmdutils&ffmpeg: use av_fopen_utf8() for passlogfile

might fix Ticket3056

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>