avconv_opt: fix avio_open2() return code check

avio_open2() only return < 0 when error.
dnxhdenc: return meaningful return codes