avformat/soxenc: return more meaningful error code

Signed-off-by: Paul B Mahol <onemda@gmail.com>