encoding: when output is pipe: or pipe:1, avoid mp_msg to stdout

I am aware this detection may occur too late, depending on other
settings, but at least it usually works and is portable.

Where the output fd can be changed, though, it'd be better to force a
similar behaviour via file descriptor use: use pipe:3 as output to FD 3,
and change the calling program to expect the stream on FD 3.