avformat/wavenc: more specific error return for "Writing 16 bit peak for 8 bit audio"

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
avformat/wavenc: simplify malloc failure checking

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>