nutdec: print error on invalid/unsupported fourcc style

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
nutdec: more specific return codes for decode_syncpoint()

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>