avfilter/formats: Avoid using non public AV_SAMPLE_FMT_NB

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>