avfilter/avfiltergraph: dont "or" together error codes

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
avfilter/vf_format: Check pix_fmts before dereferencing it

Fixes CID1224286

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>