avfilter: fix plane validity checks

Fixes out of array accesses

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>