postproc: fix null pointer dereference with invalid option strings

Fixes CID1135760
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
tools/probetest: Check av_realloc() return code

Fixess CID1135761
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>