avformat/paf: Fix integer overflow and out of array read

Found-by:  Laurent Butti <laurentb@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
avutil/mem: Fix flipped condition

Fixes return code and later null pointer dereference

Found-by: Laurent Butti <laurentb@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>