avutil/pixelutils: check for malloc failure

Found-by: Daemon404
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>