mxfenc: switch to av_reallocp_array() and check allocation errors

Signed-off-by: Anton Khirnov <anton@khirnov.net>