avcodec/avpacket: fix order of operations in case of too large allocation

Found-by: wm4
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>