encoder: ensure returned NAL count matches output array size

On CHECKED_MALLOC failure, m_packetData and/or m_nals will be NULL and thus the
returned count must be 0. Also, do not free the packet data from within this
utility function. It is cleaner to release that memory in the function which
declared the pointers on the stack.