Permit serialization to fail.

Prior to this, serialization was to a memory buffer and so never
actually fails, per se--it can certainly run out of memory, but this
crashes the entire server at present.  This adds a weak "failure
condition"; it does not abort serialization per se but it does alert
to incorrect features of the data (in this case, an array that is too
long).  The `serialize` method itself ignores this return value, but
the other consumers of datum_serialize can use it to reject overly
large arrays.