Sereal/Sereal — pull request #34 by tsee (2013-06-17 05:37:35+00:00)
Title: Sereal V2: Introducing relocatable bodies and implementing user-data-in-header

This pull request is as much an RFC as a real pull request. The changes to the Perl implementation are extensive and not all of the changes (specifically those to the tests) are exceedingly pretty. Please review at the very least the changes to the Sereal spec. We can spend some more time and effort on changing Sereal in incompatible ways now since we're introducing a major protocol change already!

Nonetheless, this branch brings significant improvements in that with all offsets now being relative to the start of the document body, a Sereal document can now fundamentally have multiple document bodies and the encoder can actually shuffle them around with memcpy without having to entirely re-encode them. In case that's not obvious: That was necessary previously, since offsets didn't only change value, they could actually change size because they're encoded as varints.

As a first use case, I've introduced custom user data into the header of a Sereal document. That can be very useful for embedding framing/routing type of information that needs to be read without deserializing the entire document.

Sereal V2 is an incompatible change. The Perl/XS implementation can generate both Sereal V1 and V2 and transparently decodes either without even requiring as much as configuration by the user.

As a side-effect of V2, we're also dropping support for the old Snappy document type - incremental Snappy is now used when selecting Snappy compression. The old doc type is still supported for V1 documents.

If somebody wants to add support for "symbols" as per ticket #31 to the spec, then now is the time to think that through carefully!