Package: mapry
Mapry generates polyglot code for de/serializing object graphs from
JSONable structures.Story. We needed a yet another domain-specific language for internal data
exchange and configuration of the system. The existing solutions mostly focused
on modeling the configuration asobject treesin which the data is nested in
hierarchies with nocross-referencesbetween the objects.For example, think of object trees as JSON objects or arrays. We found this
structure to be highly limiting for most of the complex messages and system
configurations. Our use cases required objects in the data to be referenced
among each other – instead of object trees we neededobject graphs.Moreover, we wanted the serialization itself to bereadableso that an
operator can edit it using a simple text editor. JSONable structure offered
itself as a good fit with a lot of existing tools (JSON and
YAML modulesetc.).However, JSON allows only a limited set of data types (numbers, strings, arrays
and objects/maps). We found that most of our data relied ona richer set of primitivesthan provided by standard JSON. This
extended set includes:date,datetime,time of day,time zone,duration andpath.While there exist polyglot serializers of object trees (e.g.,Protocol Buffers),
language-specific serializers of object graphs (e.g.,Gob in GoorPickle in Python) or polyglot
ones with a limited set of primitives (e.g.,Flatbuffers), to the best of our
knowledge there is currently no serializer ofobject graphsthat operates
withreadable representationsand provides arich set of primitive data typesconsistentlyacross multiple languages.Hence we developed Mapry, a generator of polyglot de/serialization code
for object graphs from JSONable structures.Theschemaof the object graph is stored in a separate JSON file and defines
all the data types used in the object graph including the object graph itself.
The code is generated based on the schema. You define the schema once and
generate code in all supported languages automatically. Schemas can be
evolved and backward compatibility is supported through optional properties.Supported languagesCurrently, Mapry implements the following language bindings:C++11,Go 1 andPython 3.Since the serialization needs to operate in different languages, only the
intersection of language features is supported. For example, since Go does not
support inheritance or union types, they are not supported in Mapry either.WorkflowThe following diagram illustrates the workflow.DocumentationThis document gives only a brief summary of Mapry. The full documentation can be
foundhere.IntroductionLet us introduce Mapry here by presenting a short example in order to give you a first
impression on how the generator can be used. To get the full picture, please consult thedocumentation.The schema starts with the name and a description, followed by the
language-specific settings (specifying the non-standard parts of the code
generation), the definition of the graph structure and finally the definition of the properties of
the object graph itself.Here is an example schema to give you an overview:{"name":"Pipeline","description":"defines an address book.","cpp":{"namespace":"book::address","path_as":"boost::filesystem::path","optional_as":"std::experimental::optional","datetime_library":"ctime"},"go":{"package":"address"},"py":{"module_name":"book.address","path_as":"pathlib.Path","timezone_as":"pytz.timezone"},"classes":[{"name":"Person","description":"defines a contactable person.","properties":{"full_name":{"type":"string","description":"gives the full name (including middle names)."},"birthday":{"type":"date","description":"indicates the birthday in UTC."},"address":{"type":"Address","description":"notes where the person lives."}}}],"embeds":[{"name":"Address","description":"defines an address.","properties":{"text":{"type":"string","description":"gives the full address."}}}],"properties":{"maintainer":{"type":"Person","description":"indicates the maintainer of the address book."}}}Once you generated the de/serialization code with Mapry, you can use it
to obtain the object graph from a JSONable.For example, assume a JSONable stored in/path/to/the/file.json:{"persons":{"alice":{"full_name":"Alice Doe","birthday":"1983-10-24","address":{"text":"Some street 12, Some City, Some Country"}},"bob":{"full_name":"Bob Johnson","birthday":"2016-07-03","address":{"text":"Another street 36, Another City, Another Country"}}},"maintainer":"alice"}You can parse the object graph in, say, Python:# Obtain a JSONablepth='/path/to/the/file.json'withopen(pth,'rt')asfid:value=json.load(fid)# Parse the JSONableerrors=book.address.parse.Errors(cap=10)pipeline=book.address.fromjsonable.pipeline_from(value=value,ref=pth+'#',errors=errors)ifnoterrors.empty():forerrorinerrors.values():print("{}:{}".format(error.ref,error.message),file=sys.stderr)return1and access the object graph aspipeline:print('Persons are:')forpersoninpipeline.persons:print('{}(full name:{}, address:{}, birthday:{})'.format(person.id,person.full_name,person.address.text,person.birthday.strftime("%d.%m.%Y")))print('The maintainer is:{}'.format(pipeline.maintainer.id))The generated code for this schema can be downloaded forC++,GoandPython.UsageMapry provides a single point-of-entry for all code generation throughmapry-tocommand.To generate the code in different languages, invoke:ForC++:mapry-tocpp\--schema/path/to/schema.json\--outdir/path/to/cpp/codeForGo:mapry-togo\--schema/path/to/schema.json\--outdir/path/to/go/codeForPython:mapry-topy\--schema/path/to/schema.json\--outdir/path/to/py/codeIf the output directory does not exist, it will be created. Any existing
files will be silently overwritten.InstallationWe provide a prepackaged PEX file that can be readily downloaded and executed.
Please see theReleases section.If you prefer to use Mapry as a library (e.g., as part of your Python-based
build system), install it with pip:pip3installmapryContributingAll contributions are highly welcome. Please consult thispagein the documentation to see how you can contribute.VersioningWe followSemantic Versioning.
We extended the standard semantic versioning with an additional format version.
The version W.X.Y.Z indicates:W is the format version (data representation is backward-incompatible),X is the major version (library interface is backward-incompatible),Y is the minor version (library interface is extended, but
backward-compatible), andZ is the patch version (backward-compatible bug fix).Related ProjectsWe compiled an extensive list of related projects and how they compare to Mapry
in thedocumentation.We present here only the most prominent projects and their main differences
to Mapry:Standard JSON librariessupport only objecttrees, not graphs. They usually lack a schema (e.g.,json module in Python).De/serializers based on annotationshandle object graphs through custom logic (e.g.,Jackson in Java). Since they are
based on annotations in source code, a polyglot code base would require
a duplication across different languages which can be cumbersome and
error-prone to keep synchronized.Standard or widely used serialization librarieshandle object graphs well and provide a rich set of primitives. However, the serialization
format differs accross languagues (e.g.,Boost.Serialization in C++orPickle in Pythonwould need to be supported in Go).Popular serializers based on generated codeusually do not de/serialize object graphs, but only object trees (e.g.,Protocol BuffersorCap’n Proto).Flatbuffersbeing the exception
handle object graphs natively, but lack support for sophisticated types such as
maps and datetimes, durationsetc.