Package: serde
Serde is a lightweight, general-purpose framework for defining, serializing,
deserializing, and validating data structures in Python.Getting startedInstallationIntroductionBasic usageModelsSubclassed modelsModel taggingInternally taggedExternally taggedAdjacently taggedAbstract modelsCustom tagsFieldsModel states and processesLicenseGetting startedInstallationSerde is available on PyPI, you can install it usingpipinstallserdeExtended features can be installed with theextfeature.pipinstallserde[ext]IntroductionIn Serdemodelsare containers forfields. Data structures are defined by
subclassingModeland assigningFieldinstances as class annotations.
These fields handle serialization, deserialization, normalization, and
validation for the corresponding model attributes.fromdatetimeimportdatefromserdeimportModel,fieldsclassArtist(Model):name:fields.Str()classAlbum(Model):title:fields.Str()release_date:fields.Optional(fields.Date)artist:fields.Nested(Artist)album=Album(title='Dangerously in Love',release_date=date(2003,6,23),artist=Artist(name='Beyoncé'))assertalbum.to_dict()=={'title':'Dangerously in Love','release_date':'2003-06-23','artist':{'name':'Beyoncé'}}album=Album.from_json("""{
    "title": "Lemonade",
    "artist": {"name": "Beyoncé"}}"
""")assertalbum==Album(title='Lemonade',artist=Artist(name='Beyoncé'))Basic usageBelow we create aUsermodel by subclassingModeland adding thenameandemailfields.>>>fromdatetimeimportdatetime>>>fromserdeimportModel,fields>>>>>>classUser(Model):...name:fields.Str(rename='username')...email:fields.Email()The corresponding attribute names are used to instantiate the model object and
access the values on the model instance.>>>user=User(name='Linus Torvalds',email='torvalds@linuxfoundation.org')>>>user.name'Linus Torvalds'>>>user.email'torvalds@linuxfoundation.org'Models are validated when they are instantiated and aValidationErroris
raised if you provide invalid values.>>>User(name='Linus Torvalds',email='not an email')Traceback(mostrecentcalllast):...serde.exceptions.ValidationError:{'email':'invalid email'}Models are serialized into primitive Python types using theto_dict()method
on the model instance.>>>user.to_dict()OrderedDict([('username','Linus Torvalds'),('email','torvalds@linuxfoundation.org')])Or to JSON using theto_json()method.>>>user.to_json()'{"username": "Linus Torvalds", "email": "torvalds@linuxfoundation.org"}'Models are also validated when they are deserialized. Models are deserialized
from primitive Python types using the reciprocalfrom_dict()class method.>>>user=User.from_dict({...'username':'Donald Knuth',...'email':'noreply@stanford.edu'...})Or from JSON using thefrom_json()method.>>>user=User.from_json('''{
...     "username": "Donald Knuth",
...     "email": "noreply@stanford.edu"
... }''')Attempting to deserialize invalid data will result in aValidationError.>>>User.from_dict({'username':'Donald Knuth'})Traceback(mostrecentcalllast):...serde.exceptions.ValidationError:{'email':"missing data, expected field 'email'"}ModelsModels can be nested and used in container-like fields.  Below we create aBlogwith an author and a list of subscribers which must all beUserinstances.>>>classBlog(Model):...title:fields.Str()...author:fields.Nested(User)...subscribers:fields.List(User)When instantiating you have to supply instances of the nested models.>>>blog=Blog(...title="sobolevn's personal blog",...author=User(name='Nikita Sobolev',email='mail@sobolevn.me'),...subscribers=[...User(name='Ned Batchelder',email='ned@nedbatchelder.com')...]...)Serializing aBlogwould serialize the entire nested structure.>>>print(blog.to_json(indent=2)){"title":"sobolevn's personal blog","author":{"username":"Nikita Sobolev","email":"mail@sobolevn.me"},"subscribers":[{"username":"Ned Batchelder","email":"ned@nedbatchelder.com"}]}Similiarly deserializing aBlogwould deserialize the entire nested
structure, and create instances of all the submodels.Subclassed modelsModels can be subclassed. The subclass will have all the fields of the parent
and any additional ones. Consider the case where we define aSuperUsermodel
which is a subclass of aUser. Simply aUserthat has an extralevelfield.>>>classSuperUser(User):...# inherits name and email fields from User...level:fields.Choice(['admin','read-only'])We instantiate a subclassed model as normal by passing in each field value.>>>superuser=SuperUser(...name='Linus Torvalds',...email='torvalds@linuxfoundation.org',...level='admin'...)This is great for many cases, however, a commonly desired paradigm is to be able
to have theUser.from_dict()class method be able to deserialize aSuperUseras well. This can be made possible throughmodel tagging.Model taggingModel tagging is a way to mark serialized data in order to show that it is a
particularvariantof a model. Serde provides three types of model tagging,
but you can also define you own customTag. ATagcan be thought of in
the same way as aFieldbut instead of deserializing data into an attribute
on a model instance, it deserializes data into a model class.Internally taggedInternally tagged data stores a tag value inside the serialized data.Let us consider an example where we define aPetmodel with atag. We
can then subclass this model and deserialize arbitrary subclasses using the
tagged model.>>>fromserdeimportModel,fields,tags>>>>>>classPet(Model):...name:fields.Str()......classMeta:...tag=tags.Internal(tag='species')...>>>classDog(Pet):...hates_cats:fields.Bool()...>>>classCat(Pet):...hates_dogs:fields.Bool()We refer to theDogandCatsubclasses asvariantsofPet. When
serializing all parent model tag serialization is done after field
serialization.>>>Cat(name='Fluffy',hates_dogs=True).to_dict()OrderedDict([('name','Fluffy'),('hates_dogs',True),('species','__main__.Cat')])When deserializing, tag deserialization is done first to determine which model
to use for the deserialization.>>>milo=Pet.from_dict({...'name':'Milo',...'hates_cats':False,...'species':'__main__.Dog'...})>>>milo.__class__<class'__main__.Dog'>>>>milo.name'Milo'>>>milo.hates_catsFalseAn invalid or missing tag will raise aValidationError.>>>Pet.from_dict({'name':'Milo','hates_cats':False})Traceback(mostrecentcalllast):...serde.exceptions.ValidationError:missingdata,expectedtag'species'>>>>>>Pet.from_dict({'name':'Duke','species':'__main__.Horse'})Traceback(mostrecentcalllast):...serde.exceptions.ValidationError:novariantfoundExternally taggedExternally tagged data uses the tag value as a key and nests the content
underneath that key. All other processes behave similarly to the internally
tagged example above.>>>classPet(Model):...name:fields.Str()......classMeta:...tag=tags.External()...>>>classDog(Pet):...hates_cats:fields.Bool()...>>>Dog(name='Max',hates_cats=True).to_dict()OrderedDict([('__main__.Dog',OrderedDict([('name','Max'),('hates_cats',True)]))])Adjacently taggedAdjacently tagged data data stores the tag value and the content underneath two
separate keys. All other processes behave similarly to the internally tagged
example.>>>classPet(Model):...name:fields.Str()......classMeta:...tag=tags.Adjacent(tag='species',content='data')...>>>classDog(Pet):...hates_cats:fields.Bool()...>>>Dog(name='Max',hates_cats=True).to_dict()OrderedDict([('species','__main__.Dog'),('data',OrderedDict([('name','Max'),('hates_cats',True)]))])Abstract modelsBy default model tagging still allows deserialization of the base model. It is
common to have this model be abstract. You can do this by setting theabstractMeta field toTrue. This will make it uninstantiatable and it
won’t be included in the variant list during deserialization.>>>classFruit(Model):...classMeta:...abstract=True...>>>Fruit()Traceback(mostrecentcalllast):...TypeError:unabletoinstantiateabstractmodel'Fruit'Custom tagsIt is possible to create your own custom tag class by subclassing any oftags.External,tags.Internal,tags.Adjacentor even the basetags.Tag. This will allow customization of how the variants are looked up,
how the tag values are generated for variants, and how the data is serialized.Consider an example where we use a class attributecodeas the tag value.>>>classCustom(tags.Internal):...deflookup_tag(self,variant):...returnvariant.code...>>>classPet(Model):...name:fields.Str()......classMeta:...abstract=True...tag=Custom(tag='code')...>>>classDog(Pet):...code=1...hates_cats:fields.Bool()...>>>Dog(name='Max',hates_cats=True).to_dict()OrderedDict([('name','Max'),('hates_cats',True),('code',1)])>>>max=Pet.from_dict({'name':'Max','hates_cats':True,'code':1})>>>max.__class__<class'__main__.Dog'>>>>max.name'Max'>>>max.hates_catsTrueFieldsFields do the work of serializing, deserializing, normalizing, and validating
the input values. Fields are always assigned to a model asinstances, and
they support extra serialization, deserialization, normalization, and validation
of values without having to subclassField. For examplefromserdeimportModel,fields,validatorsclassAlbum(Model):title:fields.Str(normalizers=[str.strip])released:fields.Date(rename='release_date',validators=[validators.Min(datetime.date(1912,4,15))])In the above example we define anAlbumclass. Thetitlefield is of
typestr, and we apply thestr.stripnormalizer to automatically strip
the input value when instantiating or deserializing theAlbum. Thereleasedfield is of typedatetime.dateand we apply an extra validator
to only accept dates after 15th April 1912. Note: therenameargument only
applies to the serializing and deserializing of the data, theAlbumclass
would still be instantiated usingAlbum(released=...).If these methods of creating customFieldclasses are not satisfactory, you
can always subclass aFieldand override the relevant methods.>>>classPercent(fields.Float):...defvalidate(self,value):...super().validate(value)...validators.Between(0.0,100.0)(value)Model states and processesIn Serde, there are two states that the data can be in:Serialized dataModel instanceThere are five different processes that the data structure can go through when
moving between these two states.Deserialization happens when you create a model instance from a serialized
version usingfrom_dict()or similar.Instantiation happens when you construct a model instance in Python using the__init__()constructor.Normalization happens after instantiation and after deserialization. This is
usually a way to transform things before they are validated. For example: this
is where anOptionalfield sets default values.Validation is where the model and fields values are validated. This happens
after normalization.Serialization is when you serialize a model instance to a supported
serialization format usingto_dict()or similar.The diagram below shows how the stages (uppercase) and processes (lowercase) fit
in with each other.+---------------+
                    | Instantiation |
                    +---------------+
                            |
                            v
+---------------+   +---------------+
|Deserialization|-->| Normalization |
+---------------+   +---------------+
        ^                   |
        |                   v
        |           +---------------+
        |           |   Validation  |
        |           +---------------+
        |                   |
        |                   v
+-------+-------+   +---------------+
|SERIALIZED DATA|   | MODEL INSTANCE|
+---------------+   +---------------+
        ^                   |
        |                   |
+-------+-------+           |
| Serialization |<----------+
+---------------+LicenseSerde is licensed under either ofApache License, Version 2.0 (LICENSE-APACHEorhttps://www.apache.org/licenses/LICENSE-2.0)MIT License (LICENSE-MITorhttps://opensource.org/licenses/MIT)at your option.