What, Why and How of (De)Serialization in Python | by Xiaoxu Gao | Towards Data Science

The first time I heard about the word serialization and deserialization, I had no idea what they mean and felt these terms quite “intimidating”. After a few days, I started to pick up this concept, but was confused by their correlation with encoding and decoding. In this article, I want to walk you through this concept as if you are a newbie. By the end of the article, you will find out that it’s not as difficult as you might think.
We create plenty of objects in Python every day, and these objects will eventually disappear if the program dies. Sometimes we want to store the state of an object in a file or in a database, or transmit it across the network for using it in the future. This helps us to reuse the object in different programs or even in different environments.
But the problem we might face is the compatibility. Imagine that you’ve created an object in Python, how could it be reused by a Java program? Boolean values in Python look like True and False, but in Java, they are represented using true and false. There must be a “middle man” creating a universal language that both programs understand — which is a sequence of bytes.
The process of converting a structured object into a sequence of bytes which can be stored in a file system or database or sent through the network is called serialization. The reversed process is called deserialization, which means reconverting a sequence of bytes into a structured object. Creating an object via deserialization is usually faster than initializing from a class.
Storing the state of an object in a file or database can save time to process huge datasets in many data science projects. For example, you only need to pre-process the dataset once and save the model into a disk. Later you just need to deserialize it and reuse the pre-cooked model as many times as you want. This is definitely preferred to pre-processing it each time. The side effect of it is that the serialized object might be too big to be loaded into memory, so several chunked objects are more preferred.
Besides, you can apply encryption and compression to your data stream to level up the security and boost your performance even more.
To serialize a complex data structure, a serialization format is needed. It converts the object into a certain format that can be stored and transmitted. There are two groups of serialization format: text-based and binary-based. As the name suggests, a text-based serialization is the process of serializing an object in a human-readable format, while binary-based is in a not human-readable format.
Encoding & Decoding
As stated in the introduction part, I didn’t really know the differences between encoding and serialization earlier in time. In short, encoding is a way of translating characters to bits using an encoding schema like UTF-8, while serialization is a long process that also stores the bits into a file, database or send them to the network. So I see encoding as a part of the serialization process. If you would like to learn more about Python encodings, I recommend you to read this thorough article from RealPython.
realpython.com
Back to the serialization format, typical text-based serialization formats are CSV, JSON, XML, YAML, TOML, etc. Binary-based formats are protobuf and Avro. Python also has several packages like pickle, numpy and pandas that supports serializing custom objects into byte format.
To let you feel the differences among several serialization formats, I use the same example throughout the article. Let’s say I have a class Article with attributes author, publication, publish_date, topics, word_count and is_vip. I want to store the state of many objects in a file and share it with a consumer.
CSV
CSV is probably the simplest serialization format. It stands for Comma-Separated Values. It is well suited to storing tabulated data (flat data). As you can see in this example, it doesn’t work very well with nested data structures. In the code, I first use asdict function from dataclasses to convert an Article object into a dictionary, then use csv.DictWriter to write that dictionary into a csv file.
Actually we have just finished a serialization process. How simple it is! At a very high level, serialization is nothing more than writing data to disk, while deserialization means reading data into memory.
But as you might notice that attribute topic which is supposed to be a list is represented as a string in csv file. This is due to the fact that csv only supports flat data, which means when reading the file, an extra step is required to convert string topic back to a list. Besides, csv file doesn’t store type information, so everything is interpreted as string, like attribute word_count. If I read article.csv file and print the row, I would get a dictionary object with every attribute in string format.
In conclusion, csv file can store the attributes and values of a flat object, but can’t always store the correct information of data type. To solve this problem, we need more flexible formats.
JSON
A good alternative is JSON. JSON is the most widespread format for data serialization due to its human-readable syntax, compatibility with almost every programming language, and support of nested data structures and multiple data types.
We can write the same article object into a JSON file using the built-in library json. dump() serializes an object into a JSON formatted string, and then write it to the file object. json uses JSONEncoder class to convert a Python type into a JSON type (e.g. True => true, False => false, None => null). It’s possible to extend JSONEncoder and serialize other types.
I would recommend another great article from RealPython on the topic of JSON data in Python.
realpython.com
On the other hand, the deserialization process would be simply reading the JSON file and recreate a dictionary object. As you can see, the recreated object is the same as the original one.
Not so surprisingly, JSON is a better choice than csv in this example because it holds more characteristics of the object than csv, thus is able to fully restore the object. But a potential problem with JSON is that you have to load the complete file into memory, while you can iterate the rows in a csv without giving too much pressure on memory.
Of course, you can create multiple chunks to relief the memory, but it’s just something you have to keep in mind.
Another format comparable to JSON is YAML. I’ve talked about their differences in this article. YAML also supports nested data structures and different data types.
XML
The next text-based format is XML, which is used for representing nested information in a tree-like structure. It’s widely used in Web and Service-Oriented Architecture. An XML file doesn’t include any type information, but you can standardize and validate an XML file through a schema, so-called XSD.
Python provides a built-in library xml.etree.ElementTree to help us create or read XML files, but the conversion from a dictionary to a tree element is not as straightforward as the previous formats. In this code example, I implement a simple dict2xml adapter myself.
Accordingly, its XSD file can look like this, where each element is bound to a type. In addition to the built-in data types, you can define your own data type with xs:restriction such as xs:maxLength, xs:pattern etc.
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">  <xs:element name="article">    <xs:complexType>      <xs:sequence>        <xs:element type="xs:string" name="author"/>        <xs:element type="xs:string" name="publication"/>        <xs:element type="xs:date" name="publish_date"/>        <xs:element type="xs:string" name="topics" maxOccurs="unbounded" minOccurs="0"/>        <xs:element type="xs:integer" name="word_count"/>        <xs:element type="xs:boolean" name="is_vip"/>      </xs:sequence>    </xs:complexType>  </xs:element></xs:schema>
Unfortunately, xml.etree.ElementTree doesn’t support XSD validation. A good alternative could be lxml.
Back to the example, let’s deserialize (read) the XML file. Due to the uncertainty of data types, every element value is interpreted as string in Python. iterparse from cElementTree returns an iterator, which is useful when you want to read a large XML file and don’t want to load everything in memory.
In conclusion, XML is a schema-based serialization format. But no matter what the type is in XSD file, Python always reads it as a string. Thus, a custom adapter is required to convert an XML data type to a Python data type.
BSON
So far, we’ve looked into 3 text-based serialization formats. Next, let’s look at binary-based serialization format. This format is not human-readable, but it can effectively compress the data, and is used a lot for intercommunication and storage in distributed systems.
BSON (binary JSON) is a binary-encoded serialization of JSON-like documents. Like JSON, BSON supports the embedding of documents and arrays. BSON also allows representation of data types that are not part of JSON spec.
BSON is mainly used in MongoDB database. When you install pymongo, bson is part of the library. The way to encode and decode using BSON is as straightforward as JSON.
A BSON record takes a bit more space than JSON is because it also contains the meta info of the record such as data type and length.
On the other hand, BSON provides a few advantages over using JSON:
BSON supports more types than JSON, for instance, bytes and date.BSON records are easy to traverse and query due to additional metadata. It allows value retrieval without reading the entire documents.
BSON supports more types than JSON, for instance, bytes and date.
BSON records are easy to traverse and query due to additional metadata. It allows value retrieval without reading the entire documents.
Avro
Another famous binary-based serialization format is Avro. Avro is a fast schema-based serialization format. A schema must be provided during serialization and deserialization. Due to its high-performance, it’s used most often with Apache Kafka.
Avro schema is defined in JSON format which makes it language independent and can be stored in nearly every environment. The schema describes records using a rich type systems that allows both data definition and documentations. By having schemas, it’s possible to model the data and create a catalogue of all the entities. Such catalogue can be served on a schema registry, making schemas available for consumption by consumers and producers.
Here is an example of Avro schema for Article entity.
Avro is supported in many programming languages, including Python. In this example, I’m using fast-avro, a CPython based library. It’s approved that fast-avro is way faster than the pure Python implementation library avro.
fast-avro provides writer/reader and schemaless_writer/schemaless_reader. The first option inserts the schema into .avro file as a file header, while the second doesn’t.
Pickle
Alright, after showing different serialization formats, I’d like to introduce you a couple of handy Python libraries that can serialize a Python object. One of them is Pickle.
“Pickling” is the process whereby a Python object hierarchy is converted into a byte stream, and “unpickling” is the inverse operation.
In terms of what can be pickled and unpickled, here is the list on Python doc.
The unpickled object belongs to the type Article and can be used right away.
However, as stated here, Pickle module is not secure due to the possibility of code injection. Tools like Bandit will not let it go through the security checks, so never unpickle objects from untrusted sources.
Pandas / Numpy
Famous data manipulation libraries like Pandas and Numpy also offer pickle-like functions. This can become handy if DataFrame or np.ndarry is agreed as a common data type between producers and consumers.
If you know more Python libraries that can serialize data into different formats, please leave a comment below to share with us.
Last but not least, let’s look at the performance of each serialization format/library. These tests were executed on a Macbook with 8-Core i9.
In this example, I want to write 100 article objects into a file, and then recreate these objects in a different function. In the end, I use assert to compare the recreated objects to the original objects. The full implementation can be found here.
The graph tells us:
pickle is the winner in this performance testing. It’s probably because an article object can be directly serialized without being converted to a JSON first.BSON performs better than JSON in both serialization and deserialization. The same for numpy v.s pandas.Deserialization of csv took more time is due to the steps to converting attributes from string to its original types (e.g. int or bool).As the only schema-based serialization format, the performance of avro is quite impressive.The slowness of XML can be blamed on the implementation. Please let me know how I can improve it. :)
pickle is the winner in this performance testing. It’s probably because an article object can be directly serialized without being converted to a JSON first.
BSON performs better than JSON in both serialization and deserialization. The same for numpy v.s pandas.
Deserialization of csv took more time is due to the steps to converting attributes from string to its original types (e.g. int or bool).
As the only schema-based serialization format, the performance of avro is quite impressive.
The slowness of XML can be blamed on the implementation. Please let me know how I can improve it. :)
First of all, thanks for finishing reading such a long article. Congratulations! You survived (I survived too)! :) In this article, we went through many possibilities of serializing and deserializing a Python object. Until now, you must not feel these 2 terms intimidating anymore, as to some extend, it’s just a fancy way to say reading and writing.
There are pros and cons on each serialization format. You should think about whether you want text-based or binary-based, schema-based or non-schema-based, what is your data format? dictionary, DataFrame or Python object? Do you want to save only the value or also the type? Do you care about the performance and memory?
I hope this article can help you make a smart decision. Please leave your comments below if you have any thought.