8 Reasons to Start Using Pydantic to Improve Data Parsing and Validation | by Ahmed Besbes | Towards Data Science

In one of my previous posts, I looked at dataclasses as a way of writing python classes that act as data containers.
I found the dataclasses project amazing: it implements a lot of nifty methods under the hood to prevent you from writing boilerplate code, it provides a concise and pythonic syntax and it enforces modern programming practices.
However, an important feature this module misses is data validation: a process by which you enforce schema constraints on your data, at runtime.
This is where the Pydantic project comes into play.
In this post, we’ll introduce data validation and why you should think about it when developing Python applications. Then, we’ll introduce the Pydantic libray as a way of solving this problem.
We’ll go through 8 of its useful features and see how you can quickly embed them into your apps with a few lines of code.
As always: less talk, more code. Let’s have a look 🔍
Data validation is a process that makes your data compliant with a set of rules, schemas or constraints that you defined for each attribute. This makes your code ingest and return data in the exact way it was expected to.
Data validation prevents unexpected errors that occur due to problems such as malformed user inputs. In that sense, it also acts as a sanitization process.
Let’s have an example: Imagine that you’re building a credit scoring API to evaluate the creditworthiness of an individual.
To make this API work, you’ll have to send a post request on a given URL and provide some payload data.
This data defines an individual: some of its fields are required and some others are optional. But this doesn't stop here.
For some reason, this individual must have a French tax id as part of the required fields. This identifier follows a certain schema: let’s say, hypothetically, that it must have 13 digits and must end with 2 uppercase letters.
The individual must provide a complete address as well: it must have a street name, a number and a zip code. The zip code must be composed of 5 digits.
This can go on and on but you see the gist.
👉 Data validation makes sure the data you send to the API follows these constraints.
You can still make your data follow these constraints by loading it and applying a series of conditions to each field. This could work but it can quickly result in a lot of code that becomes unmaintainable over time.
What if we could encapsulate the data into a class, create a typed attribute for each of its fields and validate the field constraints at runtime when the data is loaded into the class?
Well, Pydantic could be the exact answer to this problem.
“Pydantic is a library that provides data validation and settings management using type annotations.” — Pydantic official documentation.
I like to think of Pydantic as the little salt you sprinkle over your food (or in this particular case, your codebase) to make it taste better: Pydantic doesn’t care about the way you do things. It’s more of an abstraction layer that you add to your code without changing its logic.
This layer, as you may expect, will essentially handle the data parsing and validation, among other cool features.
Let’s now go through 8 of Pydantic features to understand why it’s so useful.
You can define your data inside a class that inherits from theBaseModel class.
Pydantic models are structures that ingest the data, parse it and make sure it conforms to the fields’ constraints defined in it.
Let’s start with a simple example, where we define a Personclass that has two fields without any constraints: first_name and last_name .
Just like dataclasses, we use type annotations.
Unlike dataclasses, we don’t use the @dataclass decorator. We inherit from the BaseModel class instead
By calling the typingmodule, we can add fields with more complex types.
And you can even create types that are themselves a BaseModel class.
If you want to see how Pydantic can be coupled with popular libraries such as FastAPI, have a look at my previous post on this topic.
What happens when you define a Pydantic model and pass it some data that doesn’t conform to the defined schema?
To understand how Pydantic would behave in this particular case, let’s look at this simple model:
Let’s now feed it some inconsistent data and see what error messages are returned.
Let’s omit the age field. This immediately throws a ValidationError that points out to the specific missing field.
Instead of passing an integer to the age field, let’s pass it a string instead: for example"30 years"instead of 30 .
Similarly, this throws a ValidationError and the message explicitly states that the field expects an integer type.
👉 It’s worth mentioning that Pydantic always tries to coerce the type you annotated. For example, if you try to pass “30” to the agefield, this would still work even though this field expects an integer value. Pydantic handles this situation off the shelf.
👉 You can make Pydantic error messages more explicit by importing the ValidationError class and calling it inside a try / except statement.
Pydantic integrates well with modern IDEs such as VSCode and PyCharm. This helps you quickly debug your code and avoid silly mistakes.
For example, when you instantiate an object from a Pydantic model, you immediately have autocompletion.
You also have linting, assuming you use a static type checker like mypy.
In the following example, if you try to apply the len function to the age attribute, VSCode will signal an error through mypy.
Pydantic allows you to natively add some validation on each field by wrapping it inside the Field class.
For example:
you can add constraints on the length of the string fields by using the Field’s max_lengthand min_length arguments
you can set boundaries on the numerical fields by using the Field’s ge and le arguments. (ge: greater or equal, le: lower or equal).
Let’s have a look at the following example where we add constraints on the first_name (length between 2 and 20) and age (value lower than 150).
Let’s see what errors are returned if we try to pass the following data to the Person class.
data = {"first_name": "a", "last_name": "Besbes", "age": 200}
You can perform more advanced fields’ customisation. Here are some arguments you can use inside the Field class
regex : this adds a regular expression validator. This is useful when you want some string values to match a specific pattern
multiple_of : this applies to int fields. It adds a “multiple of” validator
max_items and min_items : this applies to lists and imposes a constraint on the number of items contained in them
allow_mutation : this applies to any type of field. It defaults to False. When set to True, it makes the field immutable (or protected).
To learn more about the large possibilities of Pydantic Field customisation, have a look at this link from the documentation.
You don’t have to reinvent the wheel. Pydantic provides you with many helper functions and methods that you can use. Let’s see some examples:
Starting from the following code,
Let’s create a Personobject:
data = {"first_name": "Ahmed", "last_name": "Besbes", "age": 30}person = Person(**data)
This object has access to many useful methods.
dict() : returns a dictionary from the object
json() : returns a dictionary in JSON format
copy() : returns a copy of the object
schema() : print out the schema in JSON
To learn more about helper functions, have a look at this link.
str , int , float , Listare the usual types that we work with.
In some situations, however, we may work with values that need specific validations such as paths, email addresses, IP addresses, to name a few.
Fortunately, Pydatnic provides a list of built-in types for many of these use cases and got you covered:
FilePath: for parsing file paths
DirectoryPath: for parsing directory paths
EmailStr: for parsing email addresses
Color: for parsing HTML colors (see color type)
HttpUrl: for parsing strict HTTP URLs
IPvAnyAddress: for parsing IPv4 and IPv6 addresses
Pydantic allows you to write your own custom validator.
Let’s say that we want to add PhoneNumber field to the previous example.
We want this field to respect two constraints:
It’s a string of 10 digits
It must start with 0
To perform a custom validation, you’ll have to import validator function from Pydantic and use it as a decorator before the function that tests the field value.
Of course, you can do more sophisticated validations. Have a look at this link to learn more.
Pydantic allows you to read environment variables from .env files and parse them directly inside BaseSettings class.
To do this, you first need to install python-dotenv.
(To learn more about python-dotenv and environment variables, you can refer to my previous post on the topic)
Let’s say that you have some environment variables inside this .env file:
LOGIN=AhmedAPI_KEY=SeCR€t!SEED=42
To make Pydatnic load these variables, we’ll first have to define a Settings class that inherits from the BaseSettings class.
Inside the Settings class, we’ll define the variables that are listed in the .env file, while adding types and validators.
And finally, we specify that the environment variables should be read from a .env file.
If we run the code, the settings are printed to the terminal.
If we replace 42 by forty-two inside the .env file, here’s what we’d get:
Pretty cool, right?
Pydantic is a fantastic tool that I really enjoyed learning and using.
If you too are interested in learning more about this library, here’s a curated list of handpicked resources (blog posts and Youtube videos) I went through.
Pydantic official documentation
https://dev.to/tiangolo/the-future-of-fastapi-and-pydantic-is-bright-3pbm
https://dev.to/amal/pydantic-for-fastapi-2385
https://dev.to/nazliander/using-pydantic-as-a-parser-and-data-validation-tool-51n3
https://dev.to/izabelakowal/some-ideas-on-how-to-implement-dtos-in-python-be3
(a French blog post): https://ichi.pro/fr/le-guide-du-debutant-sur-pydantic-204731199577737
A great Pydantic introduction from the ArjanCodes Youtube channel (a shout out to this guy who does amazing videos on programming)
Pydantic: Modern Python Data Validation and Settings by Michael Kennedy — An amazing video tutorial
If you’ve stuck till the end, I really thank you for your time and hope that you found this post useful to get started using Pydantic.
If you don’t use Pydantic and come to know a better alternative, let me know about it in the comments 📥
That’ll be all for me today. Until next time for more programming tips and tutorials. 👋