Package: pandantic
pandanticpandanticintroduces the ability to validate (pandas) DataFrames usingpydantic.BaseModels. Thepandanticpackage is using the V2 version ofpydanticas it has significant improvements over its V1 versions (a performance increase up to 50 times).First, installpandanticby using pip (or any other package managing tool).pip install pandanticparse_dfTo validatepd.DataFrames using PydanticBaseModels make sure to import theBaseModelclass from thepandanticpackage.from pandantic import BaseModelThepandantic.BaseModelsubclasses the originalpydantic.BaseModelwhich means thepandantic.BaseModelincludes all functionality from the originalpydantic.BaseModelbut it adds theparse_dfclass method which should be used to parse DataFrames.A quick exampleEnough of the talking, lets just make things easier by showing a very minor but quick example. Make sure to import theBaseModelclass frompandanticand create a schema like we normally would when usingpydantic.from pydantic.types import StrictInt

from pandantic import BaseModel


class DataFrameSchema(BaseModel):
    """Example schema for testing."""

    example_str: str
    example_int: StrictIntLet's try this schema on a simplepandas.DataFrame. Use the class methodparse_dffrom the freshly definedDataFrameSchemaand specify thedataframethat should be validated using the arguments of the method. In this example, we want tofilterout the bad records (there are more options like the good oldraiseto raise a ValueError after validating the whole DataFrame). In this case, only the second record would be kept in the returned DataFrame.df_invalid = pd.DataFrame(
    data={
        "example_str": ["foo", "bar", 1],
        "example_int": ["1", 2, 3.0],
    }
)

df_filtered = DataFrameSchema.parse_df(
    dataframe=df_invalid,
    errors="filter",
)Custom validatorsOne of the great features of Pydantic is the ability to create custom validators. Luckily, those custom validators will also work when parsing DataFrames usingpandantic. Make sure to import the original decorator from thepydanticpackage and keep in mind thatpandanticis using the V2 of Pydantic (sofield_validationit is). In the example below theBaseModelwill validate theexample_intfield and makes sure it is an even number.from pydantic import ValidationError, field_validator


class DataFrameSchema(BaseModel):
    """Example schema for testing."""

    example_str: str
    example_int: int

    @field_validator("example_int")
    def validate_even_integer(  # pylint: disable=invalid-name, no-self-argument
        cls, x: int
    ) -> int:
        """Example custom validator to validate if int is even."""
        if x % 2 != 0:
            raise ValidationError(f"example_int must be even, is {x}.")
        return xBy setting theerrorsargument toraise, the code will raise an ValueError after validating every row as the first row contains an uneven number.example_df_invalid = pd.DataFrame(
    data={
        "example_str": ["foo", "bar", "baz"],
        "example_int": [1, 4, 12],
    }
)

df_raised_error = DataFrameSchema.parse_df(
    dataframe=example_df_invalid,
    errors="raise",
)Special fields and typesOptionalAs the DataFrame is being parsed into a dict, aNonevalue is considered as ananvalue in cases there are different values in the dict. Therefore, specifyingOptionalcolumns (where the value can be empty) can be speciyfied by using the custompandantic.Optionaltype. This type is a replacement fortyping.Optional.from pandantic import BaseModel, Optional

# GIVEN
class Model(BaseModel):
    a: Optional[int] = None
    b: int

df_example = pd.DataFrame({"a": [1, None, 2], "b": ["str", 2, 3]})

# WHEN
df_filtered = Model.parse_df(df_example, errors="filter", verbose=True)DocsDocumentation can be foundhere