Package: pandera
The Open-source Framework for Precision Data Testing📊 🔎 ✅Data validation for scientists, engineers, and analysts seeking correctness.panderais aUnion.aiopen
source project that provides a flexible and expressive API for performing data
validation on dataframe-like objects to make data processing pipelines more readable and robust.Dataframes contain information thatpanderaexplicitly validates at runtime.
This is useful in production-critical or reproducible research settings. Withpandera, you can:Define a schema once and use it to validatedifferent dataframe typesincludingpandas,dask,modin, andpyspark.Checkthe types and
properties of columns in aDataFrameor values in aSeries.Perform more complex statistical validation likehypothesis testing.Seamlessly integrate with existing data analysis/processing pipelines
viafunction decorators.Define dataframe models with theclass-based APIwith pydantic-style syntax and validate dataframes using the typing syntax.Synthesize datafrom schema objects for property-based testing with pandas data structures.Lazily Validatedataframes so that all validation checks are executed before raising an error.Integratewith
a rich ecosystem of python tools likepydantic,fastapi, andmypy.DocumentationThe official documentation is hosted here:https://pandera.readthedocs.ioInstallUsing pip:pip install panderaUsing conda:conda install -c conda-forge panderaExtrasInstalling additional functionality:pippipinstallpandera[hypotheses]# hypothesis checkspipinstallpandera[io]# yaml/script schema io utilitiespipinstallpandera[strategies]# data synthesis strategiespipinstallpandera[mypy]# enable static type-linting of pandaspipinstallpandera[fastapi]# fastapi integrationpipinstallpandera[dask]# validate dask dataframespipinstallpandera[pyspark]# validate pyspark dataframespipinstallpandera[modin]# validate modin dataframespipinstallpandera[modin-ray]# validate modin dataframes with raypipinstallpandera[modin-dask]# validate modin dataframes with daskpipinstallpandera[geopandas]# validate geopandas geodataframescondacondainstall-cconda-forgepandera-hypotheses# hypothesis checkscondainstall-cconda-forgepandera-io# yaml/script schema io utilitiescondainstall-cconda-forgepandera-strategies# data synthesis strategiescondainstall-cconda-forgepandera-mypy# enable static type-linting of pandascondainstall-cconda-forgepandera-fastapi# fastapi integrationcondainstall-cconda-forgepandera-dask# validate dask dataframescondainstall-cconda-forgepandera-pyspark# validate pyspark dataframescondainstall-cconda-forgepandera-modin# validate modin dataframescondainstall-cconda-forgepandera-modin-ray# validate modin dataframes with raycondainstall-cconda-forgepandera-modin-dask# validate modin dataframes with daskcondainstall-cconda-forgepandera-geopandas# validate geopandas geodataframesQuick Startimportpandasaspdimportpanderaaspa# data to validatedf=pd.DataFrame({"column1":[1,4,0,10,9],"column2":[-1.3,-1.4,-2.9,-10.1,-20.4],"column3":["value_1","value_2","value_3","value_2","value_1"]})# define schemaschema=pa.DataFrameSchema({"column1":pa.Column(int,checks=pa.Check.le(10)),"column2":pa.Column(float,checks=pa.Check.lt(-1.2)),"column3":pa.Column(str,checks=[pa.Check.str_startswith("value_"),# define custom checks as functions that take a series as input and# outputs a boolean or boolean Seriespa.Check(lambdas:s.str.split("_",expand=True).shape[1]==2)]),})validated_df=schema(df)print(validated_df)#     column1  column2  column3#  0        1     -1.3  value_1#  1        4     -1.4  value_2#  2        0     -2.9  value_3#  3       10    -10.1  value_2#  4        9    -20.4  value_1DataFrame Modelpanderaalso provides an alternative API for expressing schemas inspired
bydataclassesandpydantic. The equivalentDataFrameModelfor the aboveDataFrameSchemawould be:frompandera.typingimportSeriesclassSchema(pa.DataFrameModel):column1:int=pa.Field(le=10)column2:float=pa.Field(lt=-1.2)column3:str=pa.Field(str_startswith="value_")@pa.check("column3")defcolumn_3_check(cls,series:Series[str])->Series[bool]:"""Check that values have two elements after being split with '_'"""returnseries.str.split("_",expand=True).shape[1]==2Schema.validate(df)Development Installationgit clone https://github.com/pandera-dev/pandera.git
cd pandera
export PYTHON_VERSION=...  # specify desired python version
pip install -r dev/requirements-${PYTHON_VERSION}.txt
pip install -e .Testspip install pytest
pytest testsContributing to panderaAll contributions, bug reports, bug fixes, documentation improvements,
enhancements and ideas are welcome.A detailed overview on how to contribute can be found in thecontributing guideon GitHub.IssuesGohereto submit feature
requests or bugfixes.Need Help?There are many ways of getting help with your questions. You can ask a question
onGithub Discussionspage or reach out to the maintainers and pandera community onDiscordWhypandera?dataframe-centric data types,column nullability,
anduniquenessare first-class concepts.Definedataframe modelswith the class-based API withpydantic-style syntax and validate dataframes using the typing syntax.check_inputandcheck_outputdecoratorsenable seamless integration with existing code.Checksprovide flexibility and performance by providing access topandasAPI by design and offers built-in checks for common data tests.Hypothesisclass provides a tidy-first interface for statistical hypothesis
testing.Checks andHypothesisobjects support bothtidy and wide data validation.Use schemas as generative contracts tosynthesize datafor unit testing.Schema inferenceallows you to bootstrap schemas from data.How to CiteIf you usepanderain the context of academic or industry research, please
consider citing thepaperand/orsoftware package.Paper@InProceedings{ niels_bantilan-proc-scipy-2020,
  author    = { {N}iels {B}antilan },
  title     = { pandera: {S}tatistical {D}ata {V}alidation of {P}andas {D}ataframes },
  booktitle = { {P}roceedings of the 19th {P}ython in {S}cience {C}onference },
  pages     = { 116 - 124 },
  year      = { 2020 },
  editor    = { {M}eghann {A}garwal and {C}hris {C}alloway and {D}illon {N}iederhut and {D}avid {S}hupe },
  doi       = { 10.25080/Majora-342d178e-010 }
}Software PackageLicense and Creditspanderais licensed under theMIT licenseand is written and
maintained by Niels Bantilan (niels@pandera.ci)