Package: pandera-report
Pandera Extension for row-based reporting🚀 Descriptionpanderaprovides a flexible and expressive API for performing data
validation on dataframe-like objects to make data processing pipelines more
readable and robustIf you have to report potential quality issues resulting from the dataframe validation viapandera, thanpandera-reportis your friend. Based on the information of possible validation issues that pandera provides, your original dataframe will be extended with these issues on a row-level base.Withpandera-report, you can:Seamlessly integrates with thepanderalibrary to provide enhanced data validation capabilities without interfering with the pandera functionality.Provides a convenient way to enrich your data with information about why specific rows failed validation.⚡ SetupUsing pip:pipinstallpandera-reportUsing poetry:poetryaddpandera-reportQuick startThe following example is taken from thepanderadocumentation and shows the definition of a DataFrameSchema which will end in a valid result for the provided dataframe.importpandasaspdimportpanderaaspa# data to validatedf=pd.DataFrame({"column1":[1,4,0,10,9],"column2":[-1.3,-1.4,-2.9,-10.1,-20.4],"column3":["value_1","value_2","value_3","value_2","value_1"]})# define schemaschema=pa.DataFrameSchema({"column1":pa.Column(int,checks=pa.Check.le(10)),"column2":pa.Column(float,checks=pa.Check.lt(-1.2)),"column3":pa.Column(str,checks=[pa.Check.str_startswith("value_"),# define custom checks as functions that take a series as input and# outputs a boolean or boolean Seriespa.Check(lambdas:s.str.split("_",expand=True).shape[1]==2)]),})validated_df=schema(df)print(validated_df)#     column1  column2  column3#  0        1     -1.3  value_1#  1        4     -1.4  value_2#  2        0     -2.9  value_3#  3       10    -10.1  value_2#  4        9    -20.4  value_1To make usage of thepandera-reportfunctionality for the same schema and dataframe, you can do this:validator=DataFrameValidator()# default is quality_report=True, lazy=Trueprint(validator.validate(schema,df))#     column1  column2  column3 quality_issues quality_status#  0        1     -1.3  value_1           None          Valid#  1        4     -1.4  value_2           None          Valid#  2        0     -2.9  value_3           None          Valid#  3       10    -10.1  value_2           None          Valid#  4        9    -20.4  value_1           None          ValidYou see?! Same result but extended by the fact that the validation of the dataframe was completely valid. This can also be deactivated for the case that everything is 100% valid.But what if the dataframe contains data quality issues?panderawill throw SchemaErrors or SchemaError (depends on the lazyness). Let's see whatpandera-reportdoes, if we change the dataframe against the schema definition:# data to validatedf=pd.DataFrame({"column1":[1,4,0,10,9],"column2":[-1.3,-1.4,-2.9,-10.1,-20.4],"column3":["value_1","value_2","value_3","value_2","value1"]})validator=DataFrameValidator()print(validator.validate(schema,df))#     column1  column2  column3                              quality_issues quality_status#  0        1     -1.3  value_1                                        None          Valid#  1        4     -1.4  value_2                                        None          Valid#  2        0     -2.9  value_3                                        None          Valid#  3       10    -10.1  value_2                                        None          Valid#  4        9    -20.4   value1  Column <column3>: str_startswith('value_')        InvalidWhy is this useful? Quite simply, it becomes particularly interesting when you are not the one who has to prepare a valid file so that it can be processed into a valid DataFrame in the end.