Package: vivqu
Vivqu: easier data analysis and visualization based on pydeequ1. BackgroundFeature quality is of great important to machine learing model performance, in order to insure the intactness of data that come from multiple upstream data sources, I devised a data quality checking library for post monitoring, which is namedvivqu.Based on PyDeequ, it gives data scientists deep insight into the data quality, and enable them to check whichever metrics they are insterested in. It also simplifies the intricate process when using PyDeequ directly and make users able to do the checking job within a few lines of code.Vivqu consists of several parts:QualityChecker: provides simplified interfaces to measure the quality of a dataframe.DataLoader: provides different ways to load data from multiple sources.Visualizer: provides visualization for one analysis result or show differences for two or more results.2. Requirementpython version >= 3.7pyspark version < 3.1, >= 2.4pydeequ version >= 1.0heatmapz == 0.0.4If you don't have a java runtime environment (jre) on your computer, please choose one fromLatest releases | Adoptium.3. InstallationThis project has been packaged and uploaded tovivqu · PyPI.You can install vivqu with pip, which will automatically download pyspark, pydeequ and matplotlib with appropriate version.pipinstallvivqu4. Example with Jupyter NotebookYou can follow this example in a jupyter notebook. Prepare a sample csv file, and let's start!Import pyspark, pydeequ and vivqu.importpysparkfrompyspark.sqlimportSparkSessionimportpydeequfrompydeequ.analyzersimport*importvivquImport matplotlib, change figure format to make it more clear.importmatplotlib.pyplotasplt# add this command if you want to show chinese characterplt.rcParams['font.sans-serif']=['Arial Unicode MS']%matplotlibinline# you can choose from png, svg and retina%configInlineBackend.figure_format='retina'Create a spark session locally, you can also use a remote spark session.spark_session=(SparkSession.builder.config("spark.jars.packages",pydeequ.deequ_maven_coord).config("spark.jars.excludes",pydeequ.f2j_maven_coord).getOrCreate())Import DataLoader to load data from csv file.fromvivqu.loaderimportDataLoaderloader=DataLoader(spark_session)df=loader.load_csv("transaction_100a.csv")df.printSchema()Create quality checker by providing it with a spark session.fromvivqu.checkerimport*checker=QualityChecker(spark_session)Assign default metrics to columns that you are interested, then start analyzing.Add column names in string format means that they are analyzed in default way, which contain:Completeness,CountDistinct,Histogramfor column ofCategorytype;Completeness,ApproxQuantiles,Maximum,Minimumfor column ofNumerictype.If you want to see the start and end date of this data frame, pass the column name of time information to analyze function.result_all_default=checker.analyze(df,["Transaction_reason","Beneficiary_bank_name","Original_amount_USD",],"Datetime_created")result_all_defaultYou can also add some other metrics and run.result_add_metric=checker.analyze(df,["Original_amount_USD",Size(),Completeness("Beneficiary_bank_name"),])result_add_metricTo visualize, you need to specify checker, dataframe, columns and date_column(Optional) parameter in visualize() function.fromvivqu.visualizeimport*visualize(checker,df,["Beneficiary_bank_name","Originator_category_code","Original_amount_USD",],"Datetime_created")Use diff() function to visualize specific columns between several dataframes.df1=loader.load_csv("transaction_100a.csv")df2=loader.load_csv("transaction_100b.csv")df3=loader.load_csv("transaction_100c.csv")diff(checker,[df1,df2],["Beneficiary_bank_name","Originator_category_code","Original_amount_USD",])If you passdate_columnparameter to diff(), the start and end date of each data frame will be shown.diff(checker,[df1,df2,df3],["Beneficiary_bank_name","Original_amount_USD","Transaction_reason",],"Datetime_created")You can use Verifier to set constraints and see. verification result on given dataframeverifier=Verifier(checker)result=verifier.verify(df1,[(Uniqueness(["ID"]),lambdax:x==1,"ID has no duplicate"),(Completeness("Booking_Legal_Entity_ID"),lambdax:x==1,"Booking_Legal_Entity_ID has no null value"),(Completeness("Transaction_reason"),lambdax:x==1,"Transaction_reason has no null value"),(Size(),lambdax:x>50,"Dataset size is larger than 50")])print(result)[True, True, False, True]You can calculate correlation matrix and visualize it.corr_mat_df=checker.corr(df,["Original_amount_USD","Original_amount_orig_ccy","Original_Inverse_USD","Normalized_amount_USD",])corr_mat_dfThe color indicate whether the correlation coefficient is postive or negtive, the size of each square indicate the absolute value of each coefficient.visualize_corr(checker,df,["Original_amount_USD","Original_amount_orig_ccy","Original_Inverse_USD","Normalized_amount_USD",])5. MetricsHere are the detailed explanations and examples of all metrics provided in pydeequ.Reference:https://aws.amazon.com/cn/blogs/big-data/test-data-quality-at-scale-with-deequ/MetricDescriptionExampleApproxCountDistinctApproximate number of distinct value, computed with HyperLogLogPlusPlus sketches.ApproxCountDistinct("review_id")ApproxQuantileApproximate quantile of a distribution.ApproxQuantile("star_rating", quantile = 0.5)ApproxQuantilesApproximate quantiles of a distribution.ApproxQuantiles("star_rating", quantiles = [0.25, 0.5, 0.75])CompletenessFraction of non-null values in a column.Completeness("review_id")CorrelationPearson correlation coefficient, measures the linear correlation between two columns. The result is in the range [-1, 1], where 1 means positive linear correlation, -1 means negative linear correlation, and 0 means no correlation.Correlation("total_votes", "star_rating")CountDistinctNumber of distinct values.CountDistinct("review_id")DistinctnessFraction of distinct values of a column over the number of all values of a column. Distinct values occur at least once. Example: [a, a, b] contains two distinct values a and b, so distinctness is 2/3.Distinctness("review_id")EntropyEntropy is a measure of the level of information contained in an event (value in a column) when considering all possible events (values in a column). It is measured in nats (natural units of information). Entropy is estimated using observed value counts as the negative sum of (value_count/total_count) * log(value_count/total_count). Example: [a, b, b, c, c] has three distinct values with counts [1, 2, 2]. Entropy is then (-1/5*log(1/5)-2/5*log(2/5)-2/5*log(2/5)) = 1.055.Entropy("star_rating")MaximumMaximum value.Maximum("star_rating")MeanMean value; null values are excluded.Mean("star_rating")MinimumMinimum value.Minimum("star_rating")MutualInformationMutual information describes how much information about one column (one random variable) can be inferred from another column (another random variable). If the two columns are independent, mutual information is zero. If one column is a function of the other column, mutual information is the entropy of the column. Mutual information is symmetric and nonnegative.MutualInformation(["total_votes", "star_rating"])SizeNumber of rows in a DataFrame.Size()SumSum of all values of a column.Sum("total_votes")UniqueValueRatioFraction of unique values over the number of all distinct values of a column. Unique values occur exactly once; distinct values occur at least once. Example: [a, a, b] contains one unique value b, and two distinct values a and b, so the unique value ratio is 1/2.UniqueValueRatio("star_rating")UniquenessFraction of unique values over the number of all values of a column. Unique values occur exactly once. Example: [a, a, b] contains one unique value b, so uniqueness is 1/3.Uniqueness("star_rating")