Package: pydeequ
PyDeequPyDeequ is a Python API forDeequ, a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets. PyDeequ is written to support usage of Deequ in Python.There are 4 main components of Deequ, and they are:Metrics Computation:Profilesleverages Analyzers to analyze each column of a dataset.Analyzersserve here as a foundational module that computes metrics for data profiling and validation at scale.Constraint Suggestion:Specify rules for various groups of Analyzers to be run over a dataset to return back a collection of constraints suggested to run in a Verification Suite.Constraint Verification:Perform data validation on a dataset with respect to various constraints set by you.Metrics RepositoryAllows for persistence and tracking of Deequ runs over time.🎉 Announcements 🎉NEW!!!1.1.0 release of Python Deequ has been published to PYPIhttps://pypi.org/project/pydeequ/. This release brings many recency upgrades including support up to Spark 3.3.0! Any feedbacks are welcome through github issues.With PyDeequ v0.1.8+, we now officially support Spark3 ! Just make sure you have an environment variableSPARK_VERSIONto specify your Spark version!We've release a blogpost on integrating PyDeequ onto AWS leveraging services such as AWS Glue, Athena, and SageMaker! Check it out:Monitor data quality in your data lake using PyDeequ and AWS Glue.Check out thePyDeequ Release Announcement Blogpostwith a tutorial walkthrough the Amazon Reviews dataset!Join the PyDeequ community onPyDeequ Slackto chat with the devs!QuickstartThe following will quickstart you with some basic usage. For more in-depth examples, take a look in thetutorials/directory for executable Jupyter notebooks of each module. For documentation on supported interfaces, view thedocumentation.InstallationYou can installPyDeequ via pip.pip install pydeequSet up a PySpark sessionfrompyspark.sqlimportSparkSession,Rowimportpydeequspark=(SparkSession.builder.config("spark.jars.packages",pydeequ.deequ_maven_coord).config("spark.jars.excludes",pydeequ.f2j_maven_coord).getOrCreate())df=spark.sparkContext.parallelize([Row(a="foo",b=1,c=5),Row(a="bar",b=2,c=6),Row(a="baz",b=3,c=None)]).toDF()Analyzersfrompydeequ.analyzersimport*analysisResult=AnalysisRunner(spark)\.onData(df)\.addAnalyzer(Size())\.addAnalyzer(Completeness("b"))\.run()analysisResult_df=AnalyzerContext.successMetricsAsDataFrame(spark,analysisResult)analysisResult_df.show()Profilefrompydeequ.profilesimport*result=ColumnProfilerRunner(spark)\.onData(df)\.run()forcol,profileinresult.profiles.items():print(profile)Constraint Suggestionsfrompydeequ.suggestionsimport*suggestionResult=ConstraintSuggestionRunner(spark)\.onData(df)\.addConstraintRule(DEFAULT())\.run()# Constraint Suggestions in JSON formatprint(suggestionResult)Constraint Verificationfrompydeequ.checksimport*frompydeequ.verificationimport*check=Check(spark,CheckLevel.Warning,"Review Check")checkResult=VerificationSuite(spark)\.onData(df)\.addCheck(check.hasSize(lambdax:x>=3)\.hasMin("b",lambdax:x==0)\.isComplete("c")\.isUnique("a")\.isContainedIn("a",["foo","bar","baz"])\.isNonNegative("b"))\.run()checkResult_df=VerificationResult.checkResultsAsDataFrame(spark,checkResult)checkResult_df.show()RepositorySave to a Metrics Repository by adding theuseRepository()andsaveOrAppendResult()calls to your Analysis Runner.frompydeequ.repositoryimport*frompydeequ.analyzersimport*metrics_file=FileSystemMetricsRepository.helper_metrics_file(spark,'metrics.json')repository=FileSystemMetricsRepository(spark,metrics_file)key_tags={'tag':'pydeequ hello world'}resultKey=ResultKey(spark,ResultKey.current_milli_time(),key_tags)analysisResult=AnalysisRunner(spark)\.onData(df)\.addAnalyzer(ApproxCountDistinct('b'))\.useRepository(repository)\.saveOrAppendResult(resultKey)\.run()To load previous runs, use therepositoryobject to load previous results back in.result_metrep_df=repository.load()\.before(ResultKey.current_milli_time())\.forAnalyzers([ApproxCountDistinct('b')])\.getSuccessMetricsAsDataFrame()Wrapping upAfter you've ran your jobs with PyDeequ, be sure to shut down your Spark session to prevent any hanging processes.spark.sparkContext._gateway.shutdown_callback_server()spark.stop()ContributingPlease refer to thecontributing docfor how to contribute to PyDeequ.LicenseThis library is licensed under the Apache 2.0 License.Contributing Developer SetupSetupSDKMANSetupJavaSetupApache SparkInstallPoetryRuntests locallySetup SDKMANSDKMAN is a tool for managing parallel Versions of multiple Software Development Kits on any Unix based
system. It provides a convenient command line interface for installing, switching, removing and listing
Candidates. SDKMAN! installs smoothly on Mac OSX, Linux, WSL, Cygwin, etc... Support Bash and ZSH shells. See
documentation on theSDKMAN! website.Open your favourite terminal and enter the following:$curl-shttps://get.sdkman.io|bash
IftheenvironmentneedstweakingforSDKMANtobeinstalled,
theinstallerwillpromptyouaccordinglyandaskyoutorestart.

Next,openanewterminalorenter:

$source"$HOME/.sdkman/bin/sdkman-init.sh"Lastly,runthefollowingcodesnippettoensurethatinstallationsucceeded:

$sdkversionSetup JavaInstall Java Now open favourite terminal and enter the following:ListtheAdoptOpenJDKOpenJDKversions
$sdklistjava

ToinstallForJava11$sdkinstalljava11.0.10.hs-adpt

ToinstallForJava11$sdkinstalljava8.0.292.hs-adptSetup Apache SparkInstall Java Now open favourite terminal and enter the following:ListtheApacheSparkversions:
$sdklistspark

ToinstallForSpark3$sdkinstallspark3.0.2PoetryPoetryCommandspoetryinstall

poetryupdate# --tree: List the dependencies as a tree.# --latest (-l): Show the latest version.# --outdated (-o): Show the latest version but only for packages that are outdated.poetryshow-oRunning Tests LocallyTake a look at tests intests/dataqualityandtests/jobs$poetryrunpytest