Package: hooqu
Documentation:https://hooqu.readthedocs.ioSource Code:https://github.com/mfcabrera/hooquInstallHooqu requires Pandas >= 1.0 and Python >= 3.7. To install via pip use:pip install hooquQuick Startimportpandasaspd# data to validatedf=pd.DataFrame([(1,"Thingy A","awesome thing.","high",0),(2,"Thingy B","available at http://thingb.com",None,0),(3,None,None,"low",5),(4,"Thingy D","checkout https://thingd.ca","low",10),(5,"Thingy E",None,"high",12),],columns=["id","productName","description","priority","numViews"])Checks we want to perform:there are 5 rows in totalvalues of the id attribute are never Null/None and uniquevalues of the productName attribute are never null/Nonethe priority attribute can only contain “high” or “low” as valuenumViews should not contain negative valuesat least half of the values in description should contain a urlthe median of numViews should be less than or equal to 10In code this looks as follows:fromhooqu.checksimportCheck,CheckLevel,CheckStatusfromhooqu.verification_suiteimportVerificationSuitefromhooqu.constraintsimportConstraintStatusverification_result=(VerificationSuite().on_data(df).add_check(Check(CheckLevel.ERROR,"Basic Check").has_size(lambdasz:sz==5)# we expect 5 rows.is_complete("id")# should never be None/Null.is_unique("id")# should not contain duplicates.is_complete("productName")# should never be None/Null.is_contained_in("priority",("high","low")).is_non_negative("numViews")# .contains_url("description", lambda d: d >= 0.5) (NOT YET IMPLEMENTED).has_quantile("numViews",0.5,lambdav:v<=10)).run())After callingrun, hooqu will compute some metrics on the data. Afterwards it invokes your assertion functions
(e.g.,lambda sz: sz == 5for the size check) on these metrics to see if the constraints hold on the data.We can inspect theVerificationResultto see if the test found errors:ifverification_result.status==CheckStatus.SUCCESS:print("Alles klar: The data passed the test, everything is fine!")else:print("We found errors in the data")forcheck_resultinverification_result.check_results.values():forcrincheck_result.constraint_results:ifcr.status!=ConstraintStatus.SUCCESS:print(f"{cr.constraint}:{cr.message}")If we run the example, we get the following output:We found errors in the data
CompletenessConstraint(Completeness(productName)): Value 0.8 does not meet the constraint requirement.The test found that our assumptions are violated! Only 4 out of 5 (80%) of the values of the productName attribute are non-null.
Fortunately, we ran a test and found the errors, somebody should immediately fix the data :)ContributingAll contributions, bug reports, bug fixes, documentation improvements,
enhancements and ideas are welcome.  Please useGitHub issues: for bug reports,
feature requests, install issues, RFCs, thoughts, etc.See the fullcotributing guidefor more information.Why Hooqu?Easy to use declarative API to add data verification steps to your
data processing pipeline.TheVerificationResultallows you know not only what check fail
but the values of the computed metric, allowing for flexible
handling of issues with the data.Incremental metric computation capability allows to compare quality
metrics change across time (planned).Support for storing and loading computed metrics (planned).ReferencesThis project is a “spiritual” port ofApache Deequand thus tries to implement
the declarative API described on the paper “Automating large-scale data quality verification”
while trying to remain pythonic as much as possible. This project does not use (py)Spark but rather
Pandas (and hopefully in the future it will support other compatible dataframe implementations).NameJukumari (pronounced hooqumari) is the Aymara name for thespectacled bear(Tremarctos ornatus), also known as the Andean
bear, Andean short-faced bear, or mountain bear.ChangelogAll notable changes to this project will be documented in this file.The format is based onKeep a
Changelog, and this project
adheres toSemantic
Versioning.[0.1.0] - 2020-08-26Initial release. The following checks are available:has_completenesshas_maxhas_meanhas_minhas_quantilehas_sizehas_standard_deviationhas_sumhas_uniquenessis_completeis_contained_inis_contained_in_rangeis_non_negativeis_positiveis_unique