Package: cuallee-aws
cualleeMeaninggoodin Aztec (Nahuatl),pronounced: QUAL-EThis library provides an intuitiveAPIto describechecksinitially just forPySparkdataframesv3.3.0. And extended topandas,snowpark,duckdb, and more.
It is a replacement written in purepythonof thepydeequframework.I gave up indeequas after extensive use, the API is not user-friendly, the Python Callback servers produce additional costs in our compute clusters, and the lack of support to the newest version of PySpark.As resultcualleewas bornThis implementation goes in hand with the latest API from PySpark and uses theObservationAPI to collect metrics
at the lower cost of computation.
When benchmarking against pydeequ,cualleeuses circa <3k java classes underneath andremarkablyless memory.Supportcualleeis the data quality framework truly dataframe agnostic.ProviderAPIVersionssnowpark1.4.0pyspark3.4.0,3.3.x,3.2.xbigquery3.4.1pandas2.0.1,1.5.x,1.4.xduckdb0.7.1,0.8.0polars0.18.2Logos are trademarks of their own brands.StatsInstallpipinstallcualleeChecksThe most common checks for data integrity validations arecompletenessanduniquenessan example of this dimensions shown below:fromcualleeimportCheck,CheckLevel# WARN:0, ERR: 1# Nulls on column Idcheck=Check(CheckLevel.WARNING,"Completeness")(check.is_complete("id").is_unique("id").validate(df)).show()# Returns a pyspark.sql.DataFrameDatesPerhaps one of the most useful features ofcualleeis its extensive number of checks forDateandTimestampvalues. Including, validation of ranges, set operations like inclusion, or even a verification that confirmscontinuity on datesusing theis_dailycheck function.# Unique values on idcheck=Check(CheckLevel.WARNING,"CheckIsBetweenDates")df=spark.sql("""SELECTexplode(sequence(to_date('2022-01-01'),to_date('2022-01-10'),interval 1 day)) as date""")assert(check.is_between("date","2022-01-01","2022-01-10").validate(df).first().status=="PASS")MembershipOther common test is the validation oflist of valuesas part of the multiple integrity checks required for better quality data.df=spark.createDataFrame([[1,10],[2,15],[3,17]],["ID","value"])check=Check(CheckLevel.WARNING,"is_contained_in_number_test")check.is_contained_in("value",(10,15,20,25)).validate(df)Regular ExpressionsWhen it comes to the flexibility of matching, regular expressions are always to the rescue.cualleemakes use of the regular expressions to validate that fields of typeStringconform to specific patterns.df=spark.createDataFrame([[1,"is_blue"],[2,"has_hat"],[3,"is_smart"]],["ID","desc"])check=Check(CheckLevel.WARNING,"has_pattern_test")check.has_pattern("desc",r"^is.*t$")# only match is_smart 33% of rows.check.validate(df).first().status=="FAIL"AnomaliesStatistical tests are a great aid for verifying anomalies on data. Here an example that shows that willPASSonly when40%of data is inside the interquartile rangedf=spark.range(10)check=Check(CheckLevel.WARNING,"IQR_Test")check.is_inside_interquartile_range("id",pct=0.4)check.validate(df).first().status=="PASS"+---+-------------------+-----+-------+------+-----------------------------+-----+----+----------+---------+--------------+------+|id|timestamp|check|level|column|rule|value|rows|violations|pass_rate|pass_threshold|status|+---+-------------------+-----+-------+------+-----------------------------+-----+----+----------+---------+--------------+------+|1|2022-10-1900:09:39|IQR|WARNING|id|is_inside_interquartile_range|10000|10|4|0.6|0.4|PASS|+---+-------------------+-----+-------+------+-----------------------------+-----+----+----------+---------+--------------+------+Workflows (Process Mining)Besides the commoncitizen-likechecks,cualleeoffers out-of-the-box real-life checks. For example, suppose that you are workingSalesForceorSAPenvironment. Very likely your business processes will be driven by a lifecycle:Order-To-CashRequest-To-PayInventory-Logistics-DeliveryOthers.
In this scenario,cualleeoffers the ability that the sequence of events registered over time, are according to a sequence of events, like the example below:importpyspark.sql.functionsasFfromcualleeimportCheck,CheckLeveldata=pd.DataFrame({"name":["herminio","herminio","virginie","virginie"],"event":["new","active","new","active"],"date":["2022-01-01","2022-01-02","2022-01-03","2022-02-04"]})df=spark.createDataFrame(data).withColumn("date",F.to_date("date"))# Cuallee Process Mining# Testing that all edges on workflowscheck=Check(CheckLevel.WARNING,"WorkflowViolations")# Validate that 50% of data goes from new => activecheck.has_workflow("name","event","date",[("new","active")],pct=0.5)check.validate(df).show(truncate=False)+---+-------------------+------------------+-------+-------------------------+------------+--------------------+----+----------+---------+--------------+------+|id|timestamp|check|level|column|rule|value|rows|violations|pass_rate|pass_threshold|status|+---+-------------------+------------------+-------+-------------------------+------------+--------------------+----+----------+---------+--------------+------+|1|2022-11-0723:08:50|WorkflowViolations|WARNING|('name','event','date')|has_workflow|(('new','active'),)|4|2.0|0.5|0.5|PASS|+---+-------------------+------------------+-------+-------------------------+------------+--------------------+----+----------+---------+--------------+------+cualleeVSpydeequIn thetestfolder there aredockercontainers with the requirements to match the tests. Also aperftest.pyavailable at the root folder for interests.# 1000 rules / # of seconds

cuallee: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 162.00
pydeequ: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 322.00CatalogueCheckDescriptionDataTypeis_completeZeronullsagnosticis_uniqueZeroduplicatesagnosticare_completeZeronullson group of columnsagnosticare_uniqueComposite primary key checkagnosticis_greater_thancol > xnumericis_positivecol > 0numericis_negativecol < 0numericis_greater_or_equal_thancol >= xnumericis_less_thancol < xnumericis_less_or_equal_thancol <= xnumericis_equal_thancol == xnumericis_contained_incol in [a, b, c, ...]agnosticis_inAlias ofis_contained_inagnosticis_betweena <= col <= bnumeric, datehas_patternMatching a pattern defined as aregexstringhas_minmin(col) == xnumerichas_maxmax(col) == xnumerichas_stdσ(col) == xnumerichas_meanμ(col) == xnumerichas_sumΣ(col) == xnumerichas_percentile%(col) == xnumerichas_max_byA utilitary predicate formax(col_a) == x for max(col_b)agnostichas_min_byA utilitary predicate formin(col_a) == x for min(col_b)agnostichas_correlationFinds correlation between0..1oncorr(col_a, col_b)numerichas_entropyCalculates the entropy of a columnentropy(col) == xfor classification problemsnumericis_inside_interquartile_rangeVerifies column values reside inside limits of interquartile rangeQ1 <= col <= Q3used on anomalies.numericis_in_millionscol >= 1e6numericis_in_billionscol >= 1e9numericis_on_weekdayFor date fields confirms day is betweenMon-Fridateis_on_weekendFor date fields confirms day is betweenSat-Sundateis_on_mondayFor date fields confirms day isMondateis_on_tuesdayFor date fields confirms day isTuedateis_on_wednesdayFor date fields confirms day isWeddateis_on_thursdayFor date fields confirms day isThudateis_on_fridayFor date fields confirms day isFridateis_on_saturdayFor date fields confirms day isSatdateis_on_sundayFor date fields confirms day isSundateis_on_scheduleFor date fields confirms time windows i.e.9:00 - 17:00timestampis_dailyCan verify daily continuity on date fields by default.[2,3,4,5,6]which representsMon-Friin PySpark. However new schedules can be used for custom date continuitydatehas_workflowAdjacency matrix validation on3-columngraph, based ongroup,event,ordercolumns.agnosticsatisfiesAn openSQL expressionbuilder to construct custom checksagnosticvalidateThe ultimate transformation of a check with adataframeinput for validationagnosticISO StandardA new module has been incorporated incuallee==0.4.0which allows the verification of International Standard Organization columns in data frames. Simply access thecheck.isointerface to add the set of checks as shown below.CheckDescriptionDataTypeiso_4217currency compliantccystringiso_3166country compliantcountrystringdf=spark.createDataFrame([[1,"USD"],[2,"MXN"],[3,"CAD"],[4,"EUR"],[5,"CHF"]],["id","ccy"])check=Check(CheckLevel.WARNING,"ISO Compliant")check.iso.iso_4217("ccy")check.validate(df).show()+---+-------------------+-------------+-------+------+---------------+--------------------+----+----------+---------+--------------+------+|id|timestamp|check|level|column|rule|value|rows|violations|pass_rate|pass_threshold|status|+---+-------------------+-------------+-------+------+---------------+--------------------+----+----------+---------+--------------+------+|1|2023-05-1418:28:02|ISOCompliant|WARNING|ccy|is_contained_in|{'BHD','CRC','M...|   5|       0.0|      1.0|           1.0|  PASS|+---+-------------------+-------------+-------+------+---------------+--------------------+----+----------+---------+--------------+------+Snowflake ConnectionIn order to establish a connection to your SnowFlake accountcualleerelies in the following environment variables to be avaialble in your environment:SF_ACCOUNTSF_USERSF_PASSWORDSF_ROLESF_WAREHOUSESF_DATABASESF_SCHEMADatabricks ConnectionBy defaultcualleewill search for a SparkSession available in theglobalsso there is literally no need toSparkSession.builder. When working in a local environment it will automatically search for an available session, or start one.DuckDBFor testing onduckdbsimply pass your table name to your checket voilàimportduckdbconn=duckdb.connect(":memory:")check=Check(CheckLevel.WARNING,"DuckDB",table_name="temp/taxi/*.parquet")check.is_complete("VendorID")check.is_complete("tpep_pickup_datetime")check.validate(conn)idtimestampchecklevelcolumnrulevaluerowsviolationspass_ratepass_thresholdstatus012022-10-3123:15:06testWARNINGVendorIDis_completeN/A198175830.01.01.0PASS122022-10-3123:15:06testWARNINGtpep_pickup_datetimeis_completeN/A198175830.01.01.0PASSRoadmap100%data frame agnostic implementation of data quality checks.
Define once,run everywhere[x] PySpark 3.4.0[x] PySpark 3.3.0[x] PySpark 3.2.x[x] Snowpark DataFrame[x] Pandas DataFrame[x] DuckDB Tables[x] BigQuery Client[x] Polars DataFrameMetadata checkOpenMetadata IntegrationDagster IntegrationPDF ReportHelp us in a discussion?Whilst expanding the functionality feels a bit as an overkill because you most likely can connectsparkvia its drivers to whateverDBMSof your choice.
In the desire to make it even moreuser-friendlywe are aiming to makecualleeportable to all the providers above.Authorscanimus/ Herminio Vazquez / 🇲🇽vestalisvirginis/ Virginie Grosboillot / 🇫🇷LicenseApache License 2.0
Free for commercial use, modification, distribution, patent use, private use.
Just preserve the copyright and license.Made with ❤️ in Utrecht 🇳🇱Maintained over ⌛ from Ljubljana in 🇸🇮