Package: data-diff
data-diff: Compare datasets fast, within or across SQL databasesJoin our live virtual lab series to learn how to set it up!What's a Data Diff?A data diff is the value-level comparison between two tables—used to identify critical changes to your data and guarantee data quality.There is a lot you can do with data-diff: you can test SQL code by comparing development or staging environment data to production, or compare source and target data to identify discrepancies when moving data between databases.data-diff OSS & Datafold Clouddata-diff is an open source utility for running stateless diffs as a great single player experience.Scale up withDatafold Cloudto make data diffing a company-wide experience to both supercharge your data diffing CLI experience (ex: data-diff --dbt --cloud) and run diffs manually in your CI process and within the Datafold UI. This includescolumn-level lineagewith BI tool integrations,CI testing, faster cross-database diffing, and diff history.Use CasesData Development TestingWhen developing SQL code, data-diff helps you validate and preview changes by comparing data between development/staging environments and production. Here's how it works:Make a change to your SQL codeRun the SQL code to create a new datasetCompare this dataset with its production version or other iterationsData Migration & Replication Testingdata-diff is a powerful tool for comparing data when you're moving it between systems. Use it to ensure data accuracy and identify discrepancies during tasks like:Migratingto a new data warehouse (e.g., Oracle -> Snowflake)Validating SQL transformationsfrom legacy solutions (e.g., stored procedures) to new transformation frameworks (e.g., dbt)Continuouslyreplicating datafrom an OLTP database to OLAP data warehouse (e.g., MySQL -> Redshift)dbt Integrationdata-diff integrates withdbt Coreto seamlessly compare local development to production datasets.Learn more about how data-diff works with dbt:Read our docs to get started withdata-diff & dbtor :eyes:watch the4-min demo videodbt Cloud users should check outDatafold's out-of-the-box deployment testing integrationGet support from the dbt Community Slack in#tools-datafoldGetting Started⚡ Validating dbt model changes between dev and prodLooking to use data-diff in dbt development?Development testing with Datafold enables you to see the impact of dbt code changes on data as you write the code, whether in your IDE or CLI.Head over toourdata-diff+dbtdocumentationto get started with a development testing workflow!🔀 Compare data tables between databasesInstalldata-diffwith adaptersTo compare data between databases, installdata-diffwith specific database adapters. For example, install it for PostgreSQL and Snowflake like this:pip install data-diff 'data-diff[postgresql,snowflake]' -UAdditionally, you can install all open source supported database adapters as follows.pip install data-diff 'data-diff[all-dbs]' -URundata-diffwith connection URIsThen, we compare tables between PostgreSQL and Snowflake using the hashdiff algorithm:data-diff\postgresql://<username>:'<password>'@localhost:5432/<database>\<table>\"snowflake://<username>:<password>@<account>/<DATABASE>/<SCHEMA>?warehouse=<WAREHOUSE>&role=<ROLE>"\<TABLE>\-k<primarykeycolumn>\-c<columnstocompare>\-w<filtercondition>Set up your configurationYou can use atomlconfiguration file to run yourdata-diffjob. In this example, we compare tables between MotherDuck (hosted DuckDB) and Snowflake using the hashdiff algorithm:## DATABASE CONNECTION ##[database.duckdb_connection]driver="duckdb"# filepath = "datafold_demo.duckdb" # local duckdb file example# filepath = "md:" # default motherduck connection examplefilepath="md:datafold_demo?motherduck_token=${motherduck_token}"# API token recommended for motherduck connection[database.snowflake_connection]driver="snowflake"database="DEV"user="sung"password="${SNOWFLAKE_PASSWORD}"# or "<PASSWORD_STRING>"# the info below is only required for snowflakeaccount="${ACCOUNT}"# by33919schema="DEVELOPMENT"warehouse="DEMO"role="DEMO_ROLE"## RUN PARAMETERS ##[run.default]verbose=true## EXAMPLE DATA DIFF JOB ##[run.demo_xdb_diff]# Source 1 ("left")1.database="duckdb_connection"1.table="development.raw_orders"# Source 2 ("right")2.database="snowflake_connection"2.table="RAW_ORDERS"# note that snowflake table names are case-sensitiveverbose=falseRun yourdata-diffjobMake sure to export relevant environment variables as needed. For example, we compare data based on the earlier configuration:# export relevant environment variables, example belowexportmotherduck_token=<MOTHERDUCK_TOKEN># run the configured data-diff jobdata-diff--confdatadiff.toml\--rundemo_xdb_diff\-k"id"\-cstatus# output example-1,completed
+1,returnedReview the outputAfter running your data-diff job, review the output to identify and analyze differences in your data.Check outdocumentationfor the full command reference.Supported databasesDatabaseStatusConnection stringPostgreSQL >=10🟢postgresql://<user>:<password>@<host>:5432/<database>MySQL🟢mysql://<user>:<password>@<hostname>:5432/<database>Snowflake🟢"snowflake://<user>[:<password>]@<account>/<database>/<SCHEMA>?warehouse=<WAREHOUSE>&role=<role>[&authenticator=externalbrowser]"BigQuery🟢bigquery://<project>/<dataset>Redshift🟢redshift://<username>:<password>@<hostname>:5439/<database>DuckDB🟢duckdb://<filepath>MotherDuck🟢duckdb://<filepath>Microsoft SQL Server*🟢mssql://<user>:<password>@<host>/<database>/<schema>Oracle🟡oracle://<username>:<password>@<hostname>/servive_or_sidPresto🟡presto://<username>:<password>@<hostname>:8080/<database>Databricks🟡databricks://<http_path>:<access_token>@<server_hostname>/<catalog>/<schema>Trino🟡trino://<username>:<password>@<hostname>:8080/<database>Clickhouse🟡clickhouse://<username>:<password>@<hostname>:9000/<database>Vertica🟡vertica://<username>:<password>@<hostname>:5433/<database>ElasticSearch📝Planetscale📝Pinot📝Druid📝SQLite📝*MS SQL Server support is limited, with known performance issues that are addressed in Datafold Cloud.🟢: Implemented and thoroughly tested.🟡: Implemented, but not thoroughly tested yet.⏳: Implementation in progress.📝: Implementation planned. Contributions welcome.Your database not listed here?Contribute anew database adapter– we accept pull requests!Get in touchabout enterprise support and adding new adapters and featuresHow it worksdata-diffefficiently compares data using two modes:joindiff: Ideal for comparing data within the same database, utilizing outer joins for efficient row comparisons. It relies on the database engine for computation and has consistent performance.hashdiff: Recommended for comparing datasets across different databases or large tables with minimal differences. It uses hashing and binary search, capable of diffing data across distinct database engines.Click here to learn more about joindiff and hashdiffjoindiffRecommended for comparing data within the same databaseUses the outer join operation to diff the rows as efficiently as possible within the same databaseFully relies on the underlying database engine for computationRequires both datasets to be queryable with a single SQL queryTime complexity approximates JOIN operation and is largely independent of the number of differences in the datasethashdiff:Recommended for comparing datasets across different databasesCan also be helpful in diffing very large tables with few expected differences within the same databaseEmploys a divide-and-conquer algorithm based on hashing and binary searchCan diff data across distinct database engines, e.g., PostgreSQL <> SnowflakeTime complexity approximates COUNT(*) operation when there are few differencesPerformance degrades when datasets have a large number of differencesFor detailed algorithm and performance insights, explorehere, or head to our docs tolearn more about how Datafold diffs data.ContributorsWe thank everyone who contributed so far!We'd love to see your face here:Contributing InstructionsAnalyticsUsage Analytics & Data PrivacyLicenseThis project is licensed under the terms of theMIT License.