Package: data-diff-customize
data-diff: Compare datasets fast, within or across SQL databasesUse CasesData Migration & Replication TestingCompare source to target and check for discrepancies when moving data between systems:Migrating to a new data warehouse (e.g., Oracle > Snowflake)Converting SQL to a new transformation framework (e.g., stored procedures > dbt)Continuously replicating data from an OLTP DB to OLAP DWH (e.g., MySQL > Redshift)Data Development TestingTest SQL code and preview changes by comparing development/staging environment data to production:Make a change to some SQL codeRun the SQL code to create a new datasetCompare the dataset with its production version or another iterationdata-diff integrates with dbt Core to seamlessly compare local development to production datasetsdbt Cloud users should check out Datafold's out-of-the-box deployment testing integration:eyes:Watch4-min demo videoGet started with data-diff & dbtAlso available in aVS Code ExtensionReach out on the dbt Slack in#tools-datafoldfor advice and supportHow it worksWhen comparing the data,data-diffutilizes the resources of the underlying databases as much as possible. It has two primary modes of comparison:joindiffRecommended for comparing data within the same databaseUses the outer join operation to diff the rows as efficiently as possible within the same databaseFully relies on the underlying database engine for computationRequires both datasets to be queryable with a single SQL queryTime complexity approximates JOIN operation and is largely independent of the number of differences in the datasethashdiffRecommended for comparing datasets across different databasesCan also be helpful in diffing very large tables with few expected differences within the same databaseEmploys a divide-and-conquer algorithm based on hashing and binary searchCan diff data across distinct database engines, e.g., PostgreSQL <> SnowflakeTime complexity approximates COUNT(*) operation when there are few differencesPerformance degrades when datasets have a large number of differencesMore information about the algorithm and performance considerations can be foundhereGet startedValidating dbt model changes between dev and prod⚡ Looking to usedata-diffin dbt development? Head over toourdata-diff+dbtdocumentationto get started!Compare data tables between databases🔀 To compare data between databases, installdata-diffwith specific database adapters, e.g.:pip install data-diff 'data-diff[postgresql,snowflake]' -URundata-diffwith connection URIs. In the following example, we compare tables between PostgreSQL and Snowflake using the hashdiff algorithm:data-diff\postgresql://<username>:'<password>'@localhost:5432/<database>\<table>\"snowflake://<username>:<password>@<account>/<DATABASE>/<SCHEMA>?warehouse=<WAREHOUSE>&role=<ROLE>"\<TABLE>\-k<primarykeycolumn>\-c<columnstocompare>\-w<filtercondition>Rundata-diffwith atomlconfiguration file. In the following example, we compare tables between MotherDuck(hosted DuckDB) and Snowflake using the hashdiff algorithm:## DATABASE CONNECTION ##[database.duckdb_connection]driver="duckdb"# filepath = "datafold_demo.duckdb" # local duckdb file example# filepath = "md:" # default motherduck connection examplefilepath="md:datafold_demo?motherduck_token=${motherduck_token}"# API token recommended for motherduck connectiondatabase="datafold_demo"[database.snowflake_connection]driver="snowflake"database="DEV"user="sung"password="${SNOWFLAKE_PASSWORD}"# or "<PASSWORD_STRING>"# the info below is only required for snowflakeaccount="${ACCOUNT}"# by33919schema="DEVELOPMENT"warehouse="DEMO"role="DEMO_ROLE"## RUN PARAMETERS ##[run.default]verbose=true## EXAMPLE DATA DIFF JOB ##[run.demo_xdb_diff]# Source 1 ("left")1.database="duckdb_connection"1.table="development.raw_orders"# Source 2 ("right")2.database="snowflake_connection"2.table="RAW_ORDERS"# note that snowflake table names are case-sensitiveverbose=false# export relevant environment variables, example belowexportmotherduck_token=<MOTHERDUCK_TOKEN># run the configured data-diff jobdata-diff--confdatadiff.toml\--rundemo_xdb_diff\-k"id"\-cstatus# output example-1,completed
+1,returnedCheck outdocumentationfor the full command reference.Supported databasesDatabaseStatusConnection stringPostgreSQL >=10🟢postgresql://<user>:<password>@<host>:5432/<database>MySQL🟢mysql://<user>:<password>@<hostname>:5432/<database>Snowflake🟢"snowflake://<user>[:<password>]@<account>/<database>/<SCHEMA>?warehouse=<WAREHOUSE>&role=<role>[&authenticator=externalbrowser]"BigQuery🟢bigquery://<project>/<dataset>Redshift🟢redshift://<username>:<password>@<hostname>:5439/<database>DuckDB🟢duckdb://<dbname>@<filepath>MotherDuck🟢duckdb://<dbname>@<filepath>Oracle🟡oracle://<username>:<password>@<hostname>/servive_or_sidPresto🟡presto://<username>:<password>@<hostname>:8080/<database>Databricks🟡databricks://<http_path>:<access_token>@<server_hostname>/<catalog>/<schema>Trino🟡trino://<username>@<hostname>:8080/<database>trino://<username>:<password>@<hostname>:8080/<database>?auth=basicClickhouse🟡clickhouse://<username>:<password>@<hostname>:9000/<database>Vertica🟡vertica://<username>:<password>@<hostname>:5433/<database>ElasticSearch📝Planetscale📝Pinot📝Druid📝Kafka📝SQLite📝🟢: Implemented and thoroughly tested.🟡: Implemented, but not thoroughly tested yet.⏳: Implementation in progress.📝: Implementation planned. Contributions welcome.Your database not listed here?Contribute anew database adapter– we accept pull requests!Get in touchabout enterprise support and adding new adapters and featuresContributorsWe thank everyone who contributed so far!AnalyticsUsage Analytics & Data PrivacyLicenseThis project is licensed under the terms of theMIT License.