Package: recce
RecceRecceis a PR review tool designed for DBT projects. It facilitates the comparison of results between two environments, such as development and production, and helps in identifying any differences.FeaturesLineage diffSchema diffRow count diffAd-Hoc Query and Query DiffProfile DiffValue DiffChecklistUse casesRecce is primarily designed for PR Review purposes. However, it can be expanded to the following use cases:During development, we can verify new results by contrasting them with those from production prior to pushing the changes.While reviewing PR, you can grasp the extent of the changes and their impact before merging.For troubleshooting, you can execute ad-hoc diff queries to pinpoint the root causes.UsagePrerequisitesYou have to have at least twoenvironmentsin your dbt project. For example, one is for development and another is for production. You can prepare two targets with separateschemasin your DBT profile. Here isprofiles.ymlexamplejaffle_shop:
  target: dev
  outputs:
    dev:
      type: duckdb
      path: jaffle_shop.duckdb
      schema: dev
    prod:
      type: duckdb
      path: jaffle_shop.duckdb
      schema: mainGetting Started5 minutes walkthrough by jaffle shop exampleInstallationpipinstallrecceGo to your DBT projectcdyour-dbt-project/Prepare artifacts for base environment intarget-base/foldergitcheckoutmain# Generate artifacts for base environment to 'target-base'dbtdocsgenerate--targetprod--target-pathtarget-base/Prepare artifacts for current working environmentgitcheckoutfeature/my-awesome-feature# Run dbt and generate artifacts for current working environmentsdbtrun
dbtdocsgenerateRun the recce server.recceserver# or with persistent state# recce server issue-123.jsonRecce would diff environments betweentarget/andtarget-base/Recce use dbtartifacts, which is generated when every invocation. You can find these files in thetarget/folder.artifactsDBT commandmanifest.jsondbt docs generate,dbt run, ..catalog.jsondbt docs generate[!TIP]
The regeneration of thecatalog.jsonfile is not required after everydbt run. it is only required to regenerate this file when models or columns are added or updated.Lineage DiffAd-Hoc Query and Query DiffYou can use any dbt macros installed in your project.select * from{{ref("mymodel")}}In the query diff, which involves comparing the results from two different environments, the browser is required to pull all result data to the client side. Consequently, minimizing the data volume in the query results is essential for efficiency and performance.[!TIP]
Hotkeys
Cmd + Enter: Run query
Cmd + Shift + Enter: Run query diffSchema diffTo use schema diff, make sure that both environments havecatalog.json.Row count diffProfile diffProfile diff uses theget_profilefromdbt_profiler. Make sure that this package is installed in your project.packages:
  - package: data-mie/dbt_profiler
    version: <version>Please reference [dbt-profiler] to understand the definition of each profiling stats.Value diffAdded: New added PKs.Removed: Removed PKs.Matched: For a column, the count of matched value of common PKs.Matched %: For a column, the ratio of matched over common PKs.PK: Primary keyValue diff uses thecompare_column_valuesfromaudit-helper. Make sure that this package is installed in your project.packages:
  - package: dbt-labs/audit_helper
    version: <version>Value diff requires to select a column as private key. Thecatalog.jsonis required to list the available columns.ChecklistWhen you feel that your query is worth recording, you can add it to the checklist and name the title and leave a description. This is very helpful for subsequent posting in PR comments.Q&AQ: Howrecceconnect to my data warehouse? Does recce support my data warehouse?recceuse thedbt adapterto connect to your warehouse. So it should work for your data warehouse.Q: What credential doesrecceconnect to the two environments?Recce uses the same target in the profile to connect your warehouse. If you use the default targetdev, it uses the credentials to connect to both environments. So please make sure that the credential can access both environments.