Package: dbt-table-diff
dbt_table_diffThis repository is intended for comparingBigQuerymodelsindbtthat have changed during an open PR.UsageThe repository has been published as aGithub ActionandPyPi Package, which means it can be leveraged in a variety of ways:Directly in Pythonviarun_dbt_table_diff.Directly in Terminalviapython3 -m dbt_table_diff.In a Github Workflow FileviaGithub Actionstoautomatically add commentson Open PRs.Quick Start:pip3 install dbt_table_diffExample Code Usage:from dbt_table_diff import run_dbt_table_diff

run_dbt_table_diff(
        project_id="ultimate-bit-359101",
        keyfile_path="secrets/bq_keyfile.json",
        manifest_file="target/manifest.json",
        dev_prefix="dev_",
        prod_prefix="prod_",
        fallback_prefix="fb_",
        custom_checks_path="",
        ignored_schemas=[],
        irregular_schemas=[],
        org_name="org-not-included",
        repo_name="dbt_example",
        pr_id="2",
        auth_token="my_github_pat",
)Example CLI Usage:python3-mdbt_table_diff-t$GH_TOKEN-oorg-not-included-rdbt_example-l2\--manifest_file'target/manifest.json'--project_id'ultimate-bit-359101'\--keyfile_path'secrets/bq_keyfile.json'--dev_prefix'dev_'--prod_prefix'prod_'--fallback_prefix'fb_'Example Github Action Usage:Overviewof Github ActionsOpen PRshowing how to usedbt_table_diffas a Github Action.Github Actions Input Arguments:Input ParameterDescriptionGCP_TOKENfor connecting to BQ (runsdbt compileanddbt_table_diff/sql_checksto compare tables)GH_TOKENfor connecting to Github (ie. fetches modifiedmodels/*.sqlin your PR, adds comment on your PR)PR_NUMBERfor fetching open PR from github (Pull Request ID [int])GH_REPOfor fetching open PR from github (Repository Name)GH_ORGfor fetching open PR from github (Repository owner/organization name)DBT_PROFILE_FILEthe local path in your repo to yourprofile.ymlfor dbt (this is necessary for compilingmanifest.jsonduring setup process)dev_prefixthe prefix used when running dbt locally (Your source schema/environment for comparison)prod_prefixthe prefix used when running dbt remotely (Your target schema/environment for comparison)fallback_prefixuseful if you have an overriden macro forgenerate_schema_namein your dbt project, which leverages a different prefix for some schemas in prod.irregular_schemascomma separated string of schemas which usefallback_prefixproject_idfor connecting to BQ (BigQuery Project ID)ignored_schemascomma separated string of schemas to ignore (skip checking during github action)custom_checks_pathA local foldercontaining any custom SQL checks to run.Step-By-Step Break Down of Process:Fetches list of files modified in Pull Requestby CURLinggithub.api.com/repos/{organization}/{repository}/pulls/{pull_request_id}/filesFilters onrelevant_fileswhich are files matchingmodels/*.sqlBuildsmanifest.jsonBy runningdbt deps; dbt compileParsesmanifest.jsonforrelevant_modelsusing manifest-attributeoriginal_file_pathmatchingrelevant_filesRuns all SQL files indbt_table_diff/sql_checksfor each of therelevant_models, compare the two dbt targets (dev_prefixvsprod_prefix)Saves output to filein a format supported by Github commentsPosts comment on open PRleveragingdbt_table_diffPyPi packageDocspython3-mdbt_table_diff--helpusage: dbt_table_diff [-h] [-o ORG_NAME] [-r REPO_NAME] [-t AUTH_TOKEN] [-l PR_ID] [--manifest_file MANIFEST_FILE] [--project_id PROJECT_ID] [--keyfile_path KEYFILE_PATH] [--ignored_schemas IGNORED_SCHEMAS]
                      [--irregular_schemas IRREGULAR_SCHEMAS] [--dev_prefix DEV_PREFIX] [--prod_prefix PROD_PREFIX] [--fallback_prefix FALLBACK_PREFIX] [--custom_checks_path CUSTOM_CHECKS_PATH]

optional arguments:
  -h, --help            show this help message and exit
  -o ORG_NAME, --org_name ORG_NAME
                        Owner of GitHub repository.
  -r REPO_NAME, --repo_name REPO_NAME
                        Name of the GitHub repository.
  -t AUTH_TOKEN, --auth_token AUTH_TOKEN
                        User's GitHub Personal Access Token.
  -l PR_ID, --pr_id PR_ID
                        The issue # of the Pull Request.
  --manifest_file MANIFEST_FILE
                        The path to dbt's manifest file.
  --project_id PROJECT_ID
                        The BigQuery Project ID to leverage.
  --keyfile_path KEYFILE_PATH
                        The path to the keyfile to use during BQ calls.
  --ignored_schemas IGNORED_SCHEMAS
                        Folders in models/ to always ignore during row/col checks.
  --irregular_schemas IRREGULAR_SCHEMAS
                        Folders in models/ which use 'fallback_prefix' in prod.
  --dev_prefix DEV_PREFIX
                        Prefix used by development datasets in dbt.
  --prod_prefix PROD_PREFIX
                        Prefix used by production datasets in dbt.
  --fallback_prefix FALLBACK_PREFIX
                        Uncommon prefix used by only some production datasets in dbt.
  --custom_checks_path CUSTOM_CHECKS_PATH
                        A local folder containing any custom SQL to run.