Package: quacker
QuackerQuackeris a streamlined command-line interface (CLI) tool designed to replicatedbtsourcesas tables from a Cloud DataWarehouseinto a localDuckDBdatabase. This allows for faster and more cost-effective local development with dbt.Quackercurrently support syncing from eitherSnowflakeBigQueryFeaturesReplicates dbtsourcesto localDuckDBfiles.Simplifies localdbtdevelopment and testing.Supports flexibleprojectandmanifestdirectory paths.Supports flexible dbttargets.Supports multipledatabasesfrom the samewarehouse.(Optional) syncstablecopies of selected dbtmodelsto a DuckDB file as well.Quick terminologywarehouse- A Cloud Data Warehouse e.g.Snowflake,BigQuerydatabase- The highest level of data organisation in awarehouse. In BigQuery adatabaseis called aproject.schema- A logical grouping oftableswithin adatabase. In BigQuery aschemais called adataset.Getting StartedPrerequisitesBefore usingQuacker, you need to have the following set up:Pythoninstalled[Recommended] Avenvvirtual environmentA validdbtproject with awarehousetargetprofileA validtargetfor yourwarehousein yourdbtprofiles.ymlfileThe following adapters installed:dbt-duckdbdbt-<warehouse>e.g.dbt-bigquery(Optional)environment variablesloaded if you are using them in your dbt project.DuckDB ProfileAn exampleduckdbtargetprofile in aprofiles.ymlis seen below.In this example, mydbtsourcesexist in twosnowflakedatabasesfivetran_databasesnowflakeArbitrarily I have chosenfivetran_databaseas the name of thedatabasealldbtoutput will materialize in, but any of thewarehousedatabasenames could have been used here. All others need to beattachedto themaindatabase(here: thedatabaseof the namesnowflake).dev_duckdb:type:duckdbpath:data_duckdb/fivetran_database.duckdbattach:-path:data_duckdb/snowflake.duckdbschema:"{{env_var('SNOWFLAKE_SCHEMA')}}"Note:Thepathin theduckdbtargetprofile needs to match theduckdb_folder_namein thequacker_config.ymlfile if you are overwriting the default value. SeeOptional Configurationfor more details.Environment VariablesIf yourdbtproject usesenvironment variables, you will need to load them before runningquack sync. This is becausequack syncreads yourprofileto find the connection details of yourwarehouse.somedbtsetups useenvironment variablesto store these connection details.Here is an example of atargetwhich uses environment variables to store the connection details to thewarehousedatabase.dev_snowflake:account:"{{env_var('SNOWFLAKE_ACCOUNT')}}"database:"{{env_var('SNOWFLAKE_DATABASE')}}"password:"{{env_var('SNOWFLAKE_PASSWORD')}}"role:"{{env_var('SNOWFLAKE_ROLE')}}"schema:"{{env_var('SNOWFLAKE_SCHEMA')}}"threads:24type:snowflakeuser:"{{env_var('SNOWFLAKE_USERNAME')}}"warehouse:"{{env_var('SNOWFLAKE_WAREHOUSE')}}"[Optional] Conditionally persist docs based on contextIf you have the below in yourdbt_project.yml, you will receive the following error when running against duckdbERROR: alter_column_comment macro not implemented for adapter duckdb:models:+persist_docs:relation:truecolumns:trueTherefore, you need to have it set to only persist when running against your non-duckdbtarget. E.g.:+persist_docs:relation:"{{target.name=='dev_snowflake'}}"columns:"{{target.name=='dev_snowflake'}}"[Optional] Make your dbt models agnosticSome sql syntax is not compatible withduckdb. For example, offset() is not supported induckdb. Therefore, if you have amodelthat usesoffset(), you will need to make it agnostic to thetargettype. This can be done usingjinjaandifstatements. For example, the below code will work for bothduckdbandbigquerytargets:{%iftarget.type=='bigquery'-%}split(hubspot_contact_email_address,'@')[offset(1)]ashubspot_contact_email_domain_extracted,{%-eliftarget.type=='duckdb'-%}split_part(hubspot_contact_email_address,'@',2)ashubspot_contact_email_domain_extracted{%-endif%}NoteIf you cannot make yourmodelagnostic to thetargettype, you can add it to themodels_to_ignorelist in thequacker_config.ymlfile. SeeOptional Configurationfor more details.Installation for usepip3installquacker[Optional] Configuration;quacker_config.ymlQuackercan be configured using aquacker_config.ymlfile. This file should be placed in the same location as yourdbt_project.ymlfile. However, you don't need to create this file if you are happy with the default configuration. Below are the things you can configure in thequacker_config.ymlfile.row_limitThe maximum number of rows to query from thewarehousedatabasefor each table to be created duringquack sync. It is recommended to set this to a small number, but not at the expense of missing important data. The default value is1,000.Examplequacker_config.yml:row_limit:100duckdb_folder_nameThe name of the folder where theduckdbfiles will be stored. The default value isdata_duckdb.If you change this value, you will also need to update thepathin yourduckdbtargetprofile in yourprofiles.ymlfile. SeeDuckDB Profilefor more details.Examplequacker_config.yml:duckdb_folder_name:my_duckdb_foldermodels_to_ignoreA list ofdbtmodels to "ignore". This is useful formodelsthat are not compatible withDuckDB. For example,modelsthat useUNNEST(BigQuery) orPythondbtmodels.While we are "ignoring" thesedbtmodels during we still need to be able to rundbtagainstDuckDBafter thesync. To do this, the ignoreddbtmodelsare replicated astablesin the main DuckDB file during thesync.Examplequacker_config.yml:models_to_ignore:-stg_shopify__customers# dbt model 1-int_core__customers# dbt model 2Note:When subsequently runningdbtagainstDuckDB, you will need to pass the--excludeargument during yourdbt runsto avoid materialising thesemodelsin yourduckdbdatabase(you need to "ignore" them). For example, to "ignore" the two models during dbt operations:dbt run --exclude stg_shopify__customers int_core__customers.If you are configuringmodels_to_ignore, you will also need this setting inquacker_config.ymlsoQuackerknows where tosyncignored models to. There are plans to remove this requirement in the future.main_duckdb_database_name:fivetran_databaseNote:If you are usingmodels_to_ignore, make sure that yourduckdbtargethas the sameschemaname as thewarehousetarget. Otherwise your subsequentdbt runsmight fail because some of thetablesthat yourmodelsreference withref()are in a differently namedschemathan expected.UsageTo start usingQuacker, run thequack synccommand with the appropriateoptionalflags:quacksync\--project-dir<relative-path-to-your-dbt-project-directory>\--profiles-dir<full-path-to-your-profiles-directory>\--manifest-dir<relative-path-to-your-manifest-directory>\--compile-target<dbt-target-profile-name>Optional Argument Flags--project-dir:Relativepath to the directory containing yourdbt_project.ymlfile. If not specified evaluates to the current working directory.--profiles-dir:Fullpath to the directory containing yourprofiles.ymlfile. If not specified, path is resolvedusing dbt's method.--manifest-dir:Relativepath to the directory containing yourmanifest.jsonfile. If not specified, assumed to be intarget/relative to theproject-dir.--compile-target: The dbt target name to sync the data from. It's also the targetQuackeruses when runningdbt compilebefore extracting identifiers. If not specified,Quackeruses thedefaultprofile.Example Usagequacksync\--project-dir.\--profiles-dir/Users/username/path/to/profiles\--manifest-dir../poc_duckdb_for_local_dev/target\--compile-targetdev_snowflakedbt runafterquack syncIn order to run dbt locally against your DuckDB database, ensure you switch to using yourDuckDBtargetwhich should be configured like this. Otherwise yourdbtrunwill send queries to yourwarehouseinstead of yourDuckDBdatabase.HelpTo see the full list of available commands and arguments, runquack <subcommand> --helpe.g.quacksync--helpHow It Worksanyquackcommand performs the following steps:Reads and parses thequacker_config.ymlfile if it exists.quack syncperforms the following extra steps:Checks for the existence of a folder nameddata_duckdb/or the value ofduckdb_folder_nameinquacker_config.yml. If it does not exist, it creates it. This is where theduckdbfiles will be stored.Compiles thedbtproject.Parses themanifest.jsonfile to find identifiers of all dbtsourcesand, optionally, dbtmodelsspecifiedquacker_config.yml.Queries thewarehousedatabasefor allsourcesandmodels_to_ignorewith a cap of 1,000 rows or the value specified inrow_limitinquacker_config.yml. If any of thesourcesormodels_to_ignorehave more than this number of rows,Quackerwill randomly sample the data itsyncs.Saves the queried data intoDuckDBfiles. Forsources, oneduckdbfile is generated with the same name as thedatabasein thewarehouseinstance. Formodels_to_ignore, themainDuckDBfile (matching one of thesourcedatabases) is updated with themodel's data. See how the main file is specified in sectionDuckDB profile.After quack sync, theduckdbfiles will be stored in this general folder structure, with one database file perwarehousedatabase:data_duckdb/
├──fivetran_database.duckdb
├──snowflake.duckdb
└──...LimitationsConcurrent access to DuckDB filesYou should not attempt to query the DuckDB database files whileQuackeris running a sync operation. Two processes cannot connect to the same duckdb file at the same time.The error raised will be something likeduckdb.duckdb.IOException:IOError:Couldnotsetlockonfile"/Users/amir/Projects/poc_duckdb_for_local_dev/data_duckdb/fivetran_database.duckdb":ResourcetemporarilyunavailablePossible future enhancementsUnsorted QA feedback to be refined (i.e., not necessarily made into a TODO)Rittman delivery team's qarundbt depsbefore compileadd note to README: - you need to have done a fulldbt runbefore running quacker sync if you are using themodels_to_ignoreconfigurationfor bigquerysync: usesamplefunction instead ofrandomAmir's own qadon’t fail whenquacker_config.ymlexists but has nokeys(e.g. everything is commented out)Bugs to fixBigQuery: when asourceis atableconnected to a google sheet, this error occurs even if 1. theservice accounthas access to the sheet and 2. the same credentials work in thedbtprojectgoogle.api_core.exceptions.Forbidden: 403 Access Denied: BigQuery BigQuery: Permission denied while getting Drive credentials.Incrementality and full refreshIf a table already exists in the duckdb database, the query should no be run against the warehouse and data should not be overwritten in the duckdb database. Instead, a message should be displayed to the user to inform them that the table already exists.If the user wants to overwrite the data in the duckdb database, they should be able to pass afull-refreshflag to thequack synccommand to do so.Selective syncIf the user only wants to overwrite select tables in the duckdb database, they should further be able to pass a--tablesflag to thequack synccommand to do so.DuckDB profile check and creationIf no suitableduckdbtargetprofileexists, we could create one. This would involve:finding allduckdbtargets for theproject'sprofilechecking if any of the existing targets have all thesourcedatabasesand useattachif sources are split across multipledatabasesif none of thetargetsare suitable, creating a newduckdbtargetin theproject'sprofilereturning a message to the user to inform them of the newduckdbtargetprofileGifs for READMEAdd gifs to the README to show how to useQuackerand what it does.Emphasise side-by-side comparison of running dbt against warehouse vs duckdb afterQuackersyncError messagesTarget issuesWhen the--compile-targetis not of a supportedwarehousetypeProfiles.yml issuesWhen theprofiles.ymlfile is not found in the expected locationTODO write up more relevant error messages to implement as they are discoveredExtract the main DuckDB database name from the duckdb dbt profileThis would remove the need for themain_duckdb_database_namesetting inquacker_config.yml.Dealing with non-compatible dbt modelsphase 1 solution(done Jan 2024)Allow users to specify dbt models to ignore in a config file.Duringquack sync, replicate the ignored dbt models as tables in the main DuckDB file.phase 2 solutionTemporary until phase 4 solution is implementedadd subcommandquack recommend, which from the config generates the--excludeargument so they can manually use it when running dbt against duckdbphase 3 solutionAdd toquack recommend: suggest models to ignore based on incompatible syntax, such as UNNEST (BigQuery) or Python dbt models.phase 4 solution3- add aquack dbtcommand with args (for ignore functionality)Run dbt command passed in argument e.g.run,buildTo make this seamless use positional arguments if possible. E.g., command isquack dbt run --full-refresh. In this example, bothrunand--full-refreshare positional arguments.Might need to use some sort of (*args, **kwargs) solution here?Pass as an--exclude(?) on all the models which are ignored in the configexample dbt command generated and executed byQuacker:dbt run --full-refresh --exclude model1,model5Stop and inform user if they try toPass--excludethemselves down to dbtRun a dbt command which doesn't accept--excludeargdbt command failsphase 5 solution0-quack debugTODO should this functionality be added toquack recommendorquack syncinstead of a new command?Among others, warn if any of the ignored models are not tables in duckdb and suggest re-run of 'quack sync'Long-term solutionquack recommendcould generate DuckDB syntax equivalents wrapped in Jinja based on the target. For syntax not yet translated, it would fall back on the ignore recommendation.Debugging toolsDevelopquack debugto warn users of any ignored models that are not tables in DuckDB and suggest re-runningquack sync.Support for other databasesAs needed, we could add support for other databases in addition to the ones we currently support:Snowflake added Dec 2023BigQuery added Jan 2024Completed enhancementsSupport for table names with reserved SQL keywordsBefore this enhancement, if asourceormodelname contained a reserved SQL keyword, thequack synccommand would fail:if the database_type wassnowflake, on querying thewarehouseif the database_type wasbigquery, on creating the table in theduckdbdatabaseMorequacker_config.ymlsettings (done Jan 2024)Ability to customise row limit withrow_limitsetting inquacker_config.ymlAbility to customise the name of the folder where duckdb files are stored withduckdb_folder_namesetting inquacker_config.ymlRetrieve connection details from target profile(done Jan 2024)Extract the Snowflake credentials directly from the dbt profile, avoiding the need for separate environment variables.Investigate dbt's source code or dbt power users' methods for retrieving these credentials.failedInstead, manually code to replicate the order in which dbt searches for theprofiles.ymlfile (exact name).relevant dbt profile documentation:Specified using the--profiles-dirruntime argumentEnvironment VariableDBT_PROFILES_DIR: If you have set the DBT_PROFILES_DIR environment variable, dbt will use the directory specified in this variable to look for the profiles.yml fileCurrent Working Directory: The current working directory is the directory from which you are running the dbt command (where dbt_project.yml is)Default Directory~/.dbt/Simplifying sync(done Jan 2024)Specify compile sync target, such assnowflake-prodordev_snwflkwith--sync_targetargument.this wouldDevelopment and ContributionWe welcome contributions and feedback on our tool! Please reach out to me if you have any questions or would like to contribute:amir.jab.93+quacker@gmail.comPyPIQuacker is published toPyPIInstallation for developmentClone theQuackerrepository and install it usingpip(ideally in avenvvirtual environment):gitclonehttps://github.com/<your_username>/quacker.gitcdquacker
pip3install-e.The dot.represents the current directory. It can be replaced with a path to theQuackerrepository if you cloned it elsewhere. No matter whereQuackeris stored, you can run installed versions of it from anywhere on your machine as long as you are in the same virtual environment that you installed it in.The-eflag is optional and is used to installQuackerin editable mode, which is useful during development ofQuackeritself as it allows changes to be immediately effective without reinstallation. You don't need to use this flag if you are just usingQuacker.ContributorsAmir Jaber |GitHub|LinkedInSupportIf you encounter any issues or have questions, please open an issue in the project'sGitHub repository.LicenseQuackeris released under the MIT License.