Package: dbt-duckdb
dbt-duckdbDuckDBis an embedded database, similar to SQLite, but designed for OLAP-style analytics.
It is crazy fast and allows you to read and write data stored in CSV, JSON, and Parquet files directly, without requiring you to load
them into the database first.dbtis the best way to manage a collection of data transformations written in SQL or Python for analytics
and data science.dbt-duckdbis the project that ties DuckDB and dbt together, allowing you to create aModern Data Stack In
A Boxor a simple and powerful data lakehouse with Python.InstallationThis project is hosted on PyPI, so you should be able to install it and the necessary dependencies via:pip3 install dbt-duckdbThe latest supported version targetsdbt-core1.7.x andduckdbversion 0.10.x, but we work hard to ensure that newer
versions of DuckDB will continue to work with the adapter as they are released. If you would like to use our new (and experimental!)
support for persisting the tables that DuckDB creates to theAWS Glue Catalog, you should installdbt-duckdb[glue]in order to get the AWS dependencies as well.Configuring Your ProfileA super-minimal dbt-duckdb profile only needsonesetting:default:
  outputs:
    dev:
      type: duckdb
  target: devThis will run your dbt-duckdb pipeline against an in-memory DuckDB database that will not be persisted after your run completes. This may
not seem very useful at first, but it turns out to be a powerful tool for a) testing out data pipelines, either locally or in CI jobs and
b) running data pipelines that operate purely on external CSV, Parquet, or JSON files. More details on how to work with external data files
in dbt-duckdb are provided in the docs onreading and writing external files.To have your dbt pipeline persist relations in a DuckDB file, set thepathfield in your profile to the path
of the DuckDB file that you would like to read and write on your local filesystem. (For in-memory pipelines, thepathis automatically set to the special value:memory:).dbt-duckdbalso supports common profile fields likeschemaandthreads, but thedatabaseproperty is special: its value is automatically set
to the basename of the file in thepathargument with the suffix removed. For example, if thepathis/tmp/a/dbfile.duckdb, thedatabasefield will be set todbfile. If you are running in in-memory mode, then thedatabaseproperty will be automatically set tomemory.Using MotherDuckAs ofdbt-duckdb1.5.2, you can connect to a DuckDB instance running onMotherDuckby setting yourpathto use amd: connection string, just as you would with the DuckDB CLI
or the Python API.MotherDuck databases generally work the same way as local DuckDB databases from the perspective of dbt, but
there are afew differences to be aware of:Currently, MotherDuckrequiresa specific version of DuckDB, often the latest, as specified inMotherDuck's documentationMotherDuck databases do not suppport transactions, so there is a newdisable_transactionsprofile.
option that will be automatically enabled if you are connecting to a MotherDuck database in yourpath.MotherDuck preloads a set of the most common DuckDB extensions for you, but does not support loading custom extensions or user-defined functions.A small subset of advanced SQL features are currently unsupported; the only impact of this on the dbt adapter is that thedbt.listaggmacro and foreign-key constraints will work against a local DuckDB database, but will not work against a MotherDuck database.DuckDB Extensions, Settings, and FilesystemsYou can load any supportedDuckDB extensionsby listing them in
theextensionsfield in your profile. You can also set any additionalDuckDB configuration optionsvia thesettingsfield, including options that are supported in any loaded extensions. For example, to be able to connect to S3 and read/write
Parquet files using an AWS access key and secret, your profile would look something like this:default:
  outputs:
    dev:
      type: duckdb
      path: /tmp/dbt.duckdb
      extensions:
        - httpfs
        - parquet
      settings:
        s3_region: my-aws-region
        s3_access_key_id: "{{ env_var('S3_ACCESS_KEY_ID') }}"
        s3_secret_access_key: "{{ env_var('S3_SECRET_ACCESS_KEY') }}"
  target: devAs of version1.4.1, we have added (experimental!) support for DuckDB's (experimental!) support for filesystems
implemented viafsspec. Thefsspeclibrary provides
support for reading and writing files from avariety of cloud data storage systemsincluding S3, GCS, and Azure Blob Storage. You can configure a list of fsspec-compatible implementations for use with your dbt-duckdb project by installing the relevant Python modules
and configuring your profile like so:default:
  outputs:
    dev:
      type: duckdb
      path: /tmp/dbt.duckdb
      filesystems:
        - fs: s3
          anon: false
          key: "{{ env_var('S3_ACCESS_KEY_ID') }}"
          secret: "{{ env_var('S3_SECRET_ACCESS_KEY') }}"
          client_kwargs:
            endpoint_url: "http://localhost:4566"
  target: devHere, thefilesystemsproperty takes a list of configurations, where each entry must have a property namedfsthat indicates whichfsspecprotocol
to load (sos3,gcs,abfs, etc.) and then an arbitrary set of other key-value pairs that are used to configure thefsspecimplementation. You can see a simple example project that
illustrates the usage of this feature to connect to a Localstack instance running S3 from dbt-duckdbhere.Fetching credentials from contextInstead of specifying the credentials through the settings block, you can also use the use_credential_provider property. If you set this toaws(currently the only supported implementation) and you haveboto3installed in your python environment, we will fetch your AWS credentials using the credential provider chain as describedhere. This means that you can use any supported mechanism from AWS to obtain credentials (e.g., web identity tokens).Attaching Additional DatabasesDuckDB version0.7.0added support forattaching additional databasesto your dbt-duckdb run so that you can read
and write from multiple databases. Additional databases may be configured usingdbt run hooksor via theattachargument
in your profile that was added in dbt-duckdb1.4.0:default:
  outputs:
    dev:
      type: duckdb
      path: /tmp/dbt.duckdb
      attach:
        - path: /tmp/other.duckdb
        - path: ./yet/another.duckdb
          alias: yet_another
        - path: s3://yep/even/this/works.duckdb
          read_only: true
        - path: sqlite.db
          type: sqliteThe attached databases may be referred to in your dbt sources and models by either the basename of the database file minus its suffix (e.g.,/tmp/other.duckdbis theotherdatabase
ands3://yep/even/this/works.duckdbis theworksdatabase) or by an alias that you specify (so the./yet/another.duckdbdatabase in the above configuration is referred to
asyet_anotherinstead ofanother.) Note that these additional databases do not necessarily have to be DuckDB files: DuckDB's storage and catalog engines are pluggable, and
DuckDB0.7.0ships with support for reading and writing from attached SQLite databases. You can indicate the type of the database you are connecting to via thetypeargument,
which currently supportsduckdbandsqlite.Configuring dbt-duckdb Pluginsdbt-duckdb has its ownpluginsystem to enable advanced users to extend
dbt-duckdb with additional functionality, including:Definingcustom Python UDFson the DuckDB database connection
so that they can be used in your SQL modelsLoading source data fromExcel,Google Sheets, orSQLAlchemytablesYou can find more details onhow to write your own plugins here. To configure a plugin for use
in your dbt project, use thepluginsproperty on the profile:default:
  outputs:
    dev:
      type: duckdb
      path: /tmp/dbt.duckdb
      plugins:
        - module: gsheet
          config:
            method: oauth
        - module: sqlalchemy
          alias: sql
          config:
            connection_url: "{{ env_var('DBT_ENV_SECRET_SQLALCHEMY_URI') }}"
        - module: path.to.custom_udf_moduleEvery plugin must have amoduleproperty that indicates where thePluginclass to load is defined. There is
a set of built-in plugins that are defined indbt.adapters.duckdb.pluginsthat
may be referenced by their base filename (e.g.,excelorgsheet), while user-defined plugins (which are
described later in this document) should be referred to via their full module path name (e.g. alib.my.custommodule that defines a class namedPlugin.)Each plugin instance has a name for logging and reference purposes that defaults to the name of the module
but that may be overridden by the user by setting thealiasproperty in the configuration. Finally,
modules may be initialized using an arbitrary set of key-value pairs that are defined in theconfigdictionary. In this example, we initialize thegsheetplugin with the settingmethod: oauthand we
initialize thesqlalchemyplugin (aliased as "sql") with aconnection_urlthat is set via an environment variable.Please remember that using plugins may require you to add additional dependencies to the Python environment that your dbt-duckdb pipeline runs in:exceldepends onpandas, andopenpyxlorxlsxwriterto perform writesgsheetdepends ongspreadandpandasicebergdepends onpyicebergand Python >= 3.8sqlalchemydepends onpandas,sqlalchemy, and the driver(s) you needExperimental:deltadepends ondeltalake,an example projectNote:Be aware that experimental features can change over time, and we would like your feedback on config and possible different use cases.Using Local Python ModulesIn dbt-duckdb 1.6.0, we added a new profile setting namedmodule_pathsthat allows users to specify a list
of paths on the filesystem that contain additional Python modules that should be added to the Python processes'sys.pathproperty. This allows users to include additional helper Python modules in their dbt projects that
can be accessed by the running dbt process and used to define custom dbt-duckdb Plugins or library code that is
helpful for creating dbt Python models.Reading and Writing External FilesOne of DuckDB's most powerful features is its ability to read and write CSV, JSON, and Parquet files directly, without needing to import/export
them from the database first.Reading from external filesYou may reference external files in your dbt models either directly or as dbtsources by configuring theexternal_locationmeta option on the source:sources:
  - name: external_source
    meta:
      external_location: "s3://my-bucket/my-sources/{name}.parquet"
    tables:
      - name: source1
      - name: source2Here, themetaoptions onexternal_sourcedefinesexternal_locationas anf-stringthat
allows us to express a pattern that indicates the location of any of the tables defined for that source. So a dbt model like:SELECT *
FROM {{ source('external_source', 'source1') }}will be compiled as:SELECT *
FROM 's3://my-bucket/my-sources/source1.parquet'If one of the source tables deviates from the pattern or needs some other special handling, then theexternal_locationcan also be set on themetaoptions for the table itself, for example:sources:
  - name: external_source
    meta:
      external_location: "s3://my-bucket/my-sources/{name}.parquet"
    tables:
      - name: source1
      - name: source2
        meta:
          external_location: "read_parquet(['s3://my-bucket/my-sources/source2a.parquet', 's3://my-bucket/my-sources/source2b.parquet'])"In this situation, theexternal_locationsetting on thesource2table will take precedence, so a dbt model like:SELECT *
FROM {{ source('external_source', 'source2') }}will be compiled to the SQL query:SELECT *
FROM read_parquet(['s3://my-bucket/my-sources/source2a.parquet', 's3://my-bucket/my-sources/source2b.parquet'])Note that the value of theexternal_locationproperty does not need to be a path-like string; it can also be a function
call, which is helpful in the case that you have an external source that is a CSV file which requires special handling for DuckDB to load it correctly:sources:
  - name: flights_source
    tables:
      - name: flights
        meta:
          external_location: "read_csv('flights.csv', types={'FlightDate': 'DATE'}, names=['FlightDate', 'UniqueCarrier'])"
          formatter: oldstyleNote that we need to override the defaultstr.formatstring formatting strategy for this example
because thetypes={'FlightDate': 'DATE'}argument to theread_csvfunction will be interpreted bystr.formatas a template to be matched on, which will cause aKeyError: "'FlightDate'"when we attempt
to parse the source in a dbt model. Theformatterconfiguration option for the source indicates whether
we should usenewstylestring formatting (the default),oldstylestring formatting, ortemplatestring
formatting. You can read up on the strategies the various string formatting techniques use at thisStack Overflow answerand see examples of their use
in thisdbt-duckdb integration test.Writing to external filesWe support creating dbt models that are backed by external files via theexternalmaterialization strategy:{{ config(materialized='external', location='local/directory/file.parquet') }}
SELECT m.*, s.id IS NOT NULL as has_source_id
FROM {{ ref('upstream_model') }} m
LEFT JOIN {{ source('upstream', 'source') }} s USING (id)OptionDefaultDescriptionlocationexternal_locationmacroThe path to write the external materialization to. See below for more details.formatparquetThe format of the external file (parquet, csv, or json)delimiter,For CSV files, the delimiter to use for fields.optionsNoneAny other options to pass to DuckDB'sCOPYoperation (e.g.,partition_by,codec, etc.)glue_registerfalseIf true, try to register the file created by this model with the AWS Glue Catalog.glue_databasedefaultThe name of the AWS Glue database to register the model with.If thelocationargument is specified, it must be a filename (or S3 bucket/path), and dbt-duckdb will attempt to infer
theformatargument from the file extension of thelocationif theformatargument is unspecified (this functionality was
added in version 1.4.1.)If thelocationargument isnotspecified, then the external file will be named after the model.sql (or model.py) file that defined it
with an extension that matches theformatargument (parquet,csv, orjson). By default, the external files are created
relative to the current working directory, but you can change the default directory (or S3 bucket/prefix) by specifying theexternal_rootsetting in your DuckDB profile.dbt-duckdb supports thedelete+insertandappendstrategies for incrementaltablemodels, but unfortunately it does not yet support incremental materialization strategies forexternalmodels.Re-running external models with an in-memory version of dbt-duckdbWhen using:memory:as the DuckDB database, subsequent dbt runs can fail when selecting a subset of models that depend on external tables. This is because external files are only registered as  DuckDB views when they are created, not when they are referenced. To overcome this issue we have provided theregister_upstream_external_modelsmacro that can be triggered at the beginning of a run. To enable this automatic registration, place the following in yourdbt_project.ymlfile:on-run-start:-"{{register_upstream_external_models()}}"Python Supportdbt added support forPython models in version 1.3.0. For most data platforms,
dbt will package up the Python code defined in a.pyfile and ship it off to be executed in whatever Python environment that
data platform supports (e.g., Snowpark for Snowflake or Dataproc for BigQuery.) In dbt-duckdb, we execute Python models in the same
process that owns the connection to the DuckDB database, which by default, is the Python process that is created when you run dbt.
To execute the Python model, we treat the.pyfile that your model is defined in as a Python module and load it into the
running process usingimportlib. We then construct the arguments to themodelfunction that you defined (adbtobject that contains the names of anyrefandsourceinformation your model needs and aDuckDBPyConnectionobject for you to interact with the underlying DuckDB database), call themodelfunction, and then materialize
the returned object as a table in DuckDB.The value of thedbt.refanddbt.sourcefunctions inside of a Python model will be aDuckDB Relationobject that can be easily converted into a Pandas/Polars DataFrame or an Arrow table. The return value of themodelfunction can be
any Python object that DuckDB knows how to turn into a table, including a Pandas/PolarsDataFrame, a DuckDBRelation, or an ArrowTable,Dataset,RecordBatchReader, orScanner.Batch processing with Python modelsAs of version 1.6.1, it is possible to both read and write data in chunks, which allows for larger-than-memory
datasets to be manipulated in Python models. Here is a basic example:import pyarrow as pa

def batcher(batch_reader: pa.RecordBatchReader):
    for batch in batch_reader:
        df = batch.to_pandas()
        # Do some operations on the DF...
        # ...then yield back a new batch
        yield pa.RecordBatch.from_pandas(df)

def model(dbt, session):
    big_model = dbt.ref("big_model")
    batch_reader = big_model.record_batch(100_000)
    batch_iter = batcher(batch_reader)
    return pa.RecordBatchReader.from_batches(batch_reader.schema, batch_iter)Writing Your Own PluginsDefining your own dbt-duckdb plugin is as simple as creating a python module that defines a class namedPluginthat
inherits fromdbt.adapters.duckdb.plugins.BasePlugin. There are currently
four methods that may be implemented in your Plugin class:initialize: Takes in theconfigdictionary for the plugin that is defined in the profile to enable any
additional configuration for the module based on the project; this method is called once when an instance of thePluginclass is created.configure_connection: Takes an instance of theDuckDBPyConnectionobject used to connect to the DuckDB
database and may perform any additional configuration of that object that is needed by the plugin, like defining
custom user-defined functions.load: Takes aSourceConfiginstance, which encapsulates the configuration for a
a dbt source and can optionally return a DataFrame-like object that DuckDB knows how to turn into a table (this is
similar to a dbt-duckdb Python model, but without the ability torefany models or access any information beyond
the source config.)store: Takes aTargetConfiginstance, which encapsulates the configuration for
anexternalmaterialization and can perform additional operations once the CSV/Parquet/JSON file is written. Theglueandsqlalchemyare examples
that demonstrate how to use thestoreoperation to register an AWS Glue database table or upload a DataFrame to
an external database, respectively.dbt-duckdb ships with a number ofbuilt-in pluginsthat can be used as examples
for implementing your own.RoadmapThings that we would like to add in the near future:Support for Delta and Iceberg external table formats (both as sources and destinations)Make dbt's incremental models and snapshots work with external materializations