Package: dbt-athena-community
FeaturesQuick StartInstallationPrerequisitesCredentialsConfiguring your profileAdditional informationModelsTable ConfigurationTable locationIncremental modelsOn schema changeIcebergHighly available table (HA)HA Known issuesSnapshotsTimestamp strategyCheck strategyHard-deletesAWS Lakeformation integrationWorking exampleSnapshots Known issuesContractsContributingContributors ✨FeaturesSupports dbt version1.7.*Support for PythonSupportsseedsCorrectly detects views and their columnsSupportstable materializationIceberg tablesare supportedonly with Athena Engine v3anda unique table location(see table location section below)Hive tables are supported by both Athena engines.Supportsincremental modelsOn Iceberg tables :Supports the use ofunique_keyonly with themergestrategySupports theappendstrategyOn Hive tables :Supports two incremental update strategies:insert_overwriteandappendDoesnotsupport the use ofunique_keySupportssnapshotsSupportsPython modelsQuick StartInstallationpip install dbt-athena-communityOrpip install git+https://github.com/dbt-athena/dbt-athena.gitPrerequisitesTo start, you will need an S3 bucket, for instancemy-bucketand an Athena database:CREATEDATABASEIFNOTEXISTSanalytics_devCOMMENT'Analytics models generated by dbt (development)'LOCATION's3://my-bucket/'WITHDBPROPERTIES('creator'='Foo Bar','email'='foo@bar.com');Notes:Take note of your AWS region code (e.g.us-west-2oreu-west-2, etc.).You can also useAWS Glueto create and manage Athena
databases.CredentialsCredentials can be passed directly to the adapter, or they can
bedetermined automaticallybased
onaws cli/boto3conventions.
You can either:configureaws_access_key_idandaws_secret_access_keyconfigureaws_profile_nameto match a profile defined in your AWS credentials file
Checkout dbt profile configuration below for details.Configuring your profileA dbt profile can be configured to run against AWS Athena using the following configuration:OptionDescriptionRequired?Examples3_staging_dirS3 location to store Athena query results and metadataRequireds3://bucket/dbt/s3_data_dirPrefix for storing tables, if different from the connection'ss3_staging_dirOptionals3://bucket2/dbt/s3_data_namingHow to generate table paths ins3_data_dirOptionalschema_table_uniques3_tmp_table_dirPrefix for storing temporary tables, if different from the connection'ss3_data_dirOptionals3://bucket3/dbt/region_nameAWS region of your Athena instanceRequiredeu-west-1schemaSpecify the schema (Athena database) to build models into (lowercaseonly)RequireddbtdatabaseSpecify the database (Data catalog) to build models into (lowercaseonly)Requiredawsdatacatalogpoll_intervalInterval in seconds to use for polling the status of query results in AthenaOptional5debug_query_stateFlag if debug message with Athena query state is neededOptionalfalseaws_access_key_idAccess key ID of the user performing requests.OptionalAKIAIOSFODNN7EXAMPLEaws_secret_access_keySecret access key of the user performing requestsOptionalwJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEYaws_profile_nameProfile to use from your AWS shared credentials file.Optionalmy-profilework_groupIdentifier of Athena workgroupOptionalmy-custom-workgroupnum_retriesNumber of times to retry a failing queryOptional3spark_work_groupIdentifier of Athena Spark workgroupOptionalmy-spark-workgroupnum_boto3_retriesNumber of times to retry boto3 requests (e.g. deleting S3 files for materialized tables)Optional5seed_s3_upload_argsDictionary containing boto3 ExtraArgs when uploading to S3Optional{"ACL": "bucket-owner-full-control"}lf_tags_databaseDefault LF tags for new database if it's created by dbtOptionaltag_key: tag_valueExample profiles.yml entry:athena:target:devoutputs:dev:type:athenas3_staging_dir:s3://athena-query-results/dbt/s3_data_dir:s3://your_s3_bucket/dbt/s3_data_naming:schema_tables3_tmp_table_dir:s3://your_s3_bucket/temp/region_name:eu-west-1schema:dbtdatabase:awsdatacatalogthreads:4aws_profile_name:my-profilework_group:my-workgroupspark_work_group:my-spark-workgroupseed_s3_upload_args:ACL:bucket-owner-full-controlAdditional informationthreadsis supporteddatabaseandcatalogcan be used interchangeablyModelsTable Configurationexternal_location(default=none)If set, the full S3 path in which the table will be saved.It works only with incremental models.Does not work with Hive table withhaset to true.partitioned_by(default=none)An array list of columns by which the table will be partitionedLimited to creation of 100 partitions (currently)bucketed_by(default=none)An array list of columns to bucket data, ignored if using Icebergbucket_count(default=none)The number of buckets for bucketing your data, ignored if using Icebergtable_type(default='hive')The type of tableSupportshiveoricebergha(default=false)If the table should be built using the high-availability method. This option is only available for Hive tables
since it is by default for Iceberg tables (see the sectionbelow)format(default='parquet')The data format for the tableSupportsORC,PARQUET,AVRO,JSON,TEXTFILEwrite_compression(default=none)The compression type to use for any storage format that allows compression to be specified. To see which options are
available, check outCREATE TABLE ASfield_delimiter(default=none)Custom field delimiter, for when format is set toTEXTFILEtable_properties: table properties to add to the table, valid for Iceberg onlynative_drop: Relation drop operations will be performed with SQL, not direct Glue API calls. No S3 calls will be
made to manage data in S3. Data in S3 will only be cleared up for Iceberg
tablessee AWS docs. Note that
Iceberg DROP TABLE operations may timeout if they take longer than 60 seconds.seed_by_insert(default=false)default behaviour uploads seed data to S3. This flag will create seeds using an SQL insert statementlarge seed files cannot useseed_by_insert, as the SQL insert statement would
exceedthe Athena limit of 262144 bytesforce_batch(default=false)Skip creating the table as ctas and run the operation directly in batch insert mode.This is particularly useful when the standard table creation process fails due to partition limitations,
allowing you to work with temporary tables and persist the dataset more efficiently.lf_tags_config(default=none)AWS lakeformationtags to associate with the table and columnsenabled(default=False) whether LF tags management is enabled for a modeltagsdictionary with tags and their values to assign for the modeltags_columnsdictionary with a tag key, value and list of columns they must be assigned tolf_inherited_tags(default=none)List of Lake Formation tag keys that are intended to be inherited from the database level and thus shouldn't be
removed during association of those defined inlf_tags_configi.e., the default behavior oflf_tags_configis to be exhaustive and first remove any pre-existing tags from
tables and columns before associating the ones currently defined for a given modelThis breaks tag inheritance as inherited tags appear on tables and columns like those associated directly{{config(materialized='incremental',incremental_strategy='append',on_schema_change='append_new_columns',table_type='iceberg',schema='test_schema',lf_tags_config={'enabled':true,'tags':{'tag1':'value1','tag2':'value2'},'tags_columns':{'tag1':{'value1':['column1','column2'],'value2':['column3','column4']}},'inherited_tags':['tag1','tag2']})}}format fordbt_project.yml:+lf_tags_config:enabled:truetags:tag1:value1tag2:value2tags_columns:tag1:value1:[column1,column2]inherited_tags:[tag1,tag2]lf_grants(default=none)lakeformation grants config for data_cell filtersformat:lf_grants={'data_cell_filters':{'enabled':True|False,'filters':{'filter_name':{'row_filter':'<filter_condition>','principals':['principal_arn1','principal_arn2']}}}}Notes:lf_tagsandlf_tags_columnsconfigs support only attaching lf tags to corresponding resources.
We recommend managing LF Tags permissions somewhere outside dbt. For example, you may useterraformoraws cdkfor such purpose.data_cell_filtersmanagement can't be automated outside dbt because the filter can't be attached to the table
which doesn't exist. Once youenablethis config, dbt will set all filters and their permissions during every
dbt run. Such approach keeps the actual state of row level security configuration actual after every dbt run and
apply changes if they occur: drop, create, update filters and their permissions.Any tags listed inlf_inherited_tagsshould be strictly inherited from the database level and never overridden at
the table and column levelCurrentlydbt-athenadoes not differentiate between an inherited tag association and an override of same it made
previouslye.g. If an inherited tag is overridden by anlf_tags_configvalue in one DBT run, and that override is removed
prior to a subsequent run, the prior override will linger and no longer be encoded anywhere (in e.g. Terraform
where the inherited value is configured nor in the DBT project where the override previously existed but now is
gone)Table locationThe location in which a table is saved is determined by:Ifexternal_locationis defined, that value is used.Ifs3_data_diris defined, the path is determined by that ands3_data_namingIfs3_data_diris not defined, data is stored unders3_staging_dir/tables/Here all the options available fors3_data_naming:unique:{s3_data_dir}/{uuid4()}/table:{s3_data_dir}/{table}/table_unique:{s3_data_dir}/{table}/{uuid4()}/schema_table:{s3_data_dir}/{schema}/{table}/s3_data_naming=schema_table_unique:{s3_data_dir}/{schema}/{table}/{uuid4()}/It's possible to set thes3_data_namingglobally in the target profile, or overwrite the value in the table config,
or setting up the value for groups of model in dbt_project.yml.Note: when using a workgroup with a default output location configured,s3_data_namingand any configured buckets
are ignored and the location configured in the workgroup is used.Incremental modelsSupport forincremental models.These strategies are supported:insert_overwrite(default): The insert overwrite strategy deletes the overlapping partitions from the destination
table, and then inserts the new records from the source. This strategy depends on thepartitioned_bykeyword! If no
partitions are defined, dbt will fall back to theappendstrategy.append: Insert new records without updating, deleting or overwriting any existing data. There might be duplicate
data (e.g. great for log or historical data).merge: Conditionally updates, deletes, or inserts rows into an Iceberg table. Used in combination withunique_key.
Only available when using Iceberg.On schema changeon_schema_changeis an option to reflect changes of schema in incremental models.
The following options are supported:ignore(default)failappend_new_columnssync_all_columnsFor details, please refer
todbt docs.IcebergThe adapter supports table materialization for Iceberg.To get started just add this as your model:{{config(materialized='table',table_type='iceberg',format='parquet',partitioned_by=['bucket(user_id, 5)'],table_properties={'optimize_rewrite_delete_file_threshold':'2'})}}select'A'asuser_id,'pi'asname,'active'asstatus,17.89ascost,1asquantity,100000000asquantity_big,current_dateasmy_dateIceberg supports bucketing as hidden partitions, therefore use thepartitioned_byconfig to add specific bucketing
conditions.Iceberg supports several table formats for data :PARQUET,AVROandORC.It is possible to use Iceberg in an incremental fashion, specifically two strategies are supported:append: New records are appended to the table, this can lead to duplicates.merge: Performs an upsert (and optional delete), where new records are added and existing records are updated. Only
available with Athena engine version 3.unique_key(required): columns that define a unique record in the source and target tables.incremental_predicates(optional): SQL conditions that enable custom join clauses in the merge statement. This can
be useful for improving performance via predicate pushdown on the target table.delete_condition(optional): SQL condition used to identify records that should be deleted.update_condition(optional): SQL condition used to identify records that should be updated.insert_condition(optional): SQL condition used to identify records that should be inserted.incremental_predicates,delete_condition,update_conditionandinsert_conditioncan include any column of
the incremental table (src) or the final table (target).
Column names must be prefixed by eithersrcortargetto prevent aColumn is ambiguouserror.delete_conditionexample:{{config(materialized='incremental',table_type='iceberg',incremental_strategy='merge',unique_key='user_id',incremental_predicates=["src.quantity > 1","target.my_date >= now() - interval '4' year"],delete_condition="src.status != 'active' and target.my_date < now() - interval '2' year",format='parquet')}}select'A'asuser_id,'pi'asname,'active'asstatus,17.89ascost,1asquantity,100000000asquantity_big,current_dateasmy_dateupdate_conditionexample:{{config(materialized='incremental',incremental_strategy='merge',unique_key=['id'],update_condition='target.id > 1',schema='sandbox')}}{%ifis_incremental()%}select*from(values(1,'v1-updated'),(2,'v2-updated'))ast(id,value){%else%}select*from(values(-1,'v-1'),(0,'v0'),(1,'v1'),(2,'v2'))ast(id,value){%endif%}insert_conditionexample:{{config(materialized='incremental',incremental_strategy='merge',unique_key=['id'],insert_condition='target.status != 0',schema='sandbox')}}select*from(values(1,0),(2,1))ast(id,status)Highly available table (HA)The current implementation of the table materialization can lead to downtime, as target table is dropped and re-created.
To have the less destructive behavior it's possible to use thehaconfig on yourtablematerialized models.
It leverages the table versions feature of glue catalog, creating a tmp table and swapping the target table to the
location of the tmp table. This materialization is only available fortable_type=hiveand requires using unique
locations. For iceberg, high availability is by default.{{config(materialized='table',ha=true,format='parquet',table_type='hive',partitioned_by=['status'],s3_data_naming='table_unique')}}select'a'asuser_id,'pi'asuser_name,'active'asstatusunionallselect'b'asuser_id,'sh'asuser_name,'disabled'asstatusBy default, the materialization keeps the last 4 table versions, you can change it by settingversions_to_keep.HA Known issuesWhen swapping from a table with partitions to a table without (and the other way around), there could be a little
downtime.
In case high performances are needed consider bucketing instead of partitionsBy default, Glue "duplicates" the versions internally, so the last two versions of a table point to the same locationIt's recommended to haveversions_to_keep>= 4, as this will avoid having the older location removedThe macroathena__end_of_timeneeds to be overwritten by the user if using Athena engine v3 since it requires a
precision parameter for timestampsSnapshotsThe adapter supports snapshot materialization. It supports both timestamp and check strategy. To create a snapshot
create a snapshot file in the snapshots directory. If the directory does not exist create one.Timestamp strategyTo use the timestamp strategy refer to
thedbt docsCheck strategyTo use the check strategy refer to thedbt docsHard-deletesThe materialization also supports invalidating hard deletes. Check
thedocsto understand usage.AWS Lakeformation integrationThe adapter implements AWS Lakeformation tags management in the following way:you can enable or disable lf-tags management viaconfig(disabled by default)once you enable the feature, lf-tags will be updated on every dbt runfirst, all lf-tags for columns are removed to avoid inheritance issuesthen all redundant lf-tags are removed from table and actual tags from config are appliedfinally, lf-tags for columns are appliedIt's important to understand the following points:dbt does not manage lf-tags for databasedbt does not manage lakeformation permissionsThat's why you should handle this by yourself manually or using some automation tools like terraform, AWS CDK etc.You may find the following links useful to manage that:terraform aws_lakeformation_permissionsterraform aws_lakeformation_resource_lf_tagsPython ModelsThe adapter supports python models usingspark.SetupA spark enabled work group created in athenaSpark execution role granted access to Athena, Glue and S3The spark work group is added to the ~/.dbt/profiles.yml file and the profile is referenced in dbt_project.yml
that will be created. It is recommended to keep this same as threads.Spark specific table configurationtimeout(default=43200)Time out in seconds for each python model execution. Defaults to 12 hours/43200 seconds.spark_encryption(default=false)If this flag is set to true, encrypts data in transit between Spark nodes and also encrypts data at rest stored
locally by Spark.spark_cross_account_catalog(default=false)In spark, you can query the external account catalog and for that the consumer account has to be configured to
access the producer catalog.If this flag is set to true, "/" can be used as the glue catalog separator.
Ex: 999999999999/mydatabase.cloudfront_logs (where999999999999is the external catalog id)spark_requester_pays(default=false)When an Amazon S3 bucket is configured as requester pays, the account of the user running the query is charged for
data access and data transfer fees associated with the query.If this flag is set to true, requester pays S3 buckets are enabled in Athena for Spark.Spark notesA session is created for each unique engine configuration defined in the models that are part of the invocation.A session's idle timeout is set to 10 minutes. Within the timeout period, if there is a new calculation
(spark python model) ready for execution and the engine configuration matches, the process will reuse the same session.Number of python models running at a time depends on thethreads.  Number of sessions created for the entire run
depends on number of unique engine configurations and availability of session to maintain threads concurrency.For iceberg table, it is recommended to use table_properties configuration to set the format_version to 2. This is to
maintain compatability between iceberg tables created by Trino with those created by Spark.Example modelsSimple pandas modelimportpandasaspddefmodel(dbt,session):dbt.config(materialized="table")model_df=pd.DataFrame({"A":[1,2,3,4]})returnmodel_dfSimple sparkdefmodel(dbt,spark_session):dbt.config(materialized="table")data=[(1,),(2,),(3,),(4,)]df=spark_session.createDataFrame(data,["A"])returndfSpark incrementaldefmodel(dbt,spark_session):dbt.config(materialized="incremental")df=dbt.ref("model")ifdbt.is_incremental:max_from_this=(f"select max(run_date) from{dbt.this.schema}.{dbt.this.identifier}")df=df.filter(df.run_date>=spark_session.sql(max_from_this).collect()[0][0])returndfConfig spark modeldefmodel(dbt,spark_session):dbt.config(materialized="table",engine_config={"CoordinatorDpuSize":1,"MaxConcurrentDpus":3,"DefaultExecutorDpuSize":1},spark_encryption=True,spark_cross_account_catalog=True,spark_requester_pays=Truepolling_interval=15,timeout=120,)data=[(1,),(2,),(3,),(4,)]df=spark_session.createDataFrame(data,["A"])returndfCreate pySpark udf using imported external python filesdefmodel(dbt,spark_session):dbt.config(materialized="incremental",incremental_strategy="merge",unique_key="num",)sc=spark_session.sparkContextsc.addPyFile("s3://athena-dbt/test/file1.py")sc.addPyFile("s3://athena-dbt/test/file2.py")deffunc(iterator):fromfile2importtransformreturn[transform(i)foriiniterator]frompyspark.sql.functionsimportudffrompyspark.sql.functionsimportcoludf_with_import=udf(func)data=[(1,"a"),(2,"b"),(3,"c")]cols=["num","alpha"]df=spark_session.createDataFrame(data,cols)returndf.withColumn("udf_test_col",udf_with_import(col("alpha")))Known issues in python modelsIncremental models do not fully utilize spark capabilities. They depend partially on existing sql based logic which
runs on trino.Snapshots materializations are not supported.Spark can only reference tables within the same catalog.Working exampleseed file - employent_indicators_november_2022_csv_tables.csvSeries_reference,Period,Data_value,Suppressed
MEIM.S1WA,1999.04,80267,
MEIM.S1WA,1999.05,70803,
MEIM.S1WA,1999.06,65792,
MEIM.S1WA,1999.07,66194,
MEIM.S1WA,1999.08,67259,
MEIM.S1WA,1999.09,69691,
MEIM.S1WA,1999.1,72475,
MEIM.S1WA,1999.11,79263,
MEIM.S1WA,1999.12,86540,
MEIM.S1WA,2000.01,82552,
MEIM.S1WA,2000.02,81709,
MEIM.S1WA,2000.03,84126,
MEIM.S1WA,2000.04,77089,
MEIM.S1WA,2000.05,73811,
MEIM.S1WA,2000.06,70070,
MEIM.S1WA,2000.07,69873,
MEIM.S1WA,2000.08,71468,
MEIM.S1WA,2000.09,72462,
MEIM.S1WA,2000.1,74897,model.sql{{config(materialized='table')}}selectrow_number()over()asid,*,cast(from_unixtime(to_unixtime(now()))astimestamp(6))asrefresh_timestampfrom{{ref('employment_indicators_november_2022_csv_tables')}}timestamp strategy - model_snapshot_1{%snapshotmodel_snapshot_1%}{{config(strategy='timestamp',updated_at='refresh_timestamp',unique_key='id')}}select*from{{ref('model')}}{%endsnapshot%}invalidate hard deletes - model_snapshot_2{%snapshotmodel_snapshot_2%}{{config(unique_key='id',strategy='timestamp',updated_at='refresh_timestamp',invalidate_hard_deletes=True,)}}select*from{{ref('model')}}{%endsnapshot%}check strategy - model_snapshot_3{%snapshotmodel_snapshot_3%}{{config(unique_key='id',strategy='check',check_cols=['series_reference','data_value'])}}select*from{{ref('model')}}{%endsnapshot%}Snapshots Known issuesIncremental Iceberg models - Sync all columns on schema change can't remove columns used as partitioning.
The only way, from a dbt perspective, is to do a full-refresh of the incremental model.Tables, schemas and database should only be lowercaseIn order to avoid potential conflicts, make suredbt-athena-adapteris not
installed in the target environment.
Seehttps://github.com/dbt-athena/dbt-athena/issues/103for more details.Snapshot does not support dropping columns from the source table. If you drop a column make sure to drop the column
from the snapshot as well. Another workaround is to NULL the column in the snapshot definition to preserve historyContractsThe adapter partly supports contract definition.Concerning thedata_type, it is supported but needs to be adjusted for complex types. They must be specified
entirely (for instancearray<int>) even though they won't be checked. Indeed, as dbt recommends, we only compare
the broader type (array, map, int, varchar). The complete definition is used in order to check that the data types
defined in athena are ok (pre-flight check).the adapter does not support the constraints since no constraints don't exist in Athena.ContributingSeeCONTRIBUTINGfor more information on how to contribute to this project.Contributors ✨Thanks goes to these wonderful people (emoji key):nicor88💻🚧🐛Jesse Dobbelaere🐛🚧Lemiffe🎨Jérémy Guiselin🚧💻🐛Tom🚧💻Mattia🚧Gatsby Lee🐛BrechtDeVlieger🐛Andrea Artaria🐛Maiara Reinaldo🐛Henri Blancke💻🐛Serhii Dimchenko💻🐛chrischin478💻🐛Roman Korsun💻🐛DanyaF💻🐛Spencer💻Avinash Santhanagopalan💻This project follows theall-contributorsspecification.
Contributions of any kind welcome!