Package: dbt-glue
dbtenables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.dbt-glueThedbt-gluepackage implements thedbt adapterprotocol for AWS Glue's Spark engine.
It supports running dbt against Spark, through the new Glue Interactive Sessions API.To learn how to deploy a data pipeline in your modern data platform using thedbt-glueadapter, please read the following blog post:Build your data pipeline in your AWS modern data platform using AWS Lake Formation, AWS Glue, and dbt CoreInstallationThe package can be installed from PyPI with:$pip3installdbt-glueFor further (and more likely up-to-date) info, see theREADMEConnection MethodsConfiguring your AWS profile for Glue Interactive SessionThere are two IAM principals used with interactive sessions.Client principal: The princpal (either user or role) calling the AWS APIs (Glue, Lake Formation, Interactive Sessions)
from the local client. This is the principal configured in the AWS CLI and likely the same.Service role: The IAM role that AWS Glue uses to execute your session. This is the same as AWS Glue
ETL.Readthis documentationto configure these principals.You will find bellow a least privileged policy to enjoy all features ofdbt-glueadapter.Please to update variables between<>, here are explanations of these arguments:ArgsDescriptionregionThe region where your Glue database is storedAWS AccountThe AWS account where you run your pipelinedbt output databaseThe database updated by dbt (this is the schema configured in the profile.yml of your dbt environment)dbt source databaseAll databases used as sourcedbt output bucketThe bucket name where the data will be generate dbt (the location configured in the profile.yml of your dbt environment)dbt source bucketThe bucket name of source databases (if they are not managed by Lake Formation){"Version":"2012-10-17","Statement":[{"Sid":"Read_and_write_databases","Action":["glue:SearchTables","glue:BatchCreatePartition","glue:CreatePartitionIndex","glue:DeleteDatabase","glue:GetTableVersions","glue:GetPartitions","glue:DeleteTableVersion","glue:UpdateTable","glue:DeleteTable","glue:DeletePartitionIndex","glue:GetTableVersion","glue:UpdateColumnStatisticsForTable","glue:CreatePartition","glue:UpdateDatabase","glue:CreateTable","glue:GetTables","glue:GetDatabases","glue:GetTable","glue:GetDatabase","glue:GetPartition","glue:UpdateColumnStatisticsForPartition","glue:CreateDatabase","glue:BatchDeleteTableVersion","glue:BatchDeleteTable","glue:DeletePartition","glue:GetUserDefinedFunctions","lakeformation:ListResources","lakeformation:BatchGrantPermissions","lakeformation:ListPermissions","lakeformation:GetDataAccess","lakeformation:GrantPermissions","lakeformation:RevokePermissions","lakeformation:BatchRevokePermissions","lakeformation:AddLFTagsToResource","lakeformation:RemoveLFTagsFromResource","lakeformation:GetResourceLFTags","lakeformation:ListLFTags","lakeformation:GetLFTag",],"Resource":["arn:aws:glue:<region>:<AWSAccount>:catalog","arn:aws:glue:<region>:<AWSAccount>:table/<dbtoutputdatabase>/*","arn:aws:glue:<region>:<AWSAccount>:database/<dbtoutputdatabase>"],"Effect":"Allow"},{"Sid":"Read_only_databases","Action":["glue:SearchTables","glue:GetTableVersions","glue:GetPartitions","glue:GetTableVersion","glue:GetTables","glue:GetDatabases","glue:GetTable","glue:GetDatabase","glue:GetPartition","lakeformation:ListResources","lakeformation:ListPermissions"],"Resource":["arn:aws:glue:<region>:<AWSAccount>:table/<dbtsourcedatabase>/*","arn:aws:glue:<region>:<AWSAccount>:database/<dbtsourcedatabase>","arn:aws:glue:<region>:<AWSAccount>:database/default","arn:aws:glue:<region>:<AWSAccount>:database/global_temp"],"Effect":"Allow"},{"Sid":"Storage_all_buckets","Action":["s3:GetBucketLocation","s3:ListBucket"],"Resource":["arn:aws:s3:::<dbtoutputbucket>","arn:aws:s3:::<dbtsourcebucket>"],"Effect":"Allow"},{"Sid":"Read_and_write_buckets","Action":["s3:PutObject","s3:PutObjectAcl","s3:GetObject","s3:DeleteObject"],"Resource":["arn:aws:s3:::<dbtoutputbucket>"],"Effect":"Allow"},{"Sid":"Read_only_buckets","Action":["s3:GetObject"],"Resource":["arn:aws:s3:::<dbtsourcebucket>"],"Effect":"Allow"}]}Configuration of the local environmentBecausedbtanddbt-glueadapter are compatible with Python versions 3.7, 3.8, and 3.9, check the version of Python:$python3--versionConfigure a Python virtual environment to isolate package version and code dependencies:$sudoyuminstallgit
$python3-mvenvdbt_venv
$sourcedbt_venv/bin/activate
$python3-mpipinstall--upgradepipConfigure the last version of AWS CLI$curl"https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip"-o"awscliv2.zip"$unzipawscliv2.zip
$sudo./aws/installInstall boto3 package$sudoyuminstallgcckrb5-devel.x86_64python3-devel.x86_64-y
$pip3install—upgradeboto3Install the package:$pip3installdbt-glueExample configtype: glue
query-comment: This is a glue dbt example
role_arn: arn:aws:iam::1234567890:role/GlueInteractiveSessionRole
region: us-east-1
workers: 2
worker_type: G.1X
idle_timeout: 10
schema: "dbt_demo"
session_provisioning_timeout_in_seconds: 120
location: "s3://dbt_demo_bucket/dbt_demo_data"The table below describes all the options.OptionDescriptionMandatoryproject_nameThe dbt project name. This must be the same as the one configured in the dbt project.yestypeThe driver to use.yesquery-commentA string to inject as a comment in each query that dbt runs.norole_arnThe ARN of the glue interactive session IAM role.yesregionThe AWS Region were you run the data pipeline.yesworkersThe number of workers of a defined workerType that are allocated when a job runs.yesworker_typeThe type of predefined worker that is allocated when a job runs. Accepts a value of Standard, G.1X, or G.2X.yesschemaThe schema used to organize data stored in Amazon S3.Additionally, is the database in AWS Lake Formation that stores metadata tables in the Data Catalog.yessession_provisioning_timeout_in_secondsThe timeout in seconds for AWS Glue interactive session provisioning.yeslocationThe Amazon S3 location of your target data.yesquery_timeout_in_minutesThe timeout in minutes for a signle query. Default is 300noidle_timeoutThe AWS Glue session idle timeout in minutes. (The session stops after being idle for the specified amount of time)noglue_versionThe version of AWS Glue for this session to use. Currently, the only valid options are 2.0, 3.0 and 4.0. The default value is 4.0.nosecurity_configurationThe security configuration to use with this session.noconnectionsA comma-separated list of connections to use in the session.noconfSpecific configuration used at the startup of the Glue Interactive Session (arg --conf)noextra_py_filesExtra python Libs that can be used by the interactive session.nodelta_athena_prefixA prefix used to create Athena compatible tables for Delta tables	(if not specified, then no Athena compatible table will be created)notagsThe map of key value pairs (tags) belonging to the session. Ex:KeyName1=Value1,KeyName2=Value2noseed_formatBy defaultparquet, can be Spark format compatible likecsvorjsonnoseed_modeBy defaultoverwrite, the seed data will be overwritten, you can set it toappendif you just want to add new data in your datasetnodefault_argumentsThe map of key value pairs parameters belonging to the session. More information onJob parameters used by AWS Glue. Ex:--enable-continuous-cloudwatch-log=true,--enable-continuous-log-filter=truenoglue_session_idre-use a glue-session to run multiple dbt run commands. Will create a new glue-session using glue_session_id if it does not exists yet.noglue_session_reusere-use the glue-session to run multiple dbt run commands: If set to true, the glue session will not be closed for re-use. If set to false, the session will be closed. The glue session will close after idle_timeout time is expired after idle_timeout timenodatalake_formatsThe ACID datalake format that you want to use if you are doing merge, can behudi,ìcebergordeltanouse_arrow(experimental) use an arrow file instead of stdout to have better scalability.noConfigsConfiguring tablesWhen materializing a model astable, you may include several optional configs that are specific to the dbt-spark plugin, in addition to the standardmodel configs.OptionDescriptionRequired?Examplefile_formatThe file format to use when creating tables (parquet,csv,json,text,jdbcororc).Optionalparquetpartition_byPartition the created table by the specified columns. A directory is created for each partition.Optionaldate_dayclustered_byEach partition in the created table will be split into a fixed number of buckets by the specified columns.Optionalcountry_codebucketsThe number of buckets to create while clusteringRequired ifclustered_byis specified8custom_locationBy default, the adapter will store your data in the following path:location path/schema/table. If you don't want to follow that default behaviour, you can use this parameter to set your own custom location on S3Nos3://mycustombucket/mycustompathhudi_optionsWhen using file_formathudi, gives the ability to overwrite any of the default configuration options.Optional{'hoodie.schema.on.read.enable': 'true'}Incremental modelsdbt seeks to offer useful and intuitive modeling abstractions by means of its built-in configurations and materializations.For that reason, the dbt-glue plugin leans heavily on theincremental_strategyconfig. This config tells the incremental materialization how to build models in runs beyond their first. It can be set to one of three values:append(default): Insert new records without updating or overwriting any existing data.insert_overwrite: Ifpartition_byis specified, overwrite partitions in the table with new data. If nopartition_byis specified, overwrite the entire table with new data.merge(Apache Hudi and Apache Iceberg only): Match records based on aunique_key; update old records, insert new ones. (If nounique_keyis specified, all new data is inserted, similar toappend.)Each of these strategies has its pros and cons, which we'll discuss below. As with any model config,incremental_strategymay be specified indbt_project.ymlor within a model file'sconfig()block.Notes:The default strategy isinsert_overwriteTheappendstrategyFollowing theappendstrategy, dbt will perform aninsert intostatement with all new data. The appeal of this strategy is that it is straightforward and functional across all platforms, file types, connection methods, and Apache Spark versions. However, this strategycannotupdate, overwrite, or delete existing data, so it is likely to insert duplicate records for many data sources.Source code{{config(materialized='incremental',incremental_strategy='append',)}}--  All rows returned by this query will be appended to the existing tableselect*from{{ref('events')}}{%ifis_incremental()%}whereevent_ts>(selectmax(event_ts)from{{this}}){%endif%}Run Codecreatetemporaryviewspark_incremental__dbt_tmpasselect*fromanalytics.eventswhereevent_ts>=(selectmax(event_ts)from{{this}});insertintotableanalytics.spark_incrementalselect`date_day`,`users`fromspark_incremental__dbt_tmpTheinsert_overwritestrategyThis strategy is most effective when specified alongside apartition_byclause in your model config. dbt will run anatomicinsert overwritestatementthat dynamically replaces all partitions included in your query. Be sure to re-selectallof the relevant data for a partition when using this incremental strategy.If nopartition_byis specified, then theinsert_overwritestrategy will atomically replace all contents of the table, overriding all existing data with only the new records. The column schema of the table remains the same, however. This can be desirable in some limited circumstances, since it minimizes downtime while the table contents are overwritten. The operation is comparable to runningtruncate+inserton other databases. For atomic replacement of Delta-formatted tables, use thetablematerialization (which runscreate or replace) instead.Source Code{{config(materialized='incremental',partition_by=['date_day'],file_format='parquet')}}/*Every partition returned by this query will be overwrittenwhen this model runs*/withnew_eventsas(select*from{{ref('events')}}{%ifis_incremental()%}wheredate_day>=date_add(current_date,-1){%endif%})selectdate_day,count(*)asusersfromeventsgroupby1Run Codecreatetemporaryviewspark_incremental__dbt_tmpaswithnew_eventsas(select*fromanalytics.eventswheredate_day>=date_add(current_date,-1))selectdate_day,count(*)asusersfromeventsgroupby1;insertoverwritetableanalytics.spark_incrementalpartition(date_day)select`date_day`,`users`fromspark_incremental__dbt_tmpSpecifyinginsert_overwriteas the incremental strategy is optional, since it's the default strategy used when none is specified.ThemergestrategyCompatibility:Hudi : OKDelta Lake : OKIceberg : OKLake Formation Governed Tables : On goingNB:For Glue 3: you have to setup aGlue connectors.For Glue 4: use thedatalake_formatsoption in your profile.ymlWhen using a connector be sure that your IAM role has these policies:{
    "Sid": "access_to_connections",
    "Action": [
        "glue:GetConnection",
        "glue:GetConnections"
    ],
    "Resource": [
        "arn:aws:glue:<region>:<AWS Account>:catalog",
        "arn:aws:glue:<region>:<AWS Account>:connection/*"
    ],
    "Effect": "Allow"
}and that the managed policyAmazonEC2ContainerRegistryReadOnlyis attached.
Be sure that you follow the getting started instructionshere.Thisblog postalso explain how to setup and works with Glue ConnectorsHudiUsage notes:Themergewith Hudi incremental strategy requires:To addfile_format: hudiin your table configurationTo add a datalake_formats in your profile :datalake_formats: hudiAlternatively, to add a connections in your profile :connections: name_of_your_hudi_connectorTo add Kryo serializer in your Interactive Session Config (in your profile):conf: spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.hive.convertMetastoreParquet=falsedbt will run anatomicmergestatementwhich looks nearly identical to the default merge behavior on Snowflake and BigQuery. If aunique_keyis specified (recommended), dbt will update old records with values from new records that match on the key column. If aunique_keyis not specified, dbt will forgo match criteria and simply insert all new records (similar toappendstrategy).Profile config exampletest_project:target:devoutputs:dev:type:gluequery-comment:my commentrole_arn:arn:aws:iam::1234567890:role/GlueInteractiveSessionRoleregion:eu-west-1glue_version:"4.0"workers:2worker_type:G.1Xschema:"dbt_test_project"session_provisioning_timeout_in_seconds:120location:"s3://aws-dbt-glue-datalake-1234567890-eu-west-1/"conf:spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.hive.convertMetastoreParquet=falsedatalake_formats:hudiSource Code example{{config(materialized='incremental',incremental_strategy='merge',unique_key='user_id',file_format='hudi',hudi_options={'hoodie.datasource.write.precombine.field':'eventtime',})}}withnew_eventsas(select*from{{ref('events')}}{%ifis_incremental()%}wheredate_day>=date_add(current_date,-1){%endif%})selectuser_id,max(date_day)aslast_seenfromeventsgroupby1DeltaYou can also use Delta Lake to be able to use merge feature on tables.Usage notes:Themergewith Delta incremental strategy requires:To addfile_format: deltain your table configurationTo add a datalake_formats in your profile :datalake_formats: deltaAlternatively, to add a connections in your profile :connections: name_of_your_delta_connectorTo add the following config in your Interactive Session Config (in your profile):conf: "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalogAthena:Athena is not compatible by default with delta tables, but you can configure the adapter to create Athena tables on top of your delta table. To do so, you need to configure the two following options in your profile:delta_athena_prefix: "the_prefix_of_your_choice"If your table is partitioned, then the add of new partition is not automatic, you need to perform anMSCK REPAIR TABLE your_delta_tableafter each new partition addingProfile config exampletest_project:target:devoutputs:dev:type:gluequery-comment:my commentrole_arn:arn:aws:iam::1234567890:role/GlueInteractiveSessionRoleregion:eu-west-1glue_version:"4.0"workers:2worker_type:G.1Xschema:"dbt_test_project"session_provisioning_timeout_in_seconds:120location:"s3://aws-dbt-glue-datalake-1234567890-eu-west-1/"datalake_formats:deltaconf:"spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension--confspark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"delta_athena_prefix:"delta"Source Code example{{config(materialized='incremental',incremental_strategy='merge',unique_key='user_id',partition_by=['dt'],file_format='delta')}}withnew_eventsas(select*from{{ref('events')}}{%ifis_incremental()%}wheredate_day>=date_add(current_date,-1){%endif%})selectuser_id,max(date_day)aslast_seen,current_date()asdtfromeventsgroupby1IcebergUsage notes:Themergewith Iceberg incremental strategy requires:To addfile_format: Icebergin your table configurationTo add a datalake_formats in your profile :datalake_formats: icebergAlternatively, if you use Glue 3.0 or more, to add a connections in your profile :connections: name_of_your_iceberg_connector(For Athena version 3:The adapter is compatible with the Iceberg Connector from AWS Marketplace with Glue 3.0 as Fulfillment option and 0.14.0 (Oct 11, 2022) as Software version)the latest connector for iceberg in AWS marketplace uses Ver 0.14.0 for Glue 3.0, and Ver 1.2.1 for Glue 4.0 where Kryo serialization fails when writing iceberg, use "org.apache.spark.serializer.JavaSerializer" for spark.serializer instead, more infohereFor Athena version 2: The adapter is compatible with the Iceberg Connector from AWS Marketplace with Glue 3.0 as Fulfillment option and 0.12.0-2 (Feb 14, 2022) as Software version)For Glue 4.0, to add the following configurations in dbt-profile:--conf spark.sql.catalog.glue_catalog=org.apache.iceberg.spark.SparkCatalog
    --conf spark.sql.catalog.glue_catalog.warehouse=s3://<PATH_TO_YOUR_WAREHOUSE>
    --conf spark.sql.catalog.glue_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog 
    --conf spark.sql.catalog.glue_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO 
    --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensionsFor Glue 3.0, you need to set up more configurations :--conf spark.serializer=org.apache.spark.serializer.KryoSerializer
    --conf spark.sql.warehouse=s3://<your-bucket-name>
    --conf spark.sql.catalog.glue_catalog=org.apache.iceberg.spark.SparkCatalog 
    --conf spark.sql.catalog.glue_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog 
    --conf spark.sql.catalog.glue_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO 
    --conf spark.sql.catalog.glue_catalog.lock-impl=org.apache.iceberg.aws.glue.DynamoDbLockManager
    --conf spark.sql.catalog.glue_catalog.lock.table=myGlueLockTable  
    --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensionsAlso note that for Glue 4.0, you can choose between Glue Optimistic Locking (enabled by default) and DynamoDB Lock Manager for concurrent update to a table.If you want to activate DynamoDB Lock Manager set the below config in your profiles. A DynamoDB would be created on your behalf (if it does not exist).--conf spark.sql.catalog.glue_catalog.lock-impl=org.apache.iceberg.aws.dynamodb.DynamoDbLockManager
    --conf spark.sql.catalog.glue_catalog.lock.table=<DYNAMODB_TABLE_NAME>You'll also need to grant the dbt-glue execution role with the appropriate permissions on DynamoDB{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CommitLockTable",
            "Effect": "Allow",
            "Action": [
                "dynamodb:CreateTable",
                "dynamodb:BatchGetItem",
                "dynamodb:BatchWriteItem",
                "dynamodb:ConditionCheckItem",
                "dynamodb:PutItem",
                "dynamodb:DescribeTable",
                "dynamodb:DeleteItem",
                "dynamodb:GetItem",
                "dynamodb:Scan",
                "dynamodb:Query",
                "dynamodb:UpdateItem"
            ],
            "Resource": "arn:aws:dynamodb:<AWS_REGION>:<AWS_ACCOUNT_ID>:table/<DYNAMODB_TABLE_NAME>"
        }
    ]
}Note that if you use Glue 3.0 DynamoDB Lock Manager is the only option available and you need to setorg.apache.iceberg.aws.glue.DynamoLockManagerinstead :--conf spark.sql.catalog.glue_catalog.lock-impl=org.apache.iceberg.aws.glue.DynamoDbLockManager
    --conf spark.sql.catalog.glue_catalog.lock.table=myGlueLockTabledbt will run anatomicmergestatementwhich looks nearly identical to the default merge behavior on Snowflake and BigQuery. You need to provide aunique_keyto perform merge operation otherwise it will fail. This key is to provide in a Python list format and can contains multiple column name to create a composite unique_key.NotesWhen using a custom_location in Iceberg, avoid to use final trailing slash. Adding a final trailing slash lead to an un-proper handling of the location, and issues when reading the data from query engines like Trino. The issue should be fixed for Iceberg version > 0.13. Related Github issue can be findhere.Iceberg also supportsinsert_overwriteandappendstrategies.Thewarehouseconf must be provided, but it's overwritten by the adapterlocationin your profile orcustom_locationin model configuration.By default, this materialization hasiceberg_expire_snapshotsset to 'True', if you need to have historical auditable changes, set:iceberg_expire_snapshots='False'.Currently, due to some dbt internal, the iceberg catalog used internally when running glue interactive sessions with dbt-glue has a hardcoded nameglue_catalog. This name is an alias pointing to the AWS Glue Catalog but is specific to each session. If you want to interact with your data in another session without using dbt-glue (from a Glue Studio notebook, for example), you can configure another alias (ie. another name for the Iceberg Catalog). To illustrate this concept, you can set in your configuration file :--conf spark.sql.catalog.RandomCatalogName=org.apache.iceberg.spark.SparkCatalogAnd then run in an AWS Glue Studio Notebook a session with the following config:--conf spark.sql.catalog.AnotherRandomCatalogName=org.apache.iceberg.spark.SparkCatalogIn both cases, the underlying catalog would be the AWS Glue Catalog, unique in your AWS Account and Region, and you would be able to work with the exact same data. Also make sure that if you change the name of the Glue Catalog Alias, you change it in all the other--confwhere it's used:--conf spark.sql.catalog.RandomCatalogName=org.apache.iceberg.spark.SparkCatalog 
 --conf spark.sql.catalog.RandomCatalogName.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog 
 ...
 --conf spark.sql.catalog.RandomCatalogName.lock-impl=org.apache.iceberg.aws.glue.DynamoLockManagerA full reference totable_propertiescan be foundhere.Iceberg Tables are natively supported by Athena. Therefore, you can query tables created and operated with dbt-glue adapter from Athena.Incremental Materialization with Iceberg file format supports dbt snapshot. You are able to run a dbt snapshot command that queries an Iceberg Table and create a dbt fashioned snapshot of it.Profile config exampletest_project:target:devoutputs:dev:type:gluequery-comment:my commentrole_arn:arn:aws:iam::1234567890:role/GlueInteractiveSessionRoleregion:eu-west-1glue_version:"4.0"workers:2worker_type:G.1Xschema:"dbt_test_project"session_provisioning_timeout_in_seconds:120location:"s3://aws-dbt-glue-datalake-1234567890-eu-west-1/"datalake_formats:icebergconf:--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.warehouse=s3://aws-dbt-glue-datalake-1234567890-eu-west-1/dbt_test_project --conf spark.sql.catalog.glue_catalog=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.glue_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog --conf spark.sql.catalog.glue_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO --conf spark.sql.catalog.glue_catalog.lock-impl=org.apache.iceberg.aws.dynamodb.DynamoDbLockManager --conf spark.sql.catalog.glue_catalog.lock.table=myGlueLockTable  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensionsSource Code example{{config(materialized='incremental',incremental_strategy='merge',unique_key=['user_id'],file_format='iceberg',iceberg_expire_snapshots='False',partition_by=['status']table_properties={'write.target-file-size-bytes':'268435456'})}}withnew_eventsas(select*from{{ref('events')}}{%ifis_incremental()%}wheredate_day>=date_add(current_date,-1){%endif%})selectuser_id,max(date_day)aslast_seenfromeventsgroupby1Iceberg Snapshot source code example{%snapshotdemosnapshot%}{{config(strategy='timestamp',target_schema='jaffle_db',updated_at='dt',file_format='iceberg')}}select*from{{ref('customers')}}{%endsnapshot%}Monitoring your Glue Interactive SessionMonitoring is an important part of maintaining the reliability, availability,
and performance of AWS Glue and your other AWS solutions. AWS provides monitoring
tools that you can use to watch AWS Glue, identify the required number of workers
required for your Glue Interactive Session, report when something is wrong and
take action automatically when appropriate. AWS Glue provides Spark UI,
and CloudWatch logs and metrics for monitoring your AWS Glue jobs.
More information on:Monitoring AWS Glue Spark jobsUsage notes:Monitoring requires:To add the following IAM policy to your IAM role:{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CloudwatchMetrics",
            "Effect": "Allow",
            "Action": "cloudwatch:PutMetricData",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "cloudwatch:namespace": "Glue"
                }
            }
        },
        {
            "Sid": "CloudwatchLogs",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "logs:CreateLogStream",
                "logs:CreateLogGroup",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:*:*:/aws-glue/*",
                "arn:aws:s3:::bucket-to-write-sparkui-logs/*"
            ]
        }
    ]
}To add monitoring parameters in your Interactive Session Config (in your profile).
More information onJob parameters used by AWS GlueProfile config exampletest_project:target:devoutputs:dev:type:gluequery-comment:my commentrole_arn:arn:aws:iam::1234567890:role/GlueInteractiveSessionRoleregion:eu-west-1glue_version:"4.0"workers:2worker_type:G.1Xschema:"dbt_test_project"session_provisioning_timeout_in_seconds:120location:"s3://aws-dbt-glue-datalake-1234567890-eu-west-1/"default_arguments:"--enable-metrics=true,--enable-continuous-cloudwatch-log=true,--enable-continuous-log-filter=true,--enable-spark-ui=true,--spark-event-logs-path=s3://bucket-to-write-sparkui-logs/dbt/"If you want to use the Spark UI, you can launch the Spark history server using a
AWS CloudFormation template that hosts the server on an EC2 instance,
or launch locally using Docker. More information onLaunching the Spark history serverEnabling AWS Glue Auto ScalingAuto Scaling is available since AWS Glue version 3.0 or later. More information
on the following AWS blog post:"Introducing AWS Glue Auto Scaling: Automatically resize serverless computing resources for lower cost with optimized Apache Spark"With Auto Scaling enabled, you will get the following benefits:AWS Glue automatically adds and removes workers from the cluster depending on the parallelism at each stage or microbatch of the job run.It removes the need for you to experiment and decide on the number of workers to assign for your AWS Glue Interactive sessions.Once you choose the maximum number of workers, AWS Glue will choose the right size resources for the workload.You can see how the size of the cluster changes during the Glue Interactive sessions run by looking at CloudWatch metrics.
More information onMonitoring your Glue Interactive Session.Usage notes:AWS Glue Auto Scaling requires:To set your AWS Glue version 3.0 or later.To set the maximum number of workers (if Auto Scaling is enabled, theworkersparameter sets the maximum number of workers)To set the--enable-auto-scaling=trueparameter on your Glue Interactive Session Config (in your profile).
More information onJob parameters used by AWS GlueProfile config exampletest_project:target:devoutputs:dev:type:gluequery-comment:my commentrole_arn:arn:aws:iam::1234567890:role/GlueInteractiveSessionRoleregion:eu-west-1glue_version:"4.0"workers:2worker_type:G.1Xschema:"dbt_test_project"session_provisioning_timeout_in_seconds:120location:"s3://aws-dbt-glue-datalake-1234567890-eu-west-1/"default_arguments:"--enable-auto-scaling=true"Access Glue catalog in another AWS accountIn many cases, you may need to run you dbt jobs to read from another AWS account.Review the following linkhttps://repost.aws/knowledge-center/glue-tables-cross-accountsto set up access policies in source and target accountsAdd the following "spark.hadoop.hive.metastore.glue.catalogid=" to your conf in the DBT profile, as such, you can have multiple outputs for each of the accounts that you have access to.Note: The access cross-accounts need to be within the same AWS RegionProfile config exampletest_project:target:devoutputsAccountB:dev:type:gluequery-comment:my commentrole_arn:arn:aws:iam::1234567890:role/GlueInteractiveSessionRoleregion:eu-west-1glue_version:"4.0"workers:2worker_type:G.1Xschema:"dbt_test_project"session_provisioning_timeout_in_seconds:120location:"s3://aws-dbt-glue-datalake-1234567890-eu-west-1/"conf:"--confhive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory--confspark.hadoop.hive.metastore.glue.catalogid=<TARGET-AWS-ACCOUNT-ID-B>"Persisting model descriptionsRelation-level docs persistence is supported since dbt v0.17.0. For more
information on configuring docs persistence, seethe docs.When thepersist_docsoption is configured appropriately, you'll be able to
see model descriptions in theCommentfield ofdescribe [table] extendedorshow table extended in [database] like '*'.Alwaysschema, neverdatabaseApache Spark uses the terms "schema" and "database" interchangeably. dbt understandsdatabaseto exist at a higher level thanschema. As such, you shouldneveruse or setdatabaseas a node config or in the target profile when running dbt-glue.If you want to control the schema/database in which dbt will materialize models,
use theschemaconfig andgenerate_schema_namemacroonly.
For more information, check the dbt documentation aboutcustom schemas.AWS Lakeformation integrationThe adapter supports AWS Lake Formation tags management enabling you to associate existing tags defined out of dbt-glue to database objects built by dbt-glue (database, table, view, snapshot, incremental models, seeds).You can enable or disable lf-tags management via config, at model and dbt-project level (disabled by default)If enabled, lf-tags will be updated on every dbt run. There are table level lf-tags configs and column-level lf-tags configs.You can specify that you want to drop existing database, table column Lake Formation tags by setting the drop_existing config field to True (False by default, meaning existing tags are kept)Please note that if the tag you want to associate with the table does not exist, the dbt-glue execution will throw an errorThe adapter also supports AWS Lakeformation data cell filtering.You can enable or disable data-cell filtering via config, at model and dbt-project level (disabled by default)If enabled, data_cell_filters will be updated on every dbt run.You can specify that you want to drop existing table data-cell filters by setting the drop_existing config field to True (False by default, meaning existing filters are kept)You can leverage excluded_columns_namesORcolumns config fields to perform Column level security as well.Please note that you can use one or the other but not both.By default, if you don't specify any column or excluded_columns, dbt-glue does not perform Column level filtering and let the principal access all the columns.The below configuration let the specified principal (lf-data-scientist IAM user) access rows that have a customer_lifetime_value > 15 and all the columns specified ('customer_id', 'first_order', 'most_recent_order', 'number_of_orders')lf_grants={'data_cell_filters':{'enabled':True,'drop_existing':True,'filters':{'the_name_of_my_filter':{'row_filter':'customer_lifetime_value>15','principals':['arn:aws:iam::123456789:user/lf-data-scientist'],'column_names':['customer_id','first_order','most_recent_order','number_of_orders']}},}}The below configuration let the specified principal (lf-data-scientist IAM user) access rows that have a customer_lifetime_value > 15 and all the columnsexceptthe one specified ('first_name')lf_grants={'data_cell_filters':{'enabled':True,'drop_existing':True,'filters':{'the_name_of_my_filter':{'row_filter':'customer_lifetime_value>15','principals':['arn:aws:iam::123456789:user/lf-data-scientist'],'excluded_column_names':['first_name']}},}}See below some examples of how you can integrate LF Tags management and data cell filtering to your configurations :At model levelThis way of defining your Lakeformation rules is appropriate if you want to handle the tagging and filtering policy at object level. Remember that it overrides any configuration defined at dbt-project level.{{config(materialized='incremental',unique_key="customer_id",incremental_strategy='append',lf_tags_config={'enabled':true,'drop_existing':False,'tags_database':{'name_of_my_db_tag':'value_of_my_db_tag'},'tags_table':{'name_of_my_table_tag':'value_of_my_table_tag'},'tags_columns':{'name_of_my_lf_tag':{'value_of_my_tag':['customer_id','customer_lifetime_value','dt']}}},lf_grants={'data_cell_filters':{'enabled':True,'drop_existing':True,'filters':{'the_name_of_my_filter':{'row_filter':'customer_lifetime_value>15','principals':['arn:aws:iam::123456789:user/lf-data-scientist'],'excluded_column_names':['first_name']}},}})}}selectcustomers.customer_id,customers.first_name,customers.last_name,customer_orders.first_order,customer_orders.most_recent_order,customer_orders.number_of_orders,customer_payments.total_amountascustomer_lifetime_value,current_date()asdtfromcustomersleftjoincustomer_ordersusing(customer_id)leftjoincustomer_paymentsusing(customer_id)At dbt-project levelThis way you can specify tags and data filtering policy for a particular path in your dbt project (eg. models, seeds, models/model_group1, etc.)
This is especially useful for seeds, for which you can't define configuration in the file directly.seeds:
  +lf_tags_config:
    enabled: true
    tags_table: 
      name_of_my_table_tag: 'value_of_my_table_tag'  
    tags_database: 
      name_of_my_database_tag: 'value_of_my_database_tag'
models:
  +lf_tags_config:
    enabled: true
    drop_existing: True
    tags_database: 
      name_of_my_database_tag: 'value_of_my_database_tag'
    tags_table: 
      name_of_my_table_tag: 'value_of_my_table_tag'TestsTo perform a functional test:Install dev requirements:$pip3install-rdev-requirements.txtInstall dev locally$python3setup.pybuild&&python3setup.pyinstall_libExport variables$exportDBT_AWS_ACCOUNT=123456789101$exportDBT_GLUE_REGION=us-east-1
$exportDBT_S3_LOCATION=s3://mybucket/myprefix
$exportDBT_GLUE_ROLE_ARN=arn:aws:iam::1234567890:role/GlueInteractiveSessionRoleCaution: Be careful not to set S3 path containing important files.
dbt-glue's test suite automatically deletes all the existing files under the S3 path specified inDBT_S3_LOCATION.Run the test$python3-mpytesttests/functionalor$python3-mpytest-sFor more information, check the dbt documentation abouttesting a new adapter.CaveatsSupported FunctionalityMost dbt Core functionality is supported, but some features are only available with Apache Hudi.Apache Hudi-only features:Incremental model updates byunique_keyinstead ofpartition_by(seemergestrategy)Some dbt features, available on the core adapters, are not yet supported on Glue:Persistingcolumn-level descriptions as database commentsSnapshotsFor more information on dbt:Read theintroduction to dbt.Read thedbt viewpoint.Join thedbt community.SecuritySeeCONTRIBUTINGfor more information.LicenseThis project is licensed under the Apache-2.0 License.