Package: torch-ort-inference
dbtenables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.dbt-glueThedbt-gluepackage implements thedbt adapterprotocol for AWS Glue's Spark engine.
It supports running dbt against Spark, through the new Glue Interactive Sessions API.InstallationThe package can be installed from PyPI with:$pipinstalldbt-glueFor further (and more likely up-to-date) info, see theREADMEConnection MethodsConfiguring your AWS profile for Glue Interactive SessionThere are two IAM principals used with interactive sessions.Client principal: The princpal (either user or role) calling the AWS APIs (Glue, Lake Formation, Interactive Sessions)
from the local client. This is the principal configured in the AWS CLI and likely the same.Service role: The IAM role that AWS Glue uses to execute your session. This is the same as AWS Glue
ETL.Readthis documentationto configure these principals.To enjoy all features ofdbt-glueadapter, you will need to attach to the Service role the 3 AWS managed policies below:Servicemanaged policy requiredAmazon S3AmazonS3FullAccessAWS GlueAWSGlueConsoleFullAccessAWS Lake formationAWSLakeFormationDataAdminConfiguration of the local environmentBecausedbtanddbt-glueadapter are compatible with Python versions 3.7, 3.8, and 3.9, check the version of Python:$python3--versionConfigure a Python virtual environment to isolate package version and code dependencies:$sudoyuminstallgit
$python3-mpipinstall--upgradepip
$python3-mvenvdbt_venv
$sourcedbt_venv/bin/activate
$python3-mpipinstall--upgradepipConfigure the last version of AWS CLI$curl"https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip"-o"awscliv2.zip"$unzipawscliv2.zip
$sudo./aws/installConfigure the aws-glue-session package$sudoyuminstallgcckrb5-devel.x86_64python3-devel.x86_64-y
$pip3install—upgradeboto3
$pip3install—upgradeaws-glue-sessionsExample configtype: glue
query-comment: This is a glue dbt example
role_arn: arn:aws:iam::1234567890:role/GlueInteractiveSessionRole
region: us-east-1
workers: 2
worker_type: G.1X
idle_timeout: 10
schema: "dbt_demo"
database: "dbt_demo"
session_provisioning_timeout_in_seconds: 120
location: "s3://dbt_demo_bucket/dbt_demo_data"The table below describes all the options.OptionDescriptionMandatoryproject_nameThe dbt project name. This must be the same as the one configured in the dbt project.yestypeThe driver to use.yesquery-commentA string to inject as a comment in each query that dbt runs.norole_arnThe ARN of the interactive session role created as part of the CloudFormation template.yesregionThe AWS Region were you run the data pipeline.yesworkersThe number of workers of a defined workerType that are allocated when a job runs.yesworker_typeThe type of predefined worker that is allocated when a job runs. Accepts a value of Standard, G.1X, or G.2X.yesschemaThe schema used to organize data stored in Amazon S3.yesdatabaseThe database in Lake Formation. The database stores metadata tables in the Data Catalog.yessession_provisioning_timeout_in_secondsThe timeout in seconds for AWS Glue interactive session provisioning.yeslocationThe Amazon S3 location of your target data.yesidle_timeoutThe AWS Glue session idle timeout in minutes. (The session stops after being idle for the specified amount of time.)noglue_versionThe version of AWS Glue for this session to use. Currently, the only valid options are 2.0 and 3.0. The default value is 2.0.nosecurity_configurationThe security configuration to use with this session.noconnectionsA comma-separated list of connections to use in the session.noConfigsConfiguring tablesWhen materializing a model astable, you may include several optional configs that are specific to the dbt-spark plugin, in addition to the standardmodel configs.OptionDescriptionRequired?Examplefile_formatThe file format to use when creating tables (parquet,csv,json,text,jdbcororc).Optionalparquetpartition_byPartition the created table by the specified columns. A directory is created for each partition.Optionaldate_dayclustered_byEach partition in the created table will be split into a fixed number of buckets by the specified columns.Optionalcountry_codebucketsThe number of buckets to create while clusteringRequired ifclustered_byis specified8Incremental modelsdbt seeks to offer useful, intuitive modeling abstractions by means of its built-in configurations and materializations.For that reason, the dbt-glue plugin leans heavily on theincremental_strategyconfig. This config tells the incremental materialization how to build models in runs beyond their first. It can be set to one of three values:append(default): Insert new records without updating or overwriting any existing data.insert_overwrite: Ifpartition_byis specified, overwrite partitions in the table with new data. If nopartition_byis specified, overwrite the entire table with new data.merge(Apache Hudi only): Match records based on aunique_key; update old records, insert new ones. (If nounique_keyis specified, all new data is inserted, similar toappend.)Each of these strategies has its pros and cons, which we'll discuss below. As with any model config,incremental_strategymay be specified indbt_project.ymlor within a model file'sconfig()block.Notes:The default strategie isinsert_overwriteTheappendstrategyFollowing theappendstrategy, dbt will perform aninsert intostatement with all new data. The appeal of this strategy is that it is straightforward and functional across all platforms, file types, connection methods, and Apache Spark versions. However, this strategycannotupdate, overwrite, or delete existing data, so it is likely to insert duplicate records for many data sources.Source code{{config(materialized='incremental',incremental_strategy='append',)}}--  All rows returned by this query will be appended to the existing tableselect*from{{ref('events')}}{%ifis_incremental()%}whereevent_ts>(selectmax(event_ts)from{{this}}){%endif%}Run Codecreatetemporaryviewspark_incremental__dbt_tmpasselect*fromanalytics.eventswhereevent_ts>=(selectmax(event_ts)from{{this}});insertintotableanalytics.spark_incrementalselect`date_day`,`users`fromspark_incremental__dbt_tmpTheinsert_overwritestrategyThis strategy is most effective when specified alongside apartition_byclause in your model config. dbt will run anatomicinsert overwritestatementthat dynamically replaces all partitions included in your query. Be sure to re-selectallof the relevant data for a partition when using this incremental strategy.If nopartition_byis specified, then theinsert_overwritestrategy will atomically replace all contents of the table, overriding all existing data with only the new records. The column schema of the table remains the same, however. This can be desirable in some limited circumstances, since it minimizes downtime while the table contents are overwritten. The operation is comparable to runningtruncate+inserton other databases. For atomic replacement of Delta-formatted tables, use thetablematerialization (which runscreate or replace) instead.Source Code{{config(materialized='incremental',partition_by=['date_day'],file_format='parquet')}}/*Every partition returned by this query will be overwrittenwhen this model runs*/withnew_eventsas(select*from{{ref('events')}}{%ifis_incremental()%}wheredate_day>=date_add(current_date,-1){%endif%})selectdate_day,count(*)asusersfromeventsgroupby1Run Codecreatetemporaryviewspark_incremental__dbt_tmpaswithnew_eventsas(select*fromanalytics.eventswheredate_day>=date_add(current_date,-1))selectdate_day,count(*)asusersfromeventsgroupby1;insertoverwritetableanalytics.spark_incrementalpartition(date_day)select`date_day`,`users`fromspark_incremental__dbt_tmpSpecifyinginsert_overwriteas the incremental strategy is optional, since it's the default strategy used when none is specified.ThemergestrategyUsage notes:Themergeincremental strategy requires:file_format: hudiAWS Glue runtime 2 with hudi libraries as extra jarsYou can add hudi libraries as extra jars in the classpath using extra_jars options in your profiles.yml.
Here is an example:extra_jars: "s3://dbt-glue-hudi/Dependencies/hudi-spark.jar,s3://dbt-glue-hudi/Dependencies/spark-avro_2.11-2.4.4.jar"dbt will run anatomicmergestatementwhich looks nearly identical to the default merge behavior on Snowflake and BigQuery. If aunique_keyis specified (recommended), dbt will update old records with values from new records that match on the key column. If aunique_keyis not specified, dbt will forgo match criteria and simply insert all new records (similar toappendstrategy).Source Code{{config(materialized='incremental',incremental_strategy='merge',unique_key='user_id',file_format='hudi')}}withnew_eventsas(select*from{{ref('events')}}{%ifis_incremental()%}wheredate_day>=date_add(current_date,-1){%endif%})selectuser_id,max(date_day)aslast_seenfromeventsgroupby1Persisting model descriptionsRelation-level docs persistence is supported since dbt v0.17.0. For more
information on configuring docs persistence, seethe docs.When thepersist_docsoption is configured appropriately, you'll be able to
see model descriptions in theCommentfield ofdescribe [table] extendedorshow table extended in [database] like '*'.Alwaysschema, neverdatabaseApache Spark uses the terms "schema" and "database" interchangeably. dbt understandsdatabaseto exist at a higher level thanschema. As such, you shouldneveruse or setdatabaseas a node config or in the target profile when running dbt-glue.If you want to control the schema/database in which dbt will materialize models,
use theschemaconfig andgenerate_schema_namemacroonly.CaveatsSupported FunctionalityMost dbt Core functionality is supported, but some features are only available with Apache Hudi.Apache Hudi-only features:Incremental model updates byunique_keyinstead ofpartition_by(seemergestrategy)Some dbt features, available on the core adapters, are not yet supported on Glue:Persistingcolumn-level descriptions as database commentsSnapshotsFor more information on dbt:Read theintroduction to dbt.Read thedbt viewpoint.Join thedbt community.SecuritySeeCONTRIBUTINGfor more information.LicenseThis project is licensed under the Apache-2.0 License.