Package: dbt-athena2
dbt-athena2This is a adapter leveraged fromthis repoto better serve our custom needs. It supports addtional capabilities as below:Run on dbt-core version 1.1.xSupport boto3 session to take the credential from from aws profile nameOn schema change support for new columns addedAdd s3 bucket for storing data instead of randomly writing on staging dirQuick startedWithin your python environment, proceed below step to initate a first project. There will be some prompts at during inital steps, referConfiguring your profilesection below to properly set it up.pipinstalldbt-athena2
dbtinitmy_dbt_projectexportDBT_PROFILES_DIR=`pwd`cdmy_dbt_project
dbtdebug# to test connectiondbtseed# to dump your seed datadbtrun# to run all modelsdbtrun--selectmodel_name# to run specific model#...and more...Basic usageModel configurationBelow show an example how we configure how our model be configured.There are 4 supportedmaterializedmodes:view,table,incrementalandesphemeral. Detailshere.incremental_strategysupportsinsert_overwriteandappend. If partition is specified, it only overwrites partions available from source data.on_schema_changesupportfail,ignoreandappend_new_columnsonly and for onlyincrementalmaterialization. Detailshere.There are some usefule macro such asrun_started_atcan be referred fromhereto enhance the flexibility on the model.{{config(materialized="incremental",partitioned_by=["report_month"],incremental_strategy="insert_overwrite",on_schema_change="append_new_columns")}}{%set run_date = run_started_at.astimezone(modules.pytz.timezone("Asia/Saigon")).strftime("%Y-%m-%d") %}select cast(working_day as timestamp) working_day,sum(spend_balance) spend_balance,sum(td_balance) td_balance,sum(gs_balance) gs_balance,cast(date_format(date_parse('{{ run_date }}', '%Y-%m-%d') - interval '1' month, '%Y%m') as int) report_monthfrom {{ source('analytics', 'eod_balance') }}where cast(working_day as date) >= date_trunc('month', cast(date_parse('{{ run_date }}', '%Y-%m-%d')  as date)-interval'2'month)and cast(working_day as date) < date_trunc('month', cast(date_parse('{{ run_date }}', '%Y-%m-%d')  as date)-interval'1'month)group by working_dayorder by working_day descSeedUnder folder seeds, place csv seed file ( eg.c_ecom_rate.csv) and the yaml config (eg.c_ecom_rate.yml) as below example. Then rundbt seedversion:2seeds:-name:c_ecom_rateconfig:enabled:truequote_columns:truetags:accounting | reportFurther notesIf the workgroup is specified in theprofile.ymlwithouts3_staging_dir, it will extract the default s3 ouput attached with thatwork_group when Override client-side settings enabled.The boto3 session inherit from devlopment environment; once deployed, it should be obtained permission as role such as EC2 profile instance or K8S service account role.Athena limit ALTER ADD COLUMNS with data typedate, recommend to parse it totimestamporstringduring implementing the model. Detailshere.Athena not accept the comment like/*, to ignore these auto generated comment fromdbt, place thisquery-comment: nullindbt_project.ymlfile.Configuring your profileA dbt profile can be configured to run against AWS Athena using the following configuration:OptionDescriptionRequired?Examples3_staging_dirS3 location to store Athena query results and metadataRequireds3://athena-output-bucket/data_services/region_nameAWS region of your Athena instanceRequiredap-southeast-1schemaSpecify the schema (Athena database) to build models into (lowercaseonly)RequireddbtdatabaseSpecify the database (Data catalog) to build models into (lowercaseonly)Requiredawsdatacatalogpoll_intervalInterval in seconds to use for polling the status of query results in AthenaOptional5aws_profile_nameProfile to use from your AWS shared credentials file.Optionalmy-profilework_groupIdentifier of Athena workgroupOptionalmy-custom-workgroupnum_retriesNumber of times to retry a failing queryOptional3s3_data_dirPrefix for storing tables, if different from the connection'ss3_staging_dirOptionals3://athena-data-bucket/{schema_name}/{table_name}/Example profiles.yml entry:athena:target:devoutputs:dev:database:awsdatacatalogregion_name:ap-southeast-1aws_profile_name:dl-dev-processs3_staging_dir:s3://athena-output-bucket/data_services/s3_data_dir:s3://athena-data-bucket/{schema_name}/{table_name}/schema:accountingtype:athenaAdditional informationthreadsis supporteddatabaseandcatalogcan be used interchangeablyRunning testsFirst, install the adapter and its dependencies usingmake(seeMakefile):makeinstall_depsNext, configure the environment variables indev.envto match your Athena development environment. Finally, run the tests usingmake:makerun_testsReferencesHow to structure a dbt projectdbt best practices