Package: dbt-athena-adapter
dbt-athenaSupports dbt version1.0.*SupportsSeedsCorrectly detects views and their columnsSupportincremental modelsSupport two incremental update strategies:insert_overwriteandappendDoesnotsupport the use ofunique_keyOnlysupports Athena engine 2Changing Athena Engine VersionsInstallationpip install dbt-athena-adapterOrpip install git+https://github.com/Tomme/dbt-athena.gitPrerequisitesTo start, you will need an S3 bucket, for instancemy-staging-bucketand an Athena database:CREATEDATABASEIFNOTEXISTSanalytics_devCOMMENT'Analytics models generated by dbt (development)'LOCATION's3://my-staging-bucket/'WITHDBPROPERTIES('creator'='Foo Bar','email'='foo@bar.com');Notes:Take note of your AWS region code (e.g.us-west-2oreu-west-2, etc.).You can also useAWS Glueto create and manage Athena databases.CredentialsThis plugin does not accept any credentials directly. Instead,credentials are determined automaticallybased onaws cli/boto3conventions and
stored login info. You can configure the AWS profile name to use viaaws_profile_name. Checkout DBT profile configuration below for details.Configuring your profileA dbt profile can be configured to run against AWS Athena using the following configuration:OptionDescriptionRequired?Examples3_staging_dirS3 location to store Athena query results and metadataRequireds3://bucket/dbt/region_nameAWS region of your Athena instanceRequiredeu-west-1schemaSpecify the schema (Athena database) to build models into (lowercaseonly)RequireddbtdatabaseSpecify the database (Data catalog) to build models into (lowercaseonly)Requiredawsdatacatalogpoll_intervalInterval in seconds to use for polling the status of query results in AthenaOptional5aws_profile_nameProfile to use from your AWS shared credentials file.Optionalmy-profilework_groupIdentifier of Athena workgroupOptionalmy-custom-workgroupnum_retriesNumber of times to retry a failing queryOptional3Example profiles.yml entry:athena:target:devoutputs:dev:type:athenas3_staging_dir:s3://athena-query-results/dbt/region_name:eu-west-1schema:dbtdatabase:awsdatacatalogaws_profile_name:my-profilework_group:my-workgroupAdditional informationthreadsis supporteddatabaseandcatalogcan be used interchangeablyUsage notesModelsTable Configurationexternal_location(default=none)The location where Athena saves your table in Amazon S3Ifnonethen it will default to{s3_staging_dir}/tablesIf you are using a static value, when your table/partition is recreated underlying data will be cleaned up and overwritten by new datapartitioned_by(default=none)An array list of columns by which the table will be partitionedLimited to creation of 100 partitions (currently)bucketed_by(default=none)An array list of columns to bucket databucket_count(default=none)The number of buckets for bucketing your dataformat(default='parquet')The data format for the tableSupportsORC,PARQUET,AVRO,JSON, orTEXTFILEwrite_compression(default=none)The compression type to use for any storage format that allows compression to be specified. To see which options are available, check outCREATE TABLE ASfield_delimiter(default=none)Custom field delimiter, for when format is set toTEXTFILEMore information:CREATE TABLE ASSupported functionalitySupport for incremental models:Support two incremental update strategies with partitioned tables:insert_overwriteandappendDoesnotsupport the use ofunique_keyDue to the nature of AWS Athena, not all core dbt functionality is supported.
The following features of dbt are not implemented on Athena:SnapshotsKnown issuesQuoting is not currently supportedIf you need to quote your sources, escape the quote characters in your source definitions:version:2sources:-name:my_sourcetables:-name:first_tableidentifier:"firsttable"# Not like that-name:second_tableidentifier:"\"secondtable\""# Like thisTables, schemas and database should only be lowercaseOnlysupports Athena engine 2Changing Athena Engine VersionsRunning testsFirst, install the adapter and its dependencies usingmake(seeMakefile):makeinstall_depsNext, configure the environment variables indev.envto match your Athena development environment. Finally, run the tests usingmake:makerun_testsCommunityfishtown-analytics/dbtfishtown-analytics/dbt-prestoDandandan/dbt-athenalaughingman7743/PyAthena