Package: tap-dbt
tap-dbttap-dbtis a Singer tap for thedbt Cloud API v2.Built with theSinger SDK.InstallationFeaturesConfigurationInputsJSON exampleEnvironment variables exampleMeltano ExampleUsageExecuting the Tap DirectlyWith MeltanoInstallationpipinstalltap-dbtFeaturesStreamsFull stream metadata is available in the dbt Labs repository:openapi_schemaSelected by defaultThe following will be extracted by default if no catalog is used:Stream: accountsStream: jobsStream: projectsStream: runsConfigurableCan be enabled by settingselectedin the catalog:Stream: connectionsStream: environmentsStream: repositoriesStream: usersIncremental Run StreamOrdering the query from the Runs endpoint by-finished_at, i.e. descending Run Finished Datetime, yields:idfinished_atupdated_atcreated_at314516None2023-05-27 21:05:16.109548+00:002023-05-27 21:05:05.664170+00:00314514None2023-05-27 21:00:16.847296+00:002023-05-27 21:00:05.458908+00:00314513None2023-05-27 21:00:16.355680+00:002023-05-27 21:00:05.427258+00:00314517None2023-05-27 21:05:17.094309+00:002023-05-27 21:05:05.696222+00:003145152023-05-27 21:01:28.568431+00:002023-05-27 21:01:29.269048+00:002023-05-27 21:00:05.488543+00:003145122023-05-27 20:48:59.342035+00:002023-05-27 20:48:59.844412+00:002023-05-27 20:45:04.509746+00:003145112023-05-27 20:48:46.571106+00:002023-05-27 20:48:47.079130+00:002023-05-27 20:40:04.257950+00:003145052023-05-27 20:41:35.591976+00:002023-05-27 20:41:36.305364+00:002023-05-27 20:15:02.808079+00:003145102023-05-27 20:39:27.162437+00:002023-05-27 20:39:28.628257+00:002023-05-27 20:35:03.939439+00:003145092023-05-27 20:37:39.965974+00:002023-05-27 20:37:40.496212+00:002023-05-27 20:30:03.802620+00:00The incremental sync has been set up so that it works onreplication_key = "finished_at", when an INCREMENTAL sync is run:If the bookmark is set, the stream is queried in reversefinished_atorder.If thefinished_atvalue is not set, the run is assumed to still be running so the record is included, plus the sort order implies that there should be records with populatedfinished_atappearing later in the stream -Repeated sync operation will yield the same records if the dbt Job Run is still underway, however this adheres to the 'at least once' delivery promise -https://sdk.meltano.com/en/latest/implementation/at_least_once.htmlOnce the sync operation reaches records with populatedfinished_at, the values are compared with the bookmark and once thefinished_atvalue becomes less than the bookmark the stream finishes syncing.ConfigurationVisit theAPI docsfor instructions on how to get your API key.You can pass configuration using environment variables with theTAP_DBT_prefix followed by the uppercased field nametap-dbt--config=ENVor a JSON filetap-dbt--config=config.jsonInputsFieldDescriptionTypeRequiredDefaultapi_keyAPI key for the dbt Cloud APIstringyesaccount_idsdbt Cloud account IDslist(string)yesuser_agentUser-Agent to make requests withstringnotap-dbt/0.1.0 Singer Tap for the dbt Cloud APIbase_urlBase URL for the dbt Cloud APIstringnohttps://cloud.getdbt.com/api/v2page_sizeNumber of records per API call, sets thelimit=url parameterintegerno5000A full list of supported settings and capabilities for this tap is available by running:tap-dbt--about--formatjsonJSON example{"api_key":"da39a3ee5e6b4b0d3255bfef95601890afd80709","account_ids":["51341"],"user_agent":"tap-dbt/0.1.0 Singer Tap for the dbt Cloud API","base_url":"https://my-dbt-cloud-api.com","page_size":5000}Environment variables exampleTAP_DBT_API_KEY=da39a3ee5e6b4b0d3255bfef95601890afd80709
TAP_DBT_ACCOUNT_IDS=51341
TAP_DBT_USER_AGENT='tap-dbt/0.1.0 Singer Tap for the dbt Cloud API'
TAP_DBT_BASE_URL=https://my-dbt-cloud-api.com"
TAP_DBT_PAGE_SIZE=5000Meltano Exampleplugins:extractors:-name:tap-dbtlogo_url:https://hub.meltano.com/assets/logos/taps/dbt.pnglabel:dbt Clouddocs:https://hub.meltano.com/taps/dbtrepo:https://github.com/edgarrmondragon/tap-dbtnamespace:dbtpip_url:tap-dbtexecutable:tap-dbtcapabilities:-catalog-discoversettings:-name:base_urllabel:dbt Cloud URLplaceholder:"https://cloud.getdbt.com/api/v2"-name:api_keykind:passwordlabel:API Keydocs:"https://docs.getdbt.com/dbt-cloud/api#section/Authentication"-name:account_idskind:arraylabel:Account IDs-name:user_agentlabel:User-Agentplaceholder:"tap-dbt/0.1.0SingerTapforthedbtCloudAPI"-name:page_sizekind:integerlabel:Page SizeUsageYou can easily runtap-dbtwith the CLI or usingMeltano.Executing the Tap Directlytap-dbt--version
tap-dbt--help
tap-dbt--config.secrets/example.json--discover>./catalog/jsonWith Meltanomeltanoelttap-dbttarget-snowflake--job_iddbt_snowflake