Saravanan666/pypi-org huggingface dataset:
Package: dbt-athena
dbt-athenaNote: this repository is not actively maintained. You can find a newer version of the adapter with Athena Engine 2 support, seeds and more in the following repository:https://github.com/Tomme/dbt-athena/InstallationThis plugin can be installed via pip:$ pip install git+https://github.com/Dandandan/dbt-athena.gitConfiguring your profileA dbt profile can be configured to run against Athena using the following configuration:OptionDescriptionRequired?Examples3_staging_dirThe location where Athena stores meta infoRequireds3://bucket/stagingschemaSpecify the schema (athena database) to build models intoRequireddevdatabaseData catalogRequiredawsdatacatalogregion_nameSpecify in which AWS region it should connectRequiredeu-west-1threadsHow many threads dbt should useOptional(default=1)8max_retry_numberNumber for retries for exponential backoffOptional(default=5)8max_retry_delayMaximum delay for exponential backoff in secondsOptional(default=100)8Example profiles.yml entry:athena:
  target: athena
  outputs:
    athena:
      type: athena
      database: awsdatacatalog
      schema: dev
      region_name: eu-west-1
      threads: 8
      s3_staging_dir: s3://athena-staging-bucket/Usage NotesSupported FunctionalityDue to the nature of Athena, not all core dbt functionality is supported.
The following features of dbt are not implemented on Athena:ArchivalIncremental modelsIf you are interested in helping to add support for this functionality in dbt on Athena, pleaseopen an issue!Known issues:Quoting is not supportedReporting bugs and contributing codeWant to report a bug or request a feature? Let us know onSlack, or openan issue.Running the testsUnit testsInstall tox:pipinstalltoxRun unit tests:tox-eunitIntegration testsAt this time, an AWS account is not provided in order to run the tests in CI. We kindly ask contributors/reviewers to use their own AWS accounts in order to test contributions.You can also reach out in theSlack#athenachannel for someone to run the tests for you.Steps:Clone thedbt-integration-testsrepositorygitclone--branchathena-supporthttps://github.com/EarnestResearch/dbt-integration-tests.gitRun the tests:Additionally, you might need to set theAWS_PROFILEenvironment variable if you use another value than "default" (for example if you connect to multiple AWS accounts or assume different IAM roles)AWS_DEFAULT_REGION=us-west-2ATHENA_S3_STAGING_DIR=s3://dbt-athena-integration-tests/tests/DBT_PROFILES_DIR=$(pwd)/test/integration/tox-eintegration-athenaCode of ConductEveryone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow thePyPA Code of Conduct.
Package: dbt-bigquery
dbtenables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.dbt-bigqueryThedbt-bigquerypackage contains all of the code enabling dbt to work with Google BigQuery. For
more information on using dbt with BigQuery, consultthe docs.Getting startedInstall dbtRead theintroductionandviewpointJoin the dbt CommunityBe part of the conversation in thedbt Community SlackRead more on thedbt Community DiscourseReporting bugs and contributing codeWant to report a bug or request a feature? Let us know onSlack, or openan issueWant to help us build dbt? Check out theContributing GuideCode of ConductEveryone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow thedbt Code of Conduct.
Package: dbt-bigquery-velocity
dbtenables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.dbt-bigqueryThedbt-bigquerypackage contains all of the code enabling dbt to work with Google BigQuery. For
more information on using dbt with BigQuery, consultthe docs.Getting startedInstall dbtRead theintroductionandviewpointJoin the dbt CommunityBe part of the conversation in thedbt Community SlackRead more on thedbt Community DiscourseReporting bugs and contributing codeWant to report a bug or request a feature? Let us know onSlack, or openan issueWant to help us build dbt? Check out theContributing GuideCode of ConductEveryone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow thedbt Code of Conduct.
Package: dbt-buenavista
dbt-buenavistaBuena Vistais a Postgres proxy server written in Python. It is intended to
serve as the Controller in theModel-View-Controller architecturefor data applications, providing a locus of control for data teams in between the data warehouse (model layer) and downstream
data consumers like BI tools and notebooks (view layer.)dbtis the best way to manage a collection of data transformations written in SQL or Python for analytics
and data science.The purpose ofdbt-buenavistais to explore some new techniques for creating dbt models that a proxy service like
Buena Vista makes possible, starting with a way of executing dbt's Python models inspired by the approach taken
bydbt-duckdb: instead of pushing the Python compute into the Python execution engines
supported by Snowflake, Databricks, and BigQuery, the dbt-buenavista adapter executes the Python models on the Buena Vista
server itself, reading in any data tables that are used in the computation as data frames (either Pandas or another data frame
library supported by the data warehouse) and writing the data frame back to the data warehouse when the computation is finished.
This approach makes it possible to support Python-based models against data warehouses that do not ship with a Python execution
engine (most notably Postgres and Redshift) and may well simplify the development and debugging workflows for teams that want to
include Python models in their data pipelines (e.g.,printandloggingstatements will run on the Buena Vista server and can
be sent wherever they need to go to aid in debugging.)InstallationConfiguring Your ProfileThe profile settings for a dbt-buenavista connection are nearly identical to the settings for adbt-postgres profile, since everydbt-buenavistaadapter is a subclass of thedbt-postgresadapter. Instead of setting the profile'stypefield aspostgres, you set it to one of the Buena Vista supported types, which are currentlybv_duckdb(for a Buena Vista server
powered by DuckDB) orbv_postgres(for a Buena Vista server that is proxying a Postgres database.) Thehostandportsettings are required and should be configured to point at the Buena Vista server that you are using.default:
  outputs:
   dev:
     type: <bv_duckdb|bv_postgres>
     host: <host>
     port: <port>
     dbname: <dbname>
     ...
  target: dev
Package: dbtc
An unaffiliated python interface for dbt Cloud APIsDocumentation:https://dbtc.dpguthrie.comInteractive Demo:https://dpguthrie-dbtc-streamlit-home-yy7c0b.streamlit.app/Source Code:https://github.com/dpguthrie/dbtcV2 Docs:https://docs.getdbt.com/dbt-cloud/api-v2V3 Docs:https://docs.getdbt.com/dbt-cloud/api-v3Overviewdbtc is an unaffiliated python interface to various dbt Cloud API endpoints.This library acts as a convenient interface to two different APIs that dbt Cloud offers:Cloud API:  This is a REST API that exposes endpoints that allow users to programatically create, read, update, and delete
resources within their dbt Cloud Account.Metadata API:  This is a GraphQL API that exposes metadata generated from a job run within dbt Cloud.RequirementsPython 3.7+Requests- The elegant and simple HTTP library for Python, built for human beings.sgqlc- Simple GraphQL ClientTyper- Library for building CLI applicationsInstallationpipinstalldbtcBasic UsagePythonThe interface to both APIs are located in thedbtCloudClientclass.The example below shows how you use thecloudproperty on an instance of thedbtCloudClientclass to to access a method,trigger_job_from_failure, that allows you to restart a job from its last point of failure.fromdbtcimportdbtCloudClient# Assumes that DBT_CLOUD_SERVICE_TOKEN env var is setclient=dbtCloudClient()account_id=1job_id=1payload={'cause':'Restarting from failure'}run=client.cloud.trigger_job_from_failure(account_id,job_id,payload,should_poll=False,)# This returns a dictionary containing two keysrun['data']run['status']Similarly, use themetadataproperty to retrieve information from theDiscovery API.
Here's how you could retrieve all of the metrics for your project.fromdbtcimportdbtCloudClientclient=dbtCloudClient()query='''query ($environmentId: BigInt!, $first: Int!) {environment(id: $environmentId) {definition {metrics(first: $first) {edges {node {namedescriptiontypeformulafiltertagsparents {nameresourceType}}}}}}}'''variables={'environmentId':1,'first':500}data=client.metadata.query(query,variables)# Data will be in the edges key, which will be a list of nodesnodes=data['data']['definition']['metrics']['edges']fornodeinnodes:# node is a dictionarynode_name=node['name']...If you're unfamiliar either with the Schema to query or even how to write a GraphQL query, I highly recommend going to thedbt Cloud Discovery API playground.  You'll be able to interactively explore the Schema while watching it write a GraphQL query for you!CLIThe CLI example below will map to the python cloud example above:dbtctrigger-job-from-failure\--account-id1\--job-id1\--payload'{"cause": "Restarting from failure"}'\--no-should-pollSimilarly, for the metadata example above (assuming that you've put both thequeryandvariablesargument into variables):dbtcquery--query$query--variables$variablesIf not setting your service token as an environment variable, do the following:dbtc--tokenthis_is_my_tokenquery--query$query--variables$variablesLicenseThis project is licensed under the terms of the MIT license.
Package: dbt-cloud-api-client
dbt cloud api clientdbt cloud api clientInstall toxpip install toxUsage:importtimefromdbt_cloud_api_client.clientimportDbtCloudClient...client=DbtCloudClient(token='<token>')account=client.get_account_by_name('<ACCOUNT_NAME>')job=account.get_job_by_name('<JOB NAME IN DBT CLOUD>')run=job.run('Job triggered from python.')whilenotrun.is_finished():print('{}-{}'.format(run.id,run.status))run.reload()# print(run.data) # display all details of run.# sleep for a period of timetime.sleep(60)print('{}-{}'.format(run.id,run.status))
Package: dbt-cloud-jobs
dbt-cloud-jobsVersion control your dbt Cloud jobs with YML.Installationpipinstalldbt-cloud-jobsQuickstartCreate an API token in dbt Cloud:Generate a service account token, make sure to grant theJobs Adminpermission set.If you do not have access to create a service account token you can create aUser API token. Note that the service account method is preferred.Set an environment variable with the value of the token:exportDBT_API_TOKEN="<VALUE_FROM_PREVIOUS_STEP>"Set an environment variable for the region where your dbt Cloud account is hosted. The value must be one of "AU", "Europe" or "US" (see docshere):exportDBT_CLOUD_REGION="<REGION>"Import your existing dbt Cloud jobs:dbt_cloud_jobs--import--account-id123456--filedbt_cloud_jobs.ymlEdit the definition of your jobs indbt_cloud_jobs.yml.Sync the updated definitions to dbt Cloud:dbt_cloud_jobs--sync--filedbt_cloud_jobs.ymlRecommended usage in CI/CDCIIn CIdbt-cloud-jobsshould be used to verify that the provided YML file is valid. For example:-name:Install dbt_cloud_jobsrun:pip install dbt-cloud-jobs-name:Validate `dbt_cloud_jobs.yml`run:dbt_cloud_jobs --validate --file dbt_cloud_jobs.ymlCDIn CDdbt-cloud-jobsshould be used to sync the provided YML file to dbt Cloud. For example:-name:Install dbt_cloud_jobsrun:pip install dbt-cloud-jobs-name:Sync `dbt_cloud_jobs.yml`run:dbt_cloud_jobs --sync --file dbt_cloud_jobs.ymlLimitationsService account tokens are created at the account level. This means that if you have multiple dbt Cloud accounts you will need to create differentdbt_cloud_jobs.ymlfiles for each account. If you try to usedbt-cloud-jobswith a file that contains multipleaccount_idvalues, an error will be raised.DevelopmentTo setup your development environment, fork this repository and run:poetryshell
poetryinstallSet the following environment variables:exportDBT_ACCOUNT_ID=<DBT_ACCOUNT_ID>exportDBT_CLOUD_REGION="<DBT_CLOUD_REGION>"exportDBT_ENVIRONMENT_ID=<DBT_ENVIRONMENT_ID>exportDBT_PROJECT_ID=<DBT_PROJECT_ID>exportDBT_API_TOKEN="<DBT_API_TOKEN>"It is highly recommended that a dedicated dbt Cloud environment be used for development.All tests can be run via:maketestReleaseTrigger thePublish to PyPiworkflow, inputting the version to publish to PyPi. This workflow will:Publish the version toPyPi.Tag the HEAD commit of themainbranches (tags visiblehere).Create a release (releases visiblehere).
Package: dbt-common
Understanding dbt-commonThe shared common utilities for dbt-core and adapter implementations useGetting startedInstall dbtRead theintroductionandviewpointJoin the dbt CommunityBe part of the conversation in thedbt Community SlackRead more on thedbt Community DiscourseReporting bugs and contributing codeWant to report a bug or request a feature? Let us know and openan issueWant to help us build dbt? Check out theContributing GuideCode of ConductEveryone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow thedbt Code of Conduct.
Package: dbt-copilot-python
DBT Copilot PythonA set of utility functions for running Django & Flask apps in AWS ECS via AWS Copilot.Usingdbt-copilot-pythonSupported Python versions3.9, 3.10, 3.11 and 3.12.Installationpipinstalldbt-copilot-pythonUsageALLOWED_HOSTSsettingAdd the ECS container IP toALLOWED_HOSTSsetting so that the Application Load Balancer (ALB) healthcheck will succeed:fromdbt_copilot_python.networkimportsetup_allowed_hostsALLOWED_HOSTS=[...]ALLOWED_HOSTS=setup_allowed_hosts(ALLOWED_HOSTS)Celery health checkAdd the health check in your application's Celery config file...fromdbt_copilot_python.celery_health_checkimporthealthcheckcelery_app=Celery("application_name")...celery_app=healthcheck.setup(celery_app)Add the health check to the Celery worker service indocker-compose.yml...healthcheck:test:["CMD-SHELL","python-mdbt_copilot_python.celery_health_check.healthcheck"]interval:10stimeout:5sretries:2start_period:5sIn your*-deploycodebase, add the health check to the Celery worker service incopilot/celery-worker/manifest.yml...healthcheck:command:["CMD-SHELL","launcherbash-c'python-mdbt_copilot_python.celery_health_check.healthcheck'"]interval:10stimeout:5sretries:2start_period:10sDATABASESsettingTo configure theDATABASESsetting from an RDS JSON object stored in AWS Secrets Manager, there are two options.Configure theDATABASESsetting to use a database URL (recommended):Note: This is dependent on thedj-database-urlpackage which can be installed viapip install dj-database-url.importdj_database_urlfromdbt_copilot_python.databaseimportdatabase_url_from_envDATABASES={"default":dj_database_url.config(default=database_url_from_env("DATABASE_ENV_VAR_KEY"))}Configure theDATABASESsetting to use a dictionary containing the settings:fromdbt-copilot-python.databaseimportdatabase_from_envDATABASES=database_from_env("DATABASE_ENV_VAR_KEY")Contributing todbt-copilot-pythonRequirementsPoetry;pip install poetryInstall dependencies & pre-commit hookspoetryinstall&&poetryrunpre-commitinstallRun the testspoetryrunpytestPublishingTo publish the Python packagedbt-copilot-python, you will need an API token.Acquire API token fromPassman.Request access from the SRE team.Note: You will need access to theplatformgroup in Passman.Runpoetry config pypi-token.pypi <token>to add the token to your Poetry configuration.Update the version, as the same version cannot be published to PyPi.poetryversionpatchMore options for theversioncommand can be found in thePoetry documentation. For example, for a minor version bump:poetry version minor.Build the Python package.poetrybuildPublish the Python package.Note: Make sure your Pull Request (PR) is approved and contains the version upgrade inpyproject.tomlbefore publishing the package.poetrypublishCheck thePyPi Release historyto make sure the package has been updated.For an optional manual check, install the package locally and test everything works as expected.
Package: dbt-copilot-tools
DBT Copilot ToolsThis package contains a set of tools in the form of a Command Line Interface (CLI) primarily for transferring applications/services fromGOV.UK PaaSto Department for Business and Trade (DBT) PaaS which augmentsAWS Copilot. These tools can also be used to provision AWS resources and/or make sure the CloudFormation templates conform to best practices.Getting startedTo use the Python packagedbt-copilot-tools, follow the steps below.Installationpipinstalldbt-copilot-toolsUsageCheckdbt-copilot-toolshas installed successfully by executingcopilot-helperin the terminal emulator. You should see an output similar to the following:$copilot-helper
Usage:copilot-helper[OPTIONS]COMMAND[ARGS]...

Options:--versionShowtheversionandexit.--helpShowthismessageandexit.

Commands:bootstrapcheck-cloudformationRunsthecheckspassedinthecommandarguments.codebuildcopilotdomainEach command can be executed without any arguments or additional commands to present thehelpmessage.Below is the output for thebootstrapcommand as of version0.1.2.$copilot-helperbootstrap--help
Usage:copilot-helperbootstrap[OPTIONS]COMMAND[ARGS]...

Options:--helpShowthismessageandexit.

Commands:instructionsShowmigrationinstructions.make-configGeneratecopilotboilerplatecode.migrate-secretsMigratesecretsfromyourgovpaasapplicationto...See theCommands Referencefor a list of all available subcommands.
Package: dbt-core
dbtenables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.Understanding dbtAnalysts using dbt can transform their data by simply writing select statements, while dbt handles turning these statements into tables and views in a data warehouse.These select statements, or "models", form a dbt project. Models frequently build on top of one another – dbt makes it easy tomanage relationshipsbetween models, andvisualize these relationships, as well as assure the quality of your transformations throughtesting.Getting startedInstall dbtRead theintroductionandviewpointJoin the dbt CommunityBe part of the conversation in thedbt Community SlackRead more on thedbt Community DiscourseReporting bugs and contributing codeWant to report a bug or request a feature? Let us know onSlack, or openan issueWant to help us build dbt? Check out theContributing GuideCode of ConductEveryone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow thedbt Code of Conduct.
Package: dbt-core-interface
Dbt Core InterfaceFeaturesAn extremely simplified interface is provided to accomplish all of the following with no dependencies outside dbt-core:Parse dbt project on disk loading dbt core classes into memory from a single class/interfaceAutomatic management of the adapter and thread-safe efficient connection pool reuseRun SQL and get results in python fully independent of the dbt adapter which automatically enables support for many databasesRun SQL with dbt SQL from a single method callLoad macros at runtime enabling custom functionality in third party extensions without requiring the dbt packaging system to be managed in userlandCompile dbt jinja extremely fast and efficiently, thread-safe and stress tested at load via a Bottle server which live compiles SQLManage multiple dbt projects in a single process using the DbtProjectContainer classdbt-core-interfaceis a wrapper that allows developers to rapidly develop features and integrations for dbt. This project aims to serve as a place for the community to aggregate the best ways to interface with dbt. It is afforded a much faster iteration cycle and much more freedom due to it's independence from the dbt codebase. It is intended to act as an common library to dbt's existing APIs for developers. Implementations can land here and prove themselves out before landing in the dbt-core codebase and benefit all developers involved. Sqlfluff dbt templater, dbt-osmosis, dbt-fastapi which I am ripping out of dbt-osmosis, an impending metadata manager, a testing framework will all leverage this library. As dbt core evolves and stabilizes its python API, this project will evolve with it. This may manifest in simplification of certain methods but our goal is to maintain the API and focus on driving efficient, innovative/creative, and agile community driven integration patterns.RequirementsTheonlyrequirement is dbt-core, tested with versions1.0.*,1.1.*,1.2.*,1.3.*InstallationYou can installDbt Core InterfaceviapipfromPyPI:$pipinstalldbt-core-interfaceUsagePlease see theApi Referencefor details.To launch the Bottle server for live compiling dbt jinja:python -m dbt_core_interface.projectThis will launch the server on port 8581. You can then make requests to the server, e.g.:curl -X POST -H "Content-Type: application/json" -H "X-dbt-Project: dbt_project" -d '{"project_dir":"/app/tests/sqlfluff_templater/fixtures/dbt/dbt_project/","profiles_dir":"/app/tests/sqlfluff_templater/fixtures/dbt/profiles_yml/","target":"dev"}' http://localhost:8581/registerYou can change the server hostname and port using the--hostand--portarguments.ContributingContributions are very welcome.
To learn more, see theContributor Guide.LicenseDistributed under the terms of theMIT license,Dbt Core Interfaceis free and open source software.IssuesIf you encounter any problems,
pleasefile an issuealong with a detailed description.CreditsThis project was generated from@cjolowicz'sHypermodern Python Cookiecuttertemplate.
Package: dbt-dameng
dbt-damengdbt-damengimplementsdbt (building-a-new-adapter)functionalities for the Dameng database.dbt-dameng was created and maintained by onGitHub repoInstallationpipinstalldbt-damengDocumentationPlease refer to theDameng setup on dbt docs websitefor documentation.Sample projectTo get started, a sample dbt project can be found in the directory/dbt_adbs_test_project.ContributingThis project welcomes contributions from the community. Before submitting a pull request, pleasereview our contribution guide.SecurityPlease consult thesecurity guidefor our responsible security vulnerability disclosure process.Licensedbt-dameng is licensed under Apache 2.0 License which you can findhere.
Package: dbt-databend
dbt-databendThedbt-databendpackage contains all of the code enablingdbtto work withDatabend.Getting startedInstall dbtRead theintroductionandviewpointInstallationUse your favorite Python package manager to install the app from PyPI, e.g.python-mpipinstalldbt-databendSupported featuresDatabend does not support aALTERquery for change a tables schema. So some features are not available.Table materializationView materializationIncremental materializationSeedsSourcesDocs generateTestsSnapshotsEphemeral materializationDatabase & schemaThe dbt modeldatabase.schema.tableis not compatible with Databend because Databend does not support aschema.
So we use a simple modelschema.table, whereschemais the Databend's database.ConfigurationOptionDescriptionRequired?cluster_byFor detailed information about the CLUSTER BY clause, seeSET CLUSTER KEY.Optional (default:empty)Profile ConfigurationDatabend targets should be set up using the following configuration in yourprofiles.ymlfile.
Thedbt-databendpackage usage mysql-procotol for connect to database.your_profile_name:target:devoutputs:dev:type:databendhost:[hostname]port:[port]# default 3307user:[username]password:[password]schema:[database name]