Saravanan666/pypi-org huggingface dataset:
Package: dbt-serverless
Serverless Endpoint for dbt runsThe goal of this project is to avoid the need of an Airflow server in order to schedule dbt tasks like runs, snapshots, docs...It currently encapsulate few dbt commands into a FastAPI server which can be deployed on Cloud Run in a serverless fashion. That way we reduce costs as Cloud Run is terribly cheap!You can also test it locally or through Docker without it being serverless, but it doesn't make sense as you already have the dbt CLI for this.UsageYou'll need to make use of Google ADC (Authentification Default Credentials). Meaning either :gcloud cli already identifiedor a deployment through a google product with a service account having the roles/bigquery.adminor a GOOGLE_APPLICATION_CREDENTIALS env variable for a specific local keyfileLocal deploymentWith pippipinstalldbt-serverless
pythonrunuvicorndbt_serverless.main:app--host0.0.0.0--port8080--reloadWith poetrypoetryadddbt-serverless
poetryrunuvicorndbt_serverless.main:app--host0.0.0.0--port8080--reloadDocker deploymentSimple docker image to build dbt-serverless for local or cloud run testing (for example).ARGbuild_for=linux/amd64FROM--platform=$build_forpython:3.10-slim-bullseyeARGDBT_PROJECTARGPROFILES_DIRWORKDIR/usr/appRUNpipinstall--no-cache-dir--upgradepip&&\pipinstall--no-cache-dirdbt-serverlessCOPY${DBT_PROJECT}/${PROFILES_DIR}/profiles.yml${DBT_PROJECT}/ENTRYPOINT["uvicorn","dbt_serverless.main:app","--host","0.0.0.0","--port","8080"]If you're not on a Google product (like Cloud Run), you will need to specify google creds at docker runtime.For example you can add these cli parameters at runtime, if you're testing and deploying it locally :-v"$(HOME)/.config/gcloud:/gcp/config:ro"\-v/gcp/config/logs\--envCLOUDSDK_CONFIG=/gcp/config\--envGOOGLE_APPLICATION_CREDENTIALS=/gcp/config/application_default_credentials.json\--envGOOGLE_CLOUD_PROJECT=$(PROJECT_ID)\
Package: dbt-snapshot-analysis
dbt Snapshot AnalysisWebsite·Blogdbt Snapshot AnalysisAnalyse the stability of your key metricsChanges of a metric’s historical value harms data consumers’ trust.dbt Snapshot Analysis is Python package to get insight on the quality and stability of the metrics your report.🚀 QuickstartInstallationYou can install DBT Snapshot Analysis usingpip:pipinstalldbt-snapshot-analysisUsagedbt_snapshot_analysisThis project is developped by the DataDrift team with 💚. Check out ourmain repository.
Package: dbt-snowflake
dbtenables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.dbt-snowflakeThedbt-snowflakepackage contains all of the code enabling dbt to work with Snowflake. For
more information on using dbt with Snowflake, consultthe docs.Getting startedInstall dbtRead theintroductionandviewpointJoin the dbt CommunityBe part of the conversation in thedbt Community SlackRead more on thedbt Community DiscourseReporting bugs and contributing codeWant to report a bug or request a feature? Let us know onSlack, or openan issueWant to help us build dbt? Check out theContributing GuideCode of ConductEveryone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow thedbt Code of Conduct.
Package: dbt-spark
dbtenables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.dbt-sparkThedbt-sparkpackage contains all of the code enabling dbt to work with Apache Spark and Databricks. For
more information, consultthe docs.Getting startedInstall dbtRead theintroductionandviewpointRunning locallyAdocker-composeenvironment starts a Spark Thrift server and a Postgres database as a Hive Metastore backend.
Note: dbt-spark now supports Spark 3.3.2.The following command starts two docker containers:docker-composeup-dIt will take a bit of time for the instance to start, you can check the logs of the two containers.
If the instance doesn't start correctly, try the complete reset command listed below and then try start again.Create a profile like this one:spark_testing:target:localoutputs:local:type:sparkmethod:thrifthost:127.0.0.1port:10000user:dbtschema:analyticsconnect_retries:5connect_timeout:60retry_all:trueConnecting to the local spark instance:The Spark UI should be available athttp://localhost:4040/sqlserver/The endpoint for SQL-based testing is athttp://localhost:10000and can be referenced with the Hive or Spark JDBC drivers using connection stringjdbc:hive2://localhost:10000and default credentialsdbt:dbtNote that the Hive metastore data is persisted under./.hive-metastore/, and the Spark-produced data under./.spark-warehouse/. To completely reset you environment run the following:docker-composedown
rm-rf./.hive-metastore/
rm-rf./.spark-warehouse/Reporting bugs and contributing codeWant to report a bug or request a feature? Let us know onSlack, or openan issue.Code of ConductEveryone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow thePyPA Code of Conduct.Join the dbt CommunityBe part of the conversation in thedbt Community SlackRead more on thedbt Community DiscourseReporting bugs and contributing codeWant to report a bug or request a feature? Let us know onSlack, or openan issueWant to help us build dbt? Check out theContributing GuideCode of ConductEveryone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow thedbt Code of Conduct.
Package: dbt-spark-cde
dbt-spark-cdeThedbt-spark-cdeadapter allows you to usedbtCloudera Data Platformwith CDE API server support. This code bases use the dbt-spark project (https://github.com/dbt-labs/dbt-spark), and provides CDE API based connectivity support over it.Getting startedInstall dbtRead theintroductionandviewpointRequirementsPython >= 3.9dbt-core ~= 1.3.0requests >= 2.28.1requests-toolbelt >= 0.9.1pysparkrequests_kerberosrequests-toolbeltpython-decoupleInstalling dbt-spark-cdepip install dbt-spark-cdeProfile Setupdemo_project:
  target: dev
  outputs:
    dev:
     type: spark_cde
     method: cde
     schema: my_db
     auth_endpoint: https://service.spark-cde-gateway.my.org.com/
     host: https://spark-cde-gateway.my.org.com/dex/api/v1/
     user: my_user
     password: my_passTo obtain auth_endpoint follow the steps here: (https://docs.cloudera.com/data-engineering/cloud/api-access/topics/cde-api-get-access-token.html)CaveatsWhile using cde , in the Livy UI if you notice sessions change state to dead from starting instead of idle, make sure there is a proper mapping for the user in the IDBroker mapping sectionActions > Manage Access > IDBroker Mappings .ReferenceAlso make sure the workload password is set either through UI or CLI.ReferenceSupported featuresPlease see the original adapter documentation:https://github.com/dbt-labs/dbt-sparkandhttps://docs.getdbt.com/reference/warehouse-profiles/spark-profile
Package: dbt-spark-livy
dbt-spark-livyThedbt-spark-livyadapter allows you to usedbtalong withApache spark-livyandCloudera Data Platformwith Livy server support. This code bases use the dbt-spark project (https://github.com/dbt-labs/dbt-spark), and provides a Livy connectivity support over it.Getting startedInstall dbtRead theintroductionandviewpointRunning locallyAdocker-composeenvironment starts a Spark Thrift server and a Postgres database as a Hive Metastore backend.
Note: dbt-spark now supports Spark 3.1.1 (formerly on Spark 2.x).Python >= 3.8dbt-core ~= 1.3.0pysparksqlparamsrequests_kerberosrequests-toolbeltpython-decoupleInstalling dbt-spark-livypip install dbt-spark-livyProfile Setupdemo_project:
  target: dev
  outputs:
    dev:
     type: spark_livy
     method: livy
     schema: my_db
     host: https://spark-livy-gateway.my.org.com/dbt-spark/cdp-proxy-api/livy_for_spark3/
     user: my_user
     password: my_passCaveatsWhile using livy , in the Livy UI if you notice sessions change state to dead from starting instead of idle, make sure there is a proper mapping for the user in the IDBroker mapping sectionActions > Manage Access > IDBroker Mappings .ReferenceAlso make sure the workload password is set either through UI or CLI.ReferenceSupported featuresPlease see the original adapter documentation:https://github.com/dbt-labs/dbt-sparkandhttps://docs.getdbt.com/reference/warehouse-profiles/spark-profile
Package: dbt-sqlserver
dbt-sqlserverdbtadapter for Microsoft SQL Server and Azure SQL services.The adapter supports dbt-core 0.14 or newer and follows the same versioning scheme.
E.g. version 1.1.x of the adapter will be compatible with dbt-core 1.1.x.DocumentationWe've bundled all documentation on the dbt docs site:Profile setup & authenticationAdapter documentation, usage and important notesJoin us on thedbt Slackto ask questions, get help, or to discuss the project.InstallationThis adapter requires the Microsoft ODBC driver to be installed:Windows|macOS|LinuxDebian/UbuntuMake sure to install the ODBC headers as well as the driver linked above:sudoapt-getinstall-yunixodbc-devLatest version:pipinstall-Udbt-sqlserverLatest pre-release:pipinstall-U--predbt-sqlserverChangelogSeethe changelogContributingThis adapter is community-maintained.
You are welcome to contribute by creating issues, opening or reviewing pull requests or helping other users in Slack channel.
If you're unsure how to get started, check out ourcontributing guide.LicenseCode of ConductThis project and everyone involved is expected to follow thedbt Code of Conduct.
Package: dbt-subdocs
dbt-subdocsdbt-subdocs is a python CLI you can used to generate a dbt-docs for a subset of your dbt project🤔 DescriptionThis project is useful if you want to generate a dbt-docs site for a subset of the models in your DBT project.
By default, in dbt-docs, all your projects gets documented :all the modelsall the sourcesall the testsand all the macrosThis CLI is useful if you only want to document what your end-user will be using.This CLI simply edits themanifest.jsonandcatalog.jsonused by the dbt-docs site so they do not
contain nodes you don't want to display.✨ FeaturesConfigure an input and output directorySelect the models to document using tag within DBTChoose to exclude nodes that are useless for your users : tests, macros, seed etc...🏃 Getting StartedInstallation with pippipinstall-Udbt-subdocsThen you can rundbt-subdocs--helpFirst call to the CLIYou can call dbt-subdocs by simply using the commanddbt-subdocsSee all the options available usingdbt-subdocs --helpUsecase 1: Only display nodes with a specific tagAssuming yourmanifest.jsonandcatalog.jsonare inDIRECTORY, simply callcdDIRECTORY
dbt-subdocs--tagfinanceIf you want to select nodes with tagsfinanceORengineering, simply calldbt-subdocs--tagfinance--tagengineeringUsecase 2: Removing macros from the docsIf you want to remove macros from themanifest.jsonyou can calldbt-subdocs--tagfinance--exclude-node-typemacrosYou can also remove sources by usingdbt-subdocs--tagfinance--exclude-node-typemacros--exclude-node-typesources🛡️ LicenseThis project is licensed under the terms of theMITlicense. SeeLICENSEfor more details.📃 Citation@misc{dbt-subdocs,author={jb-delafosse},title={dbt-subdocs is a python CLI you can used to generate a dbt-docs for a subset of your dbt project},year={2022},publisher={GitHub},journal={GitHub repository},howpublished={\url{https://github.com/jb-delafosse/dbt-subdocs}}}CreditsThis project was generated withpython-package-template
Package: dbt-sugar
Documentation ongitbookIf you want to help out and join the party feel free to:create an issueor more funjoin us onDiscord:sparkles: andcheck theROADMAP.mdto see what's in the plans.dbt-sugar :candy::warning:dbt-sugar is pretty alpha, don't use on your prod models unless you have tested it in a safe place before.What is dbt-sugar?dbt-sugar is a CLI tool that allows users ofdbtto have fun and ease performing actions around dbt models such as:documentationtest enforcement
and probably more in a not too distant future.We keep track of our progress in theprojectssection.
Package: dbt-synapse
dbt-synapsedbtadapter for Azure Synapse Dedicated SQL Pool (Azure Synapse Data Warehouse).The adapter supports dbt-core 0.18 or newer and follows the same versioning scheme.
E.g. version 1.1.x of the adapter will be compatible with dbt-core 1.1.x.DocumentationWe've bundled all documentation on the dbt docs site:Profile setup & authenticationAdapter-specific configurationJoin us on thedbt Slackto ask questions, get help, or to discuss the project.InstallationThis adapter requires the Microsoft ODBC driver to be installed:Windows|macOS|LinuxDebian/UbuntuMake sure to install the ODBC headers as well as the driver linked above:sudoapt-getinstall-yunixodbc-devLatest version:pipinstall-Udbt-synapseLatest pre-release:pipinstall-U--predbt-synapseChangelogSeethe changelogContributingThis adapter is managed by Microsoft.
You are welcome to contribute by creating issues, opening or reviewing pull requests or helping other users in Slack channel.
If you're unsure how to get started, check out ourcontributing guide.This adapter uses thedbt-sqlserveradapter underneath so make sure to check out that repository as well.LicenseCode of ConductThis project and everyone involved is expected to follow thedbt Code of Conduct.
Package: dbt-tests-adapter
dbt-tests-adapterFor context and guidance on using this package, please read:"Testing a new adapter"What is it?This package includes reusable test cases that reinforce behaviors common to all or many adapter plugins. There are two categories of tests:Basic teststhat every adapter plugin is expected to pass. These are defined intests.adapter.basic. Given differences across data platforms, these may require slight modification or reimplementation. Significantly overriding or disabling these tests should be with good reason, since each represents basic functionality expected by dbt users. For example, if your adapter does not support incremental models, you should disable the test,by marking it withskiporxfail, as well as noting that limitation in any documentation, READMEs, and usage guides that accompany your adapter.Optional tests, for second-order functionality that is common across plugins, but not required for basic use. Your plugin can opt into these test cases by inheriting existing ones, or reimplementing them with adjustments. For now, this category includes all tests located outside thebasicsubdirectory. More tests will be added as we convert older tests defined on dbt-core and mature plugins to use the standard framework.How to use it?Each test case in this repo is packaged as a class, prefixedBase. To enable a test case to run with your adapter plugin, you should inherit the base class into a new class, prefixedTest. That test class will be discovered and run bypytest. It can also makes modifications if needed.classTestSimpleMaterializations(BaseSimpleMaterializations):passDistributionTo install:pipinstalldbt-tests-adapterThis package is versioned in lockstep withdbt-core, andthe same versioning guidelinesapply:New "basic" test cases MAY be added in minor versions ONLY. They may not be included in patch releases.Breaking changes to existing test cases MAY be included and communicated as part of minor version upgrades ONLY. They MAY NOT be included in patch releases. We will aim to avoid these whenever possible.New "optional" test cases, and non-breaking fixes to existing test cases, MAY be added in minor or patch versions.Assuming you adapter plugin is pinned to a specific minor version ofdbt-core(e.g.~=1.1.0), you can use the same pin fordbt-tests-adapter.Note:This is packaged as a plugin using a python namespace package. It cannot have an__init__.pyfile in the part of the hierarchy to which it needs to be attached.
Package: dbt-tidb
dbt-tidbThedbt-tidbpackage contains all of the code enablingdbtto work withTiDB.This repository is partly based ondbt-mysql.
Thanks to them for their excellent work.Table of ContentsInstallationSupported featuresSupported functionsProfile ConfigurationDatabase User PrivilegesRunning TestsExampleContributingInstallationCompile by source code.$gitclonehttps://github.com/pingcap/dbt-tidb.git
$cddbt-tidb
$pipinstall.Also, you can get it from pypi.$pipinstalldbt-tidbSupported featuresTiDB 4.XTiDB 5.0 ~ 5.2TiDB >= 5.3Feature✅✅✅Table materialization✅✅✅View materialization❌❌✅Incremental materialization❌✅✅Ephemeral materialization✅✅✅Seeds✅✅✅Sources✅✅✅Custom data tests✅✅✅Docs generate❌❌✅Snapshots✅✅✅Connection retry✅✅✅GrantNote:TiDB 4.0 ~ 5.0 does not supportCTE,
you should avoid usingWITHin your SQL code.TiDB 4.0 ~ 5.2 does not support creating atemporary table or view.TiDB 4.X does not support using SQL func inCREATE VIEW, avoid it in your SQL code.
You can find more detailhere.Supported functionscross-db macros are moved from dbt-utils into dbt-core, so you can use the following functions directly, seedbt-utilon how to use them.bool_orcast_bool_to_textdateadddatediffdate_trunchashsafe_castsplit_partlast_daycast_bool_to_textconcatescape_single_quotesexceptintersectlengthpositionreplacerightlistagg (not support yet)pay attention that datediff is a little different from dbt-util that it will round down rather than round up.Profile ConfigurationTiDB targets should be set up using the following configuration in yourprofiles.ymlfile.Example entry for profiles.yml:your_profile_name:
  target: dev
  outputs:
    dev:
      type: tidb
      server: 127.0.0.1
      port: 4000
      schema: database_name
      username: tidb_username
      password: tidb_password
      retries: 2OptionDescriptionRequired?ExampletypeThe specific adapter to useRequiredtidbserverThe server (hostname) to connect toRequiredyourorg.tidb.comportThe port to useRequired4000schemaSpecify the schema (database) to build models intoRequiredanalyticsusernameThe username to use to connect to the serverRequireddbt_adminpasswordThe password to use for authenticating to the serverRequiredcorrect-horse-battery-stapleretriesThe retry times for connection to TiDB (1 in default)Optional2Database User PrivilegesYour database user would be able to have some abilities to read or write, such asSELECT,CREATE, and so on.
You can find some helpherewith TiDB privileges management.Required PrivilegeSELECTCREATECREATE TEMPORARY TABLECREATE VIEWINSERTDROPSHOW DATABASESHOW VIEWSUPERRunning TestsSeetests/README.mdfor details on running the integration tests.ExampleClickhereto see a simple example about using dbt with dbt-tidb.ContributingWelcome to contribute for dbt-tidb. SeeContributing Guidefor more information.
Package: dbt-timescaledb
dbt-timescaledbdbtenables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.TimescaleDBTimescaleextends PostgreSQL for all of your resource-intensive production workloads, so you can build faster, scale further, and stay under budget.Supported versionsThis adapter supports dbt Core 1.7.0 and newer. It is not available on dbt Cloud.This adapter supports TimescaleDB 2.12 and newer. Older versions of TimescaleDB will probably also work if you don't use any of the features that were added in newer versions.The recommended versions of Timescale are TimescaleDB Community Edition or Timescale cloud. It is not recommended to use this adapter with TimescaleDB Apache 2 Edition. See theTimescaleDB editionspage for more information.Features & documentationRead the documentation(installation|usage) for more information.Code of ConductBoth dbt Labs and Timescale have published a code of conduct. Everyone interacting in this project's codebases, issues, discussions, and related Slack channels is expected to follow thedbt Code of Conductand theTimescale Code of Conduct.
Package: dbt-toolkit
dbt-toolkitA collection of utilities and tools for teams and organizations using dbt.This project is in an ALPHA stage. Internal and external APIs might change between minor versions.Please reach out if you try this at your own organization. Feedback is very appreciated, and we
would love to hear if you had any issues setting this up at your own.AutomationsDocumentationPropagates the documentation of columns that have the same name on downstream models, improving documentation
coverage while reducing manual repeated workMore information can be found on the package'sREADME.dbt Cloud artifactsRetrieve artifacts from a dbt Cloud project. Useful for building reports (such as test and documentation coverage)More information can be found on the package'sREADME.InstallationThis project requires Python 3.8+. You can install the latest version of this package from PyPI by running the
command below.$pipinstalldbt-toolkitDevelopmentContributingWe are open and would love to have contributions, both in Pull Requests but also in ideas and feedback. Don't hesitate
to create an Issue on this repository if you are trying this project in your organization or have anything to share.Remember to runmake lint,make typeandmake testbefore committing.Runmake install-devto install it locally in editable mode. This is necessary for running the tests.ArchitectureGroups of functionalities are encapsulated together in top-level packages, such asdbt_cloud/ordocumentation/.
Each package that exposes CLI commands should contain anactionssub-package.TestsMore information can be found on the tests'README.ReleaseThere is a GitHub Action that will trigger a release of this package on PyPI based on releases created on GitHub.
Steps:Loosely followsemantic versioningRemember to prefix the tag name withvUse the tag name as the release title on GitHubUse the auto-generated release notes from GitHubAppend a link at the end of the release notes to the released version on PyPILicenseThis project is licensed under the Apache License, Version 2.0:http://www.apache.org/licenses/LICENSE-2.0.
Package: dbt-tool-kit
dbt ToolkitSwiss knife for everyday dbt worksInstallationpipinstalldbt-tool-kitUsage>dbt-toolkit--helpUsage:dbt-toolkit[OPTIONS]COMMAND[ARGS]...

╭─Options───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│--install-completionInstallcompletionforthecurrentshell.│
│--show-completionShowcompletionforthecurrentshell,tocopyitorcustomizetheinstallation.│
│--helpShowthismessageandexit.│
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─Commands──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│convert-to-dbt-jinjaconvertaplainsqltodbtjinjasql,replacingexistingsourcenodewithdbtexpression│
│extract-cte-to-modelreadyoursqlfile,extractCTEstatementandmakedbtmodelsoutofthem│
│update-source-from-dwfindyoursource,scanitfromyourconnection,andsynccolumns,description,etc.toyoursources.yaml│
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯