Saravanan666/pypi-org huggingface dataset:
Package: dbt-ocean-spark
dbtenables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.dbt-sparkThedbt-ocean-sparkpackage contains all of the code enabling dbt to work with Apache Spark on Ocean. For
more information, consultthe docs.Getting startedInstall dbtRead theintroductionandviewpointRunning locallyAdocker-composeenvironment starts a Spark Thrift server and a Postgres database as a Hive Metastore backend.
Note: dbt-spark now supports Spark 3.1.1 (formerly on Spark 2.x).The following command would start two docker containersdocker-compose up -dIt will take a bit of time for the instance to start, you can check the logs of the two containers.
If the instance doesn't start correctly, try the complete reset command listed below and then try start again.Create a profile like this one:spark_testing:
  target: local
  outputs:
    local:
      type: ocean_spark
      method: http
      host: api.spotinst.io
      port: 443
      cluster: osc-[clusterId]
      account: acc-[accountId]
      app: spark-hive-[appId]
      schema: analytics
      connect_retries: 5
      connect_timeout: 60
      retry_all: trueConnecting to the local spark instance:The Spark UI should be available athttp://console.spotinst.com/ocean/spark/cluster/{clusterId}/app/{appId}/sparkUI/jobsThe endpoint for SQL-based testing is athttp://localhost:10000and can be referenced with the Hive or Spark JDBC drivers using connection stringjdbc:hive2://localhost:10000and default credentialsdbt:dbtNote that the Hive metastore data is persisted under./.hive-metastore/, and the Spark-produced data under./.spark-warehouse/. To completely reset you environment run the following:docker-compose down
rm -rf ./.hive-metastore/
rm -rf ./.spark-warehouse/Reporting bugs and contributing codeWant to report a bug or request a feature? Let us know onSlack, or openan issue.Code of ConductEveryone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow thePyPA Code of Conduct.Join the dbt CommunityBe part of the conversation in thedbt Community SlackRead more on thedbt Community DiscourseReporting bugs and contributing codeWant to report a bug or request a feature? Let us know onSlack, or openan issueWant to help us build dbt? Check out theContributing GuideCode of ConductEveryone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow thedbt Code of Conduct.
Package: dbt-odps
dbtenables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.ODPSODPS, calledMaxComputebefore. This adapter is a wrapper bridged PyOdps and DBT together.MaxCompute Features:FeatureStatusPartition Table❎Cluster Table❎External Table❎Table Properties❎DBT features:NameStatusMaterialization: Table✅Materialization: View✅Materialization: Incremental - Append✅Materialization: Incremental - Insert+Overwrite✅Materialization: Incremental - Merge✅Materialization: Ephemeral✅Seeds✅Tests✅Snapshots✅1Documentation✅Getting StartedInstall DBT-ODPSpython-mpipinstalldbt-odpsHow to Configure?For Fresh CodebaseRun following command after installing dbt-odps:dbtinitFor Existing CodebaseRead more in here:https://docs.getdbt.com/docs/core/connection-profilesConfiguration options:PropertyDescriptionExampleEndpointThe endpoint of odps, read more inhttps://help.aliyun.com/document_detail/34951.htmlhttp://service.cn-shanghai.maxcompute.aliyun.com/apidatabaseTheproject nameof odps, which you can find inhttps://maxcompute.console.aliyun.com/{yourarea}/project-listodps-test-projectschemaUsingdefaultif you don't know what is schema.defaultaccess_idaccess idLTAXXXXXXXXXsecret_access_keysecret access keybZXXXXXXXXXXtypeodpsodpsNOTESWhen using merge statement, ODPS required that table is a transactional table. So, we have to create the snapshot table before select. Under the hook, we using the first referred table as source data structure to create table, so this data source must be a table, view is not supported.
Package: dbt-odps-winwin
dbtenables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.ODPSODPS, calledMaxComputebefore. This adapter is a wrapper bridged PyOdps and DBT together.MaxCompute Features:FeatureStatusPartition Table❎Cluster Table❎External Table❎Table Properties❎DBT features:NameStatusMaterialization: Table✅Materialization: View✅Materialization: Incremental - Append✅Materialization: Incremental - Insert+Overwrite✅Materialization: Incremental - Merge✅Materialization: Ephemeral✅Seeds✅Tests✅Snapshots✅1Documentation✅Getting StartedInstall DBT-ODPSpythonsetup.pyinstall--forceorpipinstalldbt-odps-winwinHow to Configure?For Fresh CodebaseRun following command after installing dbt-odps:dbtinitFor Existing CodebaseRead more in here:https://docs.getdbt.com/docs/core/connection-profilesConfiguration options:PropertyDescriptionExampleEndpointThe endpoint of odps, read more inhttps://help.aliyun.com/document_detail/34951.htmlhttp://service.cn-shanghai.maxcompute.aliyun.com/apidatabaseTheproject nameof odps, which you can find inhttps://maxcompute.console.aliyun.com/{yourarea}/project-listodps-test-projectschemaUsingdefaultif you don't know what is schema.defaultaccess_idaccess idLTAXXXXXXXXXsecret_access_keysecret access keybZXXXXXXXXXXtypeodpsodpsNOTESWhen using merge statement, ODPS required that table is a transactional table. So, we have to create the snapshot table before select. Under the hook, we using the first referred table as source data structure to create table, so this data source must be a table, view is not supported.DEVELOPER REFDBT docswhat-are-adapters
Package: dbtool
dbtoolA lightweight db tools for sql.pip install dbtool# sqlite3 ....
db = dbtool.connect('sqlite:///:memory:')
db = dbtool.connect('mysql://root:123456@127.0.0.1:3306/test',  mincached=1, maxconnections=20)

# sql
db.execute(sql)
db.execute_fetchone(sql)
db.execute_cursor(sql)
db.execute_batch(sql)
db.execute_script(sql)

# crud
db.insert(user)
db.update(user)
db.delete(User, {'id': 1})
db.find(User, {'status': 1})
db.find_one(User, {'id': 1})
db.find_count(User, {'status': 1})

# transactions
with db.transaction():
    db.execute(sql1)db vs driversqlite - sqlite3mysql - pymysqlpostgresql - psycopg2sqlserver - pymssql
Package: dbt-opengauss
dbtenables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.dbt-opengaussThedbt-opengausspackage contains all of the code enabling dbt to work with a openGauss database.Getting startedInstall dbtRead theintroductionandviewpointJoin the dbt CommunityBe part of the conversation in thedbt Community SlackRead more on thedbt Community DiscourseReporting bugs and contributing codeWant to report a bug or request a feature? Let us know onSlack, or openan issueWant to help us build dbt? Check out theContributing GuideCode of ConductEveryone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow thedbt Code of Conduct.
Package: dbt-oracle
dbt-oracledbt-oracleimplementsdbt (data build tool)functionalities for Oracle Autonomous Database.Prior to version 1.0.0, dbt-oracle was created and maintained byIndiciumontheir GitHub repo. Contributors in this repo are credited for laying the groundwork and maintaining the adapter till version 0.4.3.
From version 1.0.0, dbt-oracle is maintained and distributed by Oracle.Installationpipinstalldbt-oracleDocumentationPlease refer to theOracle setup on dbt docs websitefor documentation.Sample projectTo get started, a sample dbt project can be found in the directory/dbt_adbs_test_project.ContributingThis project welcomes contributions from the community. Before submitting a pull request, pleasereview our contribution guide.SecurityPlease consult thesecurity guidefor our responsible security vulnerability disclosure process.Licensedbt-oracle is licensed under Apache 2.0 License which you can findhere.
Package: dbtos3
Replication & Full Load Application for multiple databases to s3How to usePlease make use of theWikiwhich will be updated continuously with information.Current featuresDatabasesPostgreSQL to S3 - full load and replicationMySQL to S3 - full load and replicationApplicationsSentry to S3 - full load and replicationorganization projectsorganisation project eventsExchange Rates API to S3 - full load and replicationHistorical Exchange RatesDaily Exchange RatesCourtesy ofexchangeratesapi.ioMore will be added over timeDevelopmentThis project is developed and maintained by thePerceptech Data Software Foundation
Package: dbt-postgres
dbtenables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.dbt-postgresThedbt-postgrespackage contains all of the code enabling dbt to work with a Postgres database. For
more information on using dbt with Postgres, consultthe docs.Getting startedInstall dbtRead theintroductionandviewpointJoin the dbt CommunityBe part of the conversation in thedbt Community SlackRead more on thedbt Community DiscourseReporting bugs and contributing codeWant to report a bug or request a feature? Let us know onSlack, or openan issueWant to help us build dbt? Check out theContributing GuideCode of ConductEveryone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow thedbt Code of Conduct.
Package: dbt-quicksight-lineage
dbt_quicksight_lineagePyPI:https://pypi.org/project/dbt-quicksight-lineage/Target DBT Version: 1.5.*Table of ContentsInstallationLicenseInstallationpip install dbt-quicksight-lineageUsageUsage: dbt-quicksight-lineage [OPTIONS] COMMAND [ARGS]...dbt-quicksight-lineage: DBT to QuickSight Lineage command helperOptions:--version                       Show the version and exit.--log-level [DEBUG|INFO|WARNING|ERROR|CRITICAL]Set log level--no-color / --color            Disable color in output-h, --help                      Show this message and exit.Commands:init             Modify schema.yml to add QuickSight metadata with Data...update-data-set  Update QuickSight DataSet from DBT ManifestQuickStartdbt-quicksight-lineage init --project-dir /path/to/dbt/project --data-set-id <data-set-id>modify yourschema.ymlto add QuickSight metadata with Data Lineagefor example:version:2models:-name:my_first_dbt_modeldescription:"Astarterdbtmodel"meta:quicksight:logical_table_name:My First DBT Modeldata_sets:-id:00000000-0000-0000-0000-000000000000data_source_arn:arn:aws:quicksight:ap-northeast-1:123456789012:datasource/00000000-0000-0000-0000-000000000000folders:-name:Keydescription:"thisisKeyfolder,mustdistinctcounts"columns:-name:iddescription:"Theprimarykeyforthistable"meta:quicksight:field_name:IDfolder:Keydata_type:stringtests:-unique-not_null-name:geodescription:"Citynameofgeometry"meta:quicksight:geographic_role:cityfield_name:Geometryfolder:Key-name:hidden_valuemeta:quicksight:hidden:true-name:my_second_dbt_modeldescription:"Astarterdbtmodel"columns:-name:iddescription:"Theprimarykeyforthistable"tests:-unique-not_nulland update data setdbt-quicksight-lineage update-data-set --project-dir /path/to/dbt/project --data-set-id <data-set-id>Licensedbt-quicksight-lineageis distributed under the terms of theMITlicense.
Package: dbt-redshift
dbtenables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.dbt-redshiftThedbt-redshiftpackage contains all of the code enabling dbt to work with Amazon Redshift. For
more information on using dbt with Redshift, consultthe docs.Getting startedInstall dbtRead theintroductionandviewpointJoin the dbt CommunityBe part of the conversation in thedbt Community SlackRead more on thedbt Community DiscourseReporting bugs and contributing codeWant to report a bug or request a feature? Let us know onSlack, or openan issueWant to help us build dbt? Check out theContributing GuideCode of ConductEveryone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow thedbt Code of Conduct.
Package: dbtrolls
dbtrollsA simple tool to make Dev’s life happy.LicenseLicensed under the GNU General Public License (GPL).InstallFollow the steps below to get everything up and running.pipPackage Index > dbtrolls > 1.1pip.Create project folder:$mkdirdbtrolls&&cddbtrollsCreate virtualenv in the normal way:$virtualenvenv--python=pythonActive your new virtualenv:$sourceenv/bin/activateInstall dbtrolls:$pipinstalldbtrollsGitClone repository:$gitclonegit@github.com:caputomarcos/dbtrolls.gitGo to dbtrolls source folder:$cddbtrolls/Create virtualenv in the normal way:$virtualenvenv--python=pythonActive your new virtualenv:$sourceenv/bin/activateCreate dev environment:$makedevelopUsageCreate config file:$dbtrolls-c--database_source=<DATABASE_SOURCE>--database_target=<DATABASE_TARGET>Execute a single SQL file:$dbtrolls-s<SQL_FILE>--fix--preloadExecute a collection of SQL files:$dbtrolls-m
Package: dbt-rpc-client
dbt-rpc-clientA python SDK for interacting with adbt rpc server.Installation$pipinstalldbt-rpc-clientBasic Usagefromdbt_rpc_clientimportDbtRpcClientimportrequestsrpc=DbtRpcClient(hostname="0.0.0.0",port=8580)# Getting the current status of the dbt rpc server.response=rpc.status()assertisinstance(response,requests.Response)assertresponse.ok# Running dbt models via CLI command.response=rpc.cli(cli_args="dbt run --models @model1 +model2+")assertresponse.ok# Running dbt models via `run` method.response=rpc.run(models=["@model1","+model2+"])assertresponse.ok# Polling a dbt rpc operation.response=rpc.poll(request_token=response.get("id"))assertresponse.okprint(response.get("result").get("status"))# Compiling/Running a SQL query.sql="""select 1from {{ ref("my_dbt_model") }}"""response=rpc.compile_sql(sql=sql,name="my_sql_query")assertresponse.okresponse=rpc.run_sql(sql=sql,name="my_sql_query")assertresponse.okContributingThe first step to contributing is getting a copy of the source code. First,forkdbt-rpc-clienton GitHub. Then,cdinto the directory where you want your copy of the source code to live and clone the source code:$cdrepos
$gitclonegit@github.com:YourGitHubName/dbt-rpc-client.gitNow that you have a copy of the source code on your machine, create and activate a virtual envionment fordbt-rpc-client:$python3-mvenv~/.venvs/dbt-rpc-client
$source~/.venvs/dbt-rpc-client/bin/activateOnce inside the virtual environment, runmake dev_installat the root of the repository:$(dbt-rpc-client)makedev_installRun thetoxtesting suite in the appropriate python environment to ensure things are working properly:$(dbt-rpc-client)tox-epy37To format your code usingisortandflake8before commiting changes, run the following commands:$(dbt-rpc-client)makeisort
$(dbt-rpc-client)makeflake8Once you've confirmed that your changes work and the testing suite passes, feel free to put out a PR!
Package: dbt-sas
SAS DBT adapterThe dbt-sas package allows dbt to connect to SAS.Installationpip install dbt-sasConfigurationsBasicprofile.ymlfor connecting to SAS:sas-test:
  outputs:
    dev:
      type: sas
      host: SAS host
      port: 8591
      database: sas (keep 'sas', don't change)
      schema: default libray name
      user: SAS username
      password: SAS password
      autoexec: optional local path of autoexec.sas file
      lib_base_path: optional path (on the SAS server) for new libraries
      threads: 1
      fail_on_warnings: False
      lib_name_strict_mode: False
  target: devKeyRequiredDescriptiontypeYesThe specific adapter to use (sas)hostYesSAS server hostnameportYesSAS server portdatabaseYessas, don't changeschemaYesDefault SAS libnameusernameYesThe username to use to connect to the serverpasswordYesThe password to use for authenticating to the serverhandlerYesSAS handlerws(default) orsaspyautoexecNoLocal path of autoexec.sas filelib_base_pathNoBase path on the SAS server for new librariesfail_on_warningsNoRaise and error if it encounters a warninglib_name_strict_modeNoEnable strict libname/identifier checkFeaturesKeySupportedDescriptionTestsYesRun dbt tests on SASLoad seed filesYesLoad seeds from CSV files to SASView MaterializationYesMaterialize dbt models as views in SASTable MaterializationYesMaterialize dbt models as tables in SASEphemeral MaterializationPartialMaterialize dbt ephemeral tables as temporary views in SASIncremental MaterializationNoSnapshotsNoGrantsNoSQL grants are not supported by SASLimitationsSchemas (lib names) are limited to 8 characters.Table names, column names, and aliases are limited to 32 characters.UsageCreate dbt project, choose sas database and set up connection$dbtinit<project_name>Query LoggingEnable additional logging by setting theDBT_SAS_LOGenvironment variable to the log filename, e.g.:export DBT_SAS_LOG='file.log'Select the log families, e.g.:export DBT_SAS_FAMILY='sas,sql'Log families are:sassqloriginal_sqlAdapter TestingInstall dev requirements$pipinstall-rdev_requirements.txtRun pytest$python-mpytesttests/LicenceApache License, Version 2.0LinksSASpySQL Procedure
Package: dbt-schema-builder
The Schema Builder tool is used to create dbt schema files, sql models, and
default PII / non-PII views for tables in the given Snowflake schemas.For each specified application schema, the script will generate dbt models for
a<SCHEMA>and<SCHEMA>_PIIschema. We refer to these schemas as a
“trifecta”.<SCHEMA>_<RAW_SUFFIX>contains the original source tables.<SCHEMA>_PIIcontains views on the _RAW tables that have un-redacted PII.<SCHEMA>contains views on the _RAW tables sensitive data redacted.Application schemas can be sourced from multiple raw schemas. This allows you
to specify which tables should be pulled from which raw schema to construct the
“trifecta”.Schema Builder ensures that all three schemas provide the same interface to the
data (number and order of columns match what is present in the _RAW schema).Once the script is successfully run, you can execute adbt runto create or
update the views in<SCHEMA>and<SCHEMA>_PII. If your source data in
the<SCHEMA>_<RAW_SUFFIX>schema changes you should run Schema Builder frequently
to keep up with changes in the tables and columns stored there.Schema Builder will also automatically create sources in one or more other dbt
projects so that they can use the results of these models as sources.Seethe docsfor more
information.LicenseThe code in this repository is licensed under the AGPL 3.0 unless
otherwise noted.Please seeLICENSE.txtfor details.How To ContributeContributions are very welcome.
Please readHow To Contributefor details.
Even though they were written withedx-platformin mind, the guidelines
should be followed for all Open edX projects.The pull request description template should be automatically applied if you are creating a pull request from GitHub. Otherwise you
can find it atPULL_REQUEST_TEMPLATE.md.The issue report template should be automatically applied if you are creating an issue on GitHub as well. Otherwise you
can find it atISSUE_TEMPLATE.md.Reporting Security IssuesPlease do not report security issues in public. Please emailsecurity@edx.org.Getting HelpIf you’re having trouble, we have discussion forums athttps://discuss.openedx.orgwhere you can connect with others in the community.Our real-time conversations are on Slack. You can request aSlack
invitation, then join ourcommunity Slack team.For more information about these options, see theGetting Helppage.
Package: dbt-schema-generator
dbt Schema GeneratorA command-line tool to generate schema.yml files for specified dbt models.InstallationInstall the package using pip:pip install dbt_schema_generatorUsageYou can use the dbt_schema_generator command-line tool with the following options:-m, --models: A comma-separated list of model names to include in the schema.yml file.
-p, --path: A path to include models in the schema.yml file.Examples:dbt_schema_generator -m model1,model2,model3
dbt_schema_generator --models model1,model2,model3# Using this format, dbt_schema_generator would generate a schema.yml file for all models within your_folder
dbt_schema_generator -path models/your_folder
dbt_schema_generator --path models/your_folderNOTE: When using path the format should always be models/ the folder unless you have changed the model-paths in your dbt_project.ymlNOTE: When using the--pathoption, the format should always bemodels/your_folderunless you have changed themodel-pathsin yourdbt_project.yml.!!!!IMPORTANT!!!!: Be very careful when using the --path method as this would overwrite any schema.yml currently within the folder
Package: dbtsdk
dbt-sdk-pythonA Python implementation of DBT API SDK (ported fromdbt-sdk-php)RequirementsPython 3 (Recommended 3.6+)Installationsudopython3-mpipinstalldbtsdkUsageBelow are the steps neccessary to get you started with the DBT SDK.1. Sign up for an API KeyBefore you can start using the DBT SDK, you will need to sign up for an API Key athttps://www.digitalbibleplatform.com.2. (Optional) Add your API key to your environmentIt's best not to hard code your API key in your script file. Instead, save it to your environment and let Python read it from there.On Windows:setx DBT_API_KEY"<your api key>"On Mac and Linux:exportDBT_API_KEY=<yourapikey>To make this environment persist between reboots, add this line to your.bash_profilefile instead.2. Include the SDK fileIn your Python script, import the package.fromdbtsdkimportDbt3. Create an instance of the objectWhen creating the object, pass the API key.importosapi_key=os.environ['DBT_API_KEY']dbt=Dbt.Dbt(api_key)4. Retrieve volume informationFor an example of the usual work flows, as well as examples of retrieving and displaying text, audio, and video, download thesample code.API DocumentationDocumentation for the underlying REST API can be found in theDigital Bible Platform Developer Documentation.Licensedbt-sdk-python is available under the MIT license. See the LICENSE file for more info.
Package: dbt-semantic-interfaces
dbt-semantic-interfacesThis repo contains the shared semantic classes, default validation, and tests designed to be used by both the dbt-core and MetricFlow projects. By centralizing these shared resources, we aim to maintain consistency and reduce code duplication across both projects.FeaturesProtocols for shared semantic classes: Define the interfaces and common attributes that must be implemented by the objects in both projects.Validation: Ensure that the objects comply with the expected structure and constraints.Tests: Ensure that the objects' behavior is consistent and correct across both projects.ContributingWe welcome contributions to improve this codebase! If you're interested in contributing, please read ourcontributing guidelinesandcode of conductfirst.LicenseThis package is released under the Apache2 License.SupportIf you encounter any issues or have questions regarding the repo, please open an issue!