Saravanan666/pypi-org huggingface dataset:
Package: db-testtools
This package contains test fixtures and resources that bring up an
isolated database and SQLAlchemy test fixtures, so that your Python
unit tests will run without interfering with each other when you are
using the SQLAlchemy ORM.The database is initialised once at the start of a test process run, as
is the session fixture. The session fixture ensures that any commits do
not permanently commit, and rolls back the database to a clean state
after each test completes.RequirementsPython 3.7 and beyond should work.QuickstartInstall with pip:pip install db-testtoolsExample base test class:classDBTestCase(testresources.ResourcedTestCase,testtools.TestCase):"""Base class for all DB tests.

   Brings up a temporary database and gives each test its own session.
   """# These are resources that stay active for the entire# duration of all the tests being run.db_fixture=DatabaseResource(ModelBase,'myproject.models',future=True,)resources=[('database',db_fixture)]defsetUp(self):super().setUp()self.session_fixture=SessionFixture(self.database,future=True)self.useFixture(self.session_fixture)# The session itself.self.session=self.session_fixture.session# The session factory.self.Session=self.session_fixture.SessionThis base test class will start a SQLite-based database by default and
injectself.sessionas the SQLAlchemy session andself.Sessionas the
SQLAlchemy session factory.If you need to use a different database, then you can either:pass theengine_fixture_nameparameter toDatabaseResourceset an environment variable TEST_ENGINE_FIXTUREwith the name of the engine fixture to use. Currently two are
available:SqliteMemoryFixturePostgresContainerFixtureEngine driversCurrently the two drivers mentioned above are implemented. The SQLite
fixture implements a simple in-memory database which is completely
dropped and re-instated on every test.The PostgresContainerFixture starts its own Postgres instance in a local
Docker container. Therefore you must have Docker installed before using
this fixture.Help needed!This fixture suite is currently not tested itself and would benefit from
anyone willing to contribute some unit tests. However, it has been in
use daily on a large project at Cisco for the last 2 years, and is very
stable.Copyrightdb-testtools is copyright (c) 2021 Cisco Systems, Inc. and its affiliates
All rights reserved.
Package: dbt-exasol
dbt-exasoldbtenables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.Please see the dbt documentation onExasol setupfor more information on how to start using the Exasol adapter.Current profile.yml settingsdbt-exasol:target:devoutputs:dev:type:exasolthreads:1dsn:HOST:PORTuser:USERNAMEpassword:PASSWORDdbname:dbschema:SCHEMAOptional login credentials using OpenID for Exasol SaaSOpenID login through access_token or refresh_token instead of user+passwordOptional parametersconnection_timeout: defaults to pyexasol defaultsocket_timeout: defaults to pyexasol defaultquery_timeout: defaults to pyexasol defaultcompression: default: Falseencryption: default: Trueprotocol_version: default: v3row_separator: default: CRLF for windows - LF otherwisetimestamp_format: default: YYYY-MM-DDTHH:MI:SS.FF6Known isuesUsing encryption in Exasol 7 vs. 8Starting from Exasol 8, encryption is enforced by default. If you are still using Exasol 7 and have trouble connecting, you can disable encryption in profiles.yaml (see optional parameters).Materialized View & Clone operationsIn Exasol materialized views and clone operations are not suported. Default behaviour from dbt-core will fail accordingly.Null handling in test_utils null safe handlingIn Exasol empty string are NULL. Due to this behaviour and as ofthis pull request 7776 published in dbt-core 1.6,
seeds in tests that use EMPTY literal to simulate empty string have to be handled with special behaviour in exasol.
See fixture for csv in exasol__seeds__data_hash_csv for tests/functional/adapter/utils/test_utils.py::TestHashExasol.Model contractsThe following database constraints are implemented for Exasol:Constraint TypeStatuscheckNOT supportednot nullenforceduniqueNOT supportedprimary keyenforcedforeign keyenforced>=1.5 Incrmental model updateFallback to dbt-core implementation and supporting strategiesappendmergedelete+insert>=1.3 Python model not yet supported - WIPPlease followthis pull requestBreaking changes with release 1.2.2Timestamp format defaults to YYYY-MM-DDTHH:MI:SS.FF6SQL functions compatibilitysplit_partThere is no equivalent SQL function in Exasol for split_part.listagg part_numThe SQL function listagg in Exasol does not support the num_part parameter.Utilities shim packageIn order to support packages like dbt-utils and dbt-audit-helper, we needed to create theshim package exasol-utils. In this shim package we need to adapt to parts of the SQL functionality that is not compatible with Exasol - e.g. when 'final' is being used which is a keyword in Exasol. Please visitAdaopter dispatch documentationof dbt-labs for more information.Reporting bugs and contributing codePlease report bugs using the issuesReleasesGitHub Releases
Package: dbt-excel
dbt-excel: Unleash the Power of Excel in Your Data StackThe ultimate solution for the data-driven worldWelcome to dbt-excel, the revolutionary dbt adapter that combines the rigor of dbt with the flexibility and familiarity of Excel. Get ready to change the way you look at data analytics forever.FeaturesIntegrate Excel with dbt: query data that you didn't know existed, for people who didn't know you existed, all from Excel!Use your favorite Excel functions: VLOOKUP, sheet support, averageif, and more, because life's too short for learning SQL functions.Runs blazingly fast queries thanks to duckdb running in the background, so you can have your cake and eat it too!Monitor your data assets, collaborate on data models, and document your queries, all within Excel. Remember, if it's not in Excel, it's not worth doing!InstallationTo install dbt-excel, just run the following command in your terminal:python-mpipinstalldbt-excelProfile setupThe default profile will create a duckdb file in /tmp, of the customexceltype. You can adjust the path if necessary.dbt_excel:
  target: dev
  outputs:
   dev:
     type: excel
     path: /tmp/dbt.duckdbRunning your first DBT Excel modelSteps:cdinto thedbt_projectdirectory, then rundbt run -s my_first_dbt_model --profiles-dir .this will run the models in themodels/example/my_first_dbt_model.sqlfile.A Excel file exists insources/people.xlsx, which will be read and used by the models. The result will be written tofirst_model.xlsx.Boom, you just read from AND wrote to Excel!Savour this moment, it's precious.Testimonials:Enterprise Architect:As an Enterprise Architect, I have been involved in countless failed digital transformation projects. They all make the same mistake. They forget one thing: all data lives in Excel. By moving away from enterprise-grade solutions like Excel to complicated stacks based on FivetPython, dbt and XXX, companies neglect their core competence. 1,048,576 rows ought to be enough for anybody. The only digital winners are the companies that build their stack on top of Excel.CEO:Companies worldwide are pouring trillions of dollars into digital transformation. But they forget one thing. In the end, all data lives in Excel. Dbt-excel combines the rigour of software engineering from dbt with the flexibility of Excel. True data democratization starts and ends with Excel.Anonymous Analyst:I used to have impostor syndrome working with DBT. With Excel I know I Excel.CreditsBig credits toJosh Wills' excellent DBT Duckdb adapterwhich inspired this project!
Package: dbtext
dbtextUtility for storing a database as a directory of plain text files, and reading from those text files.
Currently has implementations for MS SQL server, MySQL, Sqlite3 and MongoDB.
The first 3 of these require "pyodbc" to be installed, and MongoDB requires "pymongo" to be installed.installationpip install dbtext pyodbc (for MSSQL, MySQL, Sqlite3)
pip install dbtext pymongo (for MongoDB)usageFirst, dump your test database to this format. For example, with MSSQL:fromdbtextimportMSSQL_DBTextwithpyodbc.connect(connStr)asconn:withdbtext.MSSQL_DBText("dump",conn)astestdb:# the name "dump" doesn't matter, just a temporary nametestdb.write_data("db_tables",use_master_connection=True)# creates a directory called db_tablesFor MongoDB:fromdbtextimportMongo_DBTextdbClient=Mongo_DBText()dbClient.create(host=connStr)# as in pymongo, "host" can be a full connection stringdbClient.dump_data_directory("mongodata")# creates a directory called mongodataThen you create tests, probably with TextTest, that use this directory as test data ("copy_test_path")A test harness script might do something like for MSSQL:importdbtext,ostestdbname="ttdb_"+str(os.getpid())# some temporary name not to clash with other testswithdbtext.MSSQL_DBText(testdbname)asdb:# the name you use here will be used for the directory name in the current working directory# You need a script 'create_empty.sql' that sets up the schema but no data# db.create will set up the schema and read the test data from a directory here called "db_tables"db.create(sqlfile="create_empty.sql")# Then it should take the testdbname and configure your system to start a server against the new database# ...do_some_setup()# tell it the test is starting for real now. Only necessary if the database is changed by setup via the systemdb.update_start_rv()# do whatever it is the test doesdb.dumptables("myext","*")# dump changes in all the tables you're interested in. "myext" is whatever extension you want to use, probably the TextTest oneFor MongoDB, two classes are provided, Mongo_DBText and LocalMongo_DBText.
For test usage, you generally want to use LocalMongo_DBText, which works like the MSSQL version above,
i.e. it creates you a new MongoDB server running on any free port locally and populates it with the data in question.
Mongo_DBText is used for connecting to already running instances, for example if you need to start it via Docker.importdbtextwithdbtext.LocalMongo_DBText(data_dirname=testdbname)asdb:# the name you use here will be used for the directory name in the current working directorydb.create()ifnotdb.setup_succeeded():# could not start MongoDB, for examplereturnFalsetestConnStr="mongodb://localhost:"+str(self.db.port)# provide to your system in some way# do whatever it is the test does# ...db.dump_changes("myext")# dump changes in all the tables you're interested in. "myext" is whatever extension you want to use, probably the TextTest one
Package: dbt-fabric
dbt-fabricdbtadapter for Microsoft Fabric Synapse Data Warehouse.The adapter supports dbt-core 1.4 or newer and follows the same versioning scheme.
E.g. version 1.1.x of the adapter will be compatible with dbt-core 1.1.x.DocumentationWe've bundled all documentation on the dbt docs siteProfile setup & authenticationAdapter documentation, usage and important notesInstallationThis adapter requires the Microsoft ODBC driver to be installed:Windows|macOS|LinuxDebian/UbuntuMake sure to install the ODBC headers as well as the driver linked above:sudoapt-getinstall-yunixodbc-devLatest version:pipinstall-Udbt-fabricChangelogSeethe changelogContributingThis adapter is Microsoft-maintained.
You are welcome to contribute by creating issues, opening or reviewing pull requests.
If you're unsure how to get started, check out ourcontributing guide.LicenseCode of ConductThis project and everyone involved is expected to follow theMicrosoft Code of Conduct.
Package: dbt-firebolt
dbt-fireboltThedbtadapter forFirebolt. dbt-firebolt offers the following benefits to Firebolt customers:SQL-based data transformationLife cycle management for native Firebolt objects (fact tables, dimension tables, primary indexes, aggregating indexes, join indexes, etc.)Declarative, version-controlled data modelingAuto-generated data lineage and documentationdbt-firebolt supports dbt 1.0+.InstallationInstall the dbt-firebolt package from PyPI:pip install dbt-fireboltSetupConnecting to FireboltTo connect to Firebolt from dbt, you'll need to add a new Firebolt profile to yourprofiles.ymlfile. Please see thedbt documentation on Firebolt profilesto set it up.Setup RecommendationsFor the best experience we recommend that you make the following changes to your dbt project:Set an explicit value forquote_columnsAdd thegenerate_alias_namemacro to your projectFeature SupportThe table below shows which dbt and Firebolt features are supported by the adapter. dbt-firebolt is under active development and will be gradually unlocking more features over time.FeatureSupportedTable materializations:white_check_mark:Ephemeral materializations:white_check_mark:View materializations:white_check_mark:Incremental materializations - append:white_check_mark:Incremental materializations - insert_overwrite:white_check_mark:Incremental materializations - delete+insert:white_check_mark:Incremental materializations - merge:x:Snapshots:x:Seeds:white_check_mark:Tests:white_check_mark:Documentation:white_check_mark:Custom schemas:x: (seeworkaround)Custom databases:x:Source freshness:white_check_mark:External tables:white_check_mark:Primary indexes:white_check_mark:Aggregating indexes:white_check_mark:Join indexes:x: (syntax supported, but not effective)Constraints supportMore on constraints inPlatform constraint supportConstraint typeSupportPlatform enforcementnot_null:white_check_mark: Supported:white_check_mark: Enforcedprimary_key:x: Not Supported:x: Not enforcedforeign_key:x: Not Supported:x: Not enforcedunique:white_check_mark: Supported:x: Not enforcedcheck:x: Not supported:x: Not enforcedUsing dbt-fireboltFor information on configuring dbt models and external tables for Firebolt, see thedbt documentation for Firebolt configurations.ContributingSee:CONTRIBUTING.MDChangelogSee ourchangelogor our release history for more information.
Package: dbt-flink
dbtenables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.dbt-flinkThedbt-flinkpackage contains all of the code enabling dbt to work with Flink. For
more information on using dbt with Flink, consultthe docs.Getting startedInstall dbtRead theintroductionandviewpointJoin the dbt CommunityBe part of the conversation in thedbt Community SlackIf one doesn't exist feel free to request a #db-Flink channel be made in the#channel-requestson dbt community slack channel.Read more on thedbt Community DiscourseReporting bugs and contributing codeWant to report a bug or request a feature? Let us know onSlack, or openan issueWant to help us build dbt? Check out theContributing GuideCode of ConductEveryone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow thedbt Code of Conduct.
Package: dbt-gen
dbt-genTool for generating dbt model, source files.Commands1. Generate dbt project from cookiecutter templatedbt_gengenerate_dbt2. Generate sourcedbt_gengenerate_source--help

usage:dbt_gengenerate_source[-h][--profile-pathPROFILE_PATH][--profile-namePROFILE_NAME][--targetTARGET][--databaseDATABASE][--threadsTHREADS]source_folder

positionalarguments:source_folderFoldertowritesourceYAMLfiles

optionalarguments:-h,--helpshowthishelpmessageandexit--profile-pathPROFILE_PATHPathtodbtprofileYAML.Defaultis/home/tien/.dbt/profiles.yml--profile-namePROFILE_NAMEDbtprofilename.Defaultis`default`.--targetTARGETDbtprofiletarget.Defaultis`dev`.--databaseDATABASEDatabasetoinspect.Defaultisthedatabaseinprofile.--threadsTHREADSMaxthreads.Defaultisyourmachinenumberofthreads.3. Generate base models and testsdbt_gengenerate_base_model--help

usage:dbt-gengenerate_base_model[-h][--profile-pathPROFILE_PATH][--profile-namePROFILE_NAME][--targetTARGET][--threadsTHREADS][--templateTEMPLATE]source_pathoutput_folder

positionalarguments:source_pathPathtodbtsourceYAML.output_folderFoldertowritebasemodels.

optionalarguments:-h,--helpshowthishelpmessageandexit--profile-pathPROFILE_PATHPathtodbtprofileYAML.Defaultis/home/tien/.dbt/profiles.yml--profile-namePROFILE_NAMEDbtprofilename.Defaultis`default`.--targetTARGETDbtprofiletarget.Defaultis`dev`.--threadsTHREADSMaxthreads.Defaultisyourmachinenumberofthreads.--templateTEMPLATEPathtocustomJinjatemplate.3.1 Generate base models using custom Jinja templateGet default Jinja template by runningdbt-genget_template>your_custom_template.sqlEdit your custom templateRun generating base models using the custom templatedbt-gengenerate_base_models--templateyour_custom_template.sqlpath/to/sourcepath/to/output3.2. Jinja variables in rendering base modelssource_nameType: Stringtable_nameType: StringcolumnsType: Array of objectsItem:Type: objectAttributes:database: Stringschema: Stringtable: Stringname: Stringposition: Integernullable: Booleandtype: Stringextra: Dict4. Generate base testsdbt_gengenerate_base_test--help

usage:dbt-gengenerate_base_test[-h][--profile-pathPROFILE_PATH][--profile-namePROFILE_NAME][--targetTARGET]source_pathoutput_folder

positionalarguments:source_pathPathtodbtsourceYAML.output_folderFoldertowritebasemodels.

optionalarguments:-h,--helpshowthishelpmessageandexit--profile-pathPROFILE_PATHPathtodbtprofileYAML.Defaultis/home/tien/.dbt/profiles.yml--profile-namePROFILE_NAMEDbtprofilename.Defaultis`default`.--targetTARGETDbtprofiletarget.Defaultis`dev`.CaveatsFor BigQuery sources, dbt-gen supports making tests only based on Stitch meta table_sdc_primary_keys. If there is no such table, test YAML will be empty.For Snowflake, dbt-gen makes tests based on Snowflake table constraints.TODOsAdd primary keys information into variables in rendering base model SQLs
Package: dbt-graph-builder
dbt-graph-builderOverviewThis library provides a set of tools for building a graph of dbt models and their dependencies. It can be used to create an Airflow DAG for dbt models
or to create a graph of dbt models and their dependencies in a format that can be used by other tools like GCP Workflows.Project Organization.devcontainer - This directory contains required files for creating aCodespace..githubworkflows - Contains GitHub Actions used for building, testing and publishing.prepare-release.yml - Contains steps to prepare a release.publish.yml - Publish wheels tohttps://pypi.org/test-linting.yml - Build and Test pull requests before commiting to main.template-sync.yml - Update GitHub Repo with enhancments to base templatedocs - collect documents (default format .md)src - place new source code herepython_package - sample package (this can be deleted when creating a new repository)tests - contains Python based test cases to validation src code.pre-commit-config.yaml - Contains various pre-check fixes for PythonMANIFEST.in - Declares additional files to include in Python whlpyproject.toml - Python Project DeclarationREADME.md - This fileContributingThis project welcomes contributions and suggestions.  Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visithttps://cla.opensource.microsoft.com.When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.This project has adopted theMicrosoft Open Source Code of Conduct.
For more information see theCode of Conduct FAQor
contactopencode@microsoft.comwith any additional questions or comments.TrademarksThis project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
trademarks or logos is subject to and must followMicrosoft's Trademark & Brand Guidelines.
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Any use of third-party trademarks or logos are subject to those third-party's policies.
Package: dbt-hive
dbt-hiveThedbt-hiveadapter allows you to usedbtalong withApache HiveandCloudera Data PlatformGetting startedInstall dbtRead theintroductionandviewpointCreditsThe initial adapter code was developed by bachng2017 who agreed to transfer the ownership and continue active development.
This code base is now being actively developed and maintained by Cloudera.RequirementsCurrent version of dbt-hive use dbt-core 1.4.*. We are actively working on supporting the next version of dbt-core 1.5Python >= 3.8
dbt-core ~= 1.4.*
impyla >= 0.18Installpip3 install --user dbt-hiveSample profiledemo_project:
  target: dev
  outputs:
  dev:
    type: hive
    auth_type: LDAP
    user: [username]
    password: [password]
    schema: [schema]
    host: [hive-meta-store-host]
    port: 443
    http_path: [http-path]
    thread: 1Supported featuresNameSupportedIcebergMaterialization: ViewYesN/AMaterialization: TableYesYesMaterialization: Table with PartitionsYesYesMaterialization: Incremental - AppendYesYesMaterialization: Incremental - Append with PartitionsYesYesMaterialization: Incremental - Insert+OverwriteNoNoMaterialization: Incremental - Insert+Overwrite with PartitionsYesNoMaterialization: Incremental - MergeNoYesMaterialization: EphemeralNoNoSeedsYesYesTestsYesYesSnapshotsNoNoDocumentationYesNoAuthentication: LDAPYesYesAuthentication: KerberosYesYesIncrementalIncremental models are explained indbt documentation. This section covered the details about the incremental strategy supported by the dbt-hive.StrategyACID TableIceberg TableIncremental Full-RefreshYesYesIncremental AppendYesYesIncremental Append with PartitionsYesYesIncremental Insert OverwriteNoNoIncremental Insert Overwrite with PartitionsYesNoIncremental MergeNoYesIncremental Merge with PartitionsNoYesSupport forOn-Schema Changestrategy in dbt-hive:StrategyACID TableIceberg Tableignore (default)SupportedSupportedfailSupportedSupportedappend_new_columnsAdds new columnsAdds new columnssync_all_columnsAdds new columns and updates datatypes but doesn't remove existing columnsAdds new columns, updates datatypes and removes existing columnsTests CoverageFunctional TestsNameBaseIcebergMaterialization: ViewYesN/AMaterialization: TableYesYesMaterialization: Table with PartitionsYesYesMaterialization: Incremental - AppendYesYesMaterialization: Incremental - Append with PartitionsYesYesMaterialization: Incremental - Insert+OverwriteYesYesMaterialization: Incremental - Insert+Overwrite with PartitionsYesYesMaterialization: Incremental - MergeNoNoMaterialization: EphemeralNoNoSeedsYesYesTestsYesYesSnapshotsNoNoDocumentationYesNoAuthentication: LDAPYesYesAuthentication: KerberosYesYesNote: Kerberos is only qualified on Unix platform.
Package: dbThrottle
# dbThrottle
Python Class to run code with consideration to DB load.## Why
Throttling code to work under the constraints of a server makes sense in certain situations, especially when replica lag comes into play. This project hopes to simplify acting on this consideration.## What
A, still early, python module, which introduces a functional type to complete a task or continiously run an operation, taking database load and similar metrics into account.
The code will work with any metric you want to define, but we’ve included replia lag an open threads/connections as default metrics for mssql, mysql, postgresql, and oracle.## Who
Ryan Birmingham, under GPL3
Package: dbt-ibis
dbt-ibisWith dbt-ibis you can write yourdbtmodels usingIbis. You can find the full documentationhere.A simple dbt-ibis model looks like this:fromdbt_ibisimportdepends_on,ref@depends_on(ref("stg_stores"))defmodel(stores):returnstores.filter(stores["country"]=="USA")You can installdbt-ibisvia pip:pipinstalldbt-ibisIn addition, you'll need to install the relevantIbisbackendfor your database.You can read about the advantages of combining dbt and Ibis inthis blog post.Developmentpipinstall-e'.[dev]'You can run linters and tests withhatchrunlinters
hatchruntests