Saravanan666/pypi-org huggingface dataset:
Package: dbt-ibmdb2
dbt-ibmdb2This plugin portsdbtfunctionality to IBM DB2.This is an experimental plugin:We have not tested it extensivelyOnly basic tests are implementedCompatibility with otherdbt packages(likedbt_utils) is only partially testedPlease read these docs carefully and use at your own risk.Issueswelcome!Table of ContentsInstallationSupported featuresConfiguring your profileRunning TestsReporting bugsInstallationThis plugin can be installed via pip:$pipinstalldbt-ibmdb2Supported featuresDB2 LUWDB2 z/OSFeature✅🤷Table materialization✅🤷View materialization✅🤷Incremental materialization✅🤷Ephemeral materialization✅🤷Seeds✅🤷Sources✅🤷Custom data tests✅🤷Docs generate✅🤷SnapshotsNotes:dbt-ibmdb2 is built on the ibm_db python package and there are some known encoding issues related to z/OS.Configuring your profileA dbt profile can be configured to run against DB2 using the following configuration example:Example entry for profiles.yml:your_profile_name:
  target: dev
  outputs:
    dev:
      type: ibmdb2
      schema: analytics
      database: test
      host: localhost
      port: 50000
      protocol: TCPIP
      user: my_username
      password: my_password
      extra_connect_opts: my_extra_config_optionsOptionDescriptionRequired?ExampletypeThe specific adapter to useRequiredibmdb2schemaSpecify the schema (database) to build models intoRequiredanalyticsdatabaseSpecify the database you want to connect toRequiredtestdbhostHostname or IP-adressRequiredlocalhostportThe port to useOptional50000protocolProtocol to useOptionalTCPIPuserThe username to use to connect to the serverRequiredmy-usernamepasswordThe password to use for authenticating to the serverRequiredmy-passwordextra_connect_optsExtra connection optionsOptionalSecurity=SSL;SSLClientKeyStoreDB=<path-to-client-keystore>;SSLClientKeyStash=<path-to-client-keystash>Setup dev environment and run testsMake sure you have docker and poetry installed globally.make install
make test
make uninstallReporting bugsWant to report a bug or request a feature? Openan issue.Creditsdbt-ibmdb2 is heavily inspired by and borrows fromdbt-mysqlanddbt-oracle.
Package: db-timetable-api
DB Timetable APISimple access to the Deutsche Bahn timetable APIInstallationWindows:pipinstalldb_timetable_apiUnix:pip3installdb_timetable_apiUsage/ExamplesLoad package and enter API keyfromdb_timetable_apiimporttimetabledb=timetable.timetable_api(clientid=<YOUR_CLIENT_ID>,clientsecret=<YOUR_CLIENT_SECRET>)To get an API key you have to register athttps://developers.deutschebahn.com/, create an application and subscribe to the free Timetables API in the API catalogGet information about the station using the station namedb.get_station(<STATION_NAME>)### EXAMPLE ###db.get_station("Berlin Hbf")This public interface allows access to information about a station.Returns planned data for a specified station within an hourly time slicedb.get_timetable(<STATIONSEVAID>,<DATEYYYY-MM-DD>,<HOURHH>)### EXAMPLE ###db.get_timetable("8011160","2023-10-31","12")Returns a Timetable object (see Timetable) that contains planned data for the specified station (evaNo) within the hourly time slice given by date and hour. The data includes stops for all trips that arrive or depart within that slice. There is a small overlap between slices since some trips arrive in one slice and depart in another.Planned data does never contain messages. On event level, planned data contains the 'plannned' attributes pt, pp, ps and ppth while the 'changed' attributes ct, cp, cs and cpth are absent.Planned data is generated many hours in advance and is static, i.e. it does never change. It should be cached by web caches.public interface allows access to information about a station.Returns all known changes for a stationdb.get_changes(<STATIONSEVAID>)### EXAMPLE ###db.get_changes("8011160")Returns a Timetable object (see Timetable) that contains all known changes for the station given by evaNo.The data includes all known changes from now on until ndefinitely into the future. Once changes become obsolete (because their trip departs from the station) they are removed from this resource.Changes may include messages. On event level, they usually contain one or more of the 'changed' attributes ct, cp, cs or cpth. Changes may also include 'planned' attributes if there is no associated planned data for the change (e.g. an unplanned stop or trip).Returns all recent changes for a stationdb.get_recent_changes(<STATIONSEVAID>)### EXAMPLE ###db.get_recent_changes("8011160")Returns a Timetable object (see Timetable) that contains all recent changes for the station given by evaNo. Recent changes are always a subset of the full changes. They may equal full changes but are typically much smaller. Data includes only those changes that became known within the last 2 minutes.A client that updates its state in intervals of less than 2 minutes should load full changes initially and then proceed to periodically load only the recent changes in order to save bandwidth.
Package: dbt-impala
dbt-impalaThedbt-impalaadapter allows you to usedbtalong withApache ImpalaandCloudera Data PlatformGetting startedInstall dbtRead theintroductionandviewpointRequirementsCurrent version of dbt-impala work only with dbt-core 1.4 but not with dbt-core >= 1.5.
We are actively working on next release 1.5 which will work with dbt-core 1.5Python >= 3.7
dbt-core == 1.3.*For development/testing or contribution to the dbt-impala, please followContributingguidelines.Installing dbt-impalapip install dbt-impalaProfile Setupdemo_project:
  target: dev
  outputs:
    dev:
     type: impala
     host: impala-coordinator.my.org.com
     port: 443
     dbname: my_db
     schema: my_db
     user: my_user
     password: my_pass
     auth_type: ldap
     http_path: cliserviceSupported featuresNameSupportedIcebergMaterialization: ViewYesN/AMaterialization: TableYesYesMaterialization: Table with PartitionsYesYesMaterialization: Incremental - AppendYesYesMaterialization: Incremental - Append with PartitionsYesYesMaterialization: Incremental - Insert+OverwriteYesYesMaterialization: Incremental - Insert+Overwrite with PartitionYesYesMaterialization: Incremental - MergeNoNoMaterialization: EphemeralYesYesSeedsYesYesTestsYesYesSnapshotsNoNoDocumentationYesYesAuthentication: LDAPYesYesAuthentication: KerberosYesNoTests CoverageFunctional TestsNameBaseIcebergMaterialization: ViewYesN/AMaterialization: TableYesYesMaterialization: Table with PartitionsYesYesMaterialization: Incremental - AppendYesYesMaterialization: Incremental - Append with PartitionsYesYesMaterialization: Incremental - Insert+OverwriteYesYesMaterialization: Incremental - Insert+Overwrite with PartitionYesYesMaterialization: EphemeralYesYesSeedsYesYesTestsYesYesSnapshotsNoNoDocumentationYesYesAuthentication: LDAPYesYesAuthentication: KerberosNoNo
Package: dbt-infer
dbt-inferThedbtadapter forInfer.dbt-inferallows you to connect your dbt instance to Infer and useSQL-infsyntax in your dbt models.dbt-inferacts as a layer between your existing data warehouse and your dbt instance, enabling you to perform
ML analytics within your dbt models.dbt-inferpackages are hosted onPyPi.SQL-infSQL-inf is an extension of SQL that introduces machine learning primitives to SQL.
These primitives can be used within any part of your SQL queries, or DBT models, and allow you to build ML
analytics use cases.Read more about SQL-infhere.ExamplesIllustrative example based on an idealised tableuserswith some simple demographic data, whether the user has
churned or not, their lifetime value(LTV) and a text field with customer feedback.Predict columnhas_churnfrom the other columns in the tableusers.SELECT*FROMusersPREDICT(has_churned)Understand what columns drive values ofhas_churnedSELECT*FROMusersEXPLAIN(PREDICT(has_churned))Predict and understand the LTV in columnltvfrom tableusers.SELECT*FROMusersPREDICT(ltv)SELECT*FROMusersEXPLAIN(PREDICT(ltv))Perform text analysis, sentiment and topic analysis, on user feedbackSELECT*FROMusersSENTIMENT(feedback)SELECT*FROMusersTOPICS(feedback)Create user segmentations on demographicsSELECTage,location,gender,job,educationFROMusersCLUSTER()Find the sizes of the user segmentationsSELECTcluster_id,COUNT(*)assizeFROM(SELECTage,location,gender,job,educationFROMusersCLUSTER())GROUPBYcluster_idFind users similar to the user withuser_id=123SELECT*FROMusersSIMILAR_TO(user_id=123)Analyse what, if any, demographic features drive feedback sentimentSELECTage,location,gender,job,education,predictionFROM(SELECT*FROMusersSENTIMENT(feedback))EXPLAIN(PREDICT(prediction))InstallationDetailed installation and setup guidehere.RequirementsYou should be using dbt 1.2 or later.Setup Infer accountFirst you need to setup your Infer account and generate your api key.Read about how to do thathere.Installdbt-inferpipinstalldbt-inferSetting up dbt-inferSetup a target in your profile fordbt-inferwith the following shape<target_name>:url:<infer-api-endpoint>username:<infer-api-username>apikey:<infer-apikey>type:inferdata_config:<here goes your normal data warehouse config>wheredata_configcontains the profile settings for your underlying data warehouse.
Package: dbt-init
dbt-initA tool to create dbt projects for consulting.
dbt-init will create a project as a subdirectory within the target directory you
provide it, and populate as much of the dbt project as possibleInstallation & usageInstall usingpip install dbt-initTo create a new client project run a command like following:$dbt-init--clientjaffle_shop--warehousesnowflake--target_dir~/clients/You can also check the available arguments withdbt-init --helpOnce you've created your projectUpdatesample.profile.ymlto contain the correct profile details for your
client,excludingthe actual credentials – e.g. username and password:You can often pre-fill the host and database name.You may want to use an alternate connection method (e.g. OAuth) and update
the sample file to reflect this.Ensure that the users/groups/roles within a warehouse match the grant
statements in the post-run hooks (defined indbt_project.yml).Building out the starter projectIf you're interested in helping build out the starter project, here is a list of
variables you can use – a lot of them have defaults based on the client name.{{ project.name }}: The name of the project, as defined in `dbt_project.yml`, e.g. jaffle_shop.
{{ project.warehouse }}: The warehouse that a client is using.
{{ project.client_name }}: The name of the client, e.g. jaffle_shop.
{{ project.dir_name }}: The name of the directory this project is in, e.g. jaffle-shop-dbt.
{{ project.profile_name }}: The name of the profile used by this project, e.g. jaffle_shop.To-do:Configure a new profile as part of init process
Package: dbt-iomete
dbtenables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.dbt-iometeThedbt-iometepackage contains all the code enabling dbt to work with iomete.This adapter is forked from thedbt-sparkGetting startedInstallationpipinstalldbt-iometeAlternatively, you can install the package from GitHub with:pipinstallgit+https://github.com/iomete/dbt-iomete.gitProfile Setupiomete:target:devoutputs:dev:type:iometehost:<host>port:443https:true# or httplakehouse:<serverless_lakehouse_name>schema:<database_name>user:"{{env_var('DBT_IOMETE_USER_NAME')}}"token:"{{env_var('DBT_IOMETE_TOKEN')}}"For more information, consultthe docs.
Package: dbt-ipy
dbt-iPyRun dbt commands inside a IPython session.Installationpip install dbt-ipyTutorialYou can access thetutorial notebook hereQuickstart GuideLoad the extension%load_ext dbt-ipyConnect to dbt RPCThe next thing you'll probably want to do is to run a dbt RPC server in the background. That will let you run queries, compile SQL code among other things:%dbt rpc <args>(the args will be passed directly to dbt as command line arguments)The two most important parameters would be:--port [port_number]Passing this parameter will run the dbt RPC on that port. Default is 8580--existingThis will skip creating a new dbt RPC and instead it will connect to an existing one on the specified portCustom magic commandsCompiling queriesThe sql query in the cell will be compiled with the dbt RPC server and IPython will output the text::%%compile_sql
SELECT ...Running queriesThe sql query in the cell will be run on the dbt RPC server and IPython will output the agate table. Also, it will run its.print_table()method::%%run_sql
SELECT ...Copying to clipboardIt is also possible to copy the last result returned by dbt to the clipboard. If that's a%%compile_sqlcompiled query it will return the SQL text but if it's a%%run_sqltable, it will transform it to CSV and copy that to the clipboard. You can then paste the contents of the clipboard into a spreadsheet as usual.%dbt_clipboard
Package: dbt-junitxml
dbt-junitxmlConvert your dbt test results into jUnit XML format so that CI/CD platforms (such as Jenkins, CircleCI, etc.)
can better report on tests in their UI.Installationpipinstalldbt-junitxmlUsageWhen you run your dbt test suite, the output is saved undertarget/run_results.json. Run the following command
to parse your run results and output a jUnit XML formatted report namedreport.xml.dbt-junitxmlparsetarget/run_results.jsonreport.xmlLimitationsCurrently, only v4 of theRun Resultsspecifications is supported.
Package: dbt-lineage-viewer
DBT Dependency ViewerDBT Lineage Viewer is a command-line tool for visualizing the upstream and
downstream dependencies of a specified DBT model within a DBT project. It
generates an HTML file containing a diagram of the dependency tree.InstallationInstall the package using pip:pipinstalldbt-lineage-viewerUsageTo use the DBT Dependency Viewer, please navigate into your dbt project folder
and run the following command:dbt-lineage-viewer<model_name>--max-depth<max_depth>--output<output_file>Replace<model_name>with the name of the DBT model you want to analyze,<max_depth>with the maximum depth for the dependency tree, and<output_file>with the name of the output HTML file.Parametersmodel_name: The name of the DBT model to analyze (required).
max_depth: The maximum depth for the dependency tree (optional, default: 3).
output: The name of the output HTML file (optional, default: "test.html").ExampleAnalyze the tbl_acquisition_funnel model with a maximum depth of 3, and
generate an output HTML file named dependency_tree.html:dbt-lineage-viewertbl_acquisition_funnel--max-depth3--outputdependency_tree.html
Package: dbt-log-parser
dbt log parserPython package and CLI tool for parsing structured (JSON) data from adbtlog.Change Logv0.2.1🐛 Bug Fixes[bugfix]: import fails on <3.7 b/c of dataclasses dep; closes #18 (#19) @mdzhangv0.2.0🚀 New Features and Enhancements[feature]: parse string argument; closes #13 (#14) @mdzhang🔨 Maintenance[ci]: Add code coverage to CI checks; closes #10 (#11) @mdzhang[docs]: Add Sphinx docs; closes #8 (#9) @mdzhang[ci]: Add release pipeline using GitHub Actions; closes #6 (#7) @mdzhang[ci]: Add CI test pipeline; closes #2 (#3) @mdzhangv0.1.0Initial non-alpha release, now that project has improved tests and
documentationv0.1.0a0Initial release
Package: dbt-meshify
dbt-meshifyEXPERIMENTALmaintained with :heart: by dbt practitioners for dbt practitionersClick here for full package documentationOverviewdbt-meshifyis a CLI tool that automates the creation of model governance and cross-project lineage features introduced in dbt-core v1.5 and v1.6. This package will leverage your dbt project metadata to create and/or edit the files in your project to properly configure the models in your project with these features.These features include:Groups- group your models into logical sets.Contracts- add model contracts to your models to ensure consistent data shape.Access- control theaccesslevel of models within groupsVersions- create and increment versions of particular models.Project dependencies- split a monolithic dbt project into component projects, or connect multiple pre-existing dbt projects using cross-projectref.InstallationTo install dbt-meshify, run:pipinstalldbt-meshifyTo upgrade dbt-meshify, run:pipinstall--upgradedbt-meshify
Package: dbt-metricflow
Welcome to dbt-metricflowThis repo encapsulates the dbt-core, MetricFlow, and supported dbt-adapters packages. This package will manage the versioning between these packages such that they are compatible with each other.Repo use casesdbt-core and MetricFlow both depend on dbt-semantic-interfaces, which includes the schemas and interfaces for objects related to the semantic layer. Bundled versioning is necessary to ensure that the dbt-core and MetricFlow versions have compatible dbt-semantic-interfaces dependencies.Bundling the installation makes it much easier on end users, as they no longer need to installdbt-core+metricflow+dbt-adapterand reconcile versions - instead they can simply installdbt-metricflow[adapter].Because this encapsulates both dbt-core and MetricFlow, this repo can be used to build logic that should be shared across the packages. For example, the CLI from MetricFlow can live in this repo, as it uses logic from all of the bundled packages. This can streamline dependency requirements in MetricFlow.
Package: dbt-metrics-converter
The dbt converter is a CLI tool that simplifies the process of migrating from the metrics spec supported in dbt v1.5 and earlier to the metrics specification supported in dbt ≥v1.6.Installing dbt converterpipinstalldbt-metrics-converterUpgrading to dbt v1.6 specNavigate to the root of your dbt project.pip install dbt-metrics-converter. Note that the converter requires dbt-core==1.5. You may need to update the version range for the dbt_metrics package.Rundbtc convert-metrics. The converter assumes you are in the root of your dbt project. Optionally, you can pass the project path-dbt-project-path path/to/dbt_project. NOTE: All metrics must have a timestamp for the upgrade to run. If your metrics do not have a timestamp, add one before running the converter.Semantic models and metrics files will be created in thesemantic_modelsandmetricsdirectories. Move these folders into your model path.Delete themetricspackage, move the old metrics configs out of your model path, or delete them.Testing your metrics in dbt v1.6Upgrade to dbt v1.6, andpip install dbt-metricflowRemove the old metrics package from yourdbt_packagesand rundbt clean && dbt deps.Rundbt parseto generate a semantic manifestView the metrics in your dbt project by runningmf list metricsRunmf validate-configs —skip-dwto validate metrics configs.To run a test query, runmf query --metrics <metric_name>Gotchas:Some packages, like fivetran/ad_reporting have metrics defined in them using the v1.5 spec, this may cause an error if you try to run your project after upgrading to v1.6Make sure to delete any calls ofmetrics.calculateormetrics.developthey won’t work without the dbt_metrics package
Package: dbt-mindsdb
🚧 dbt-mindsdb 🚧The dbt-mindsdb package allows dbt to connect toMindsDB.InstallationAtm, can only be installed from source:git clone git@github.com:mindsdb/dbt-mindsdb.git
python setup.py developThere is an dependency issue with latest dbt-core, so make sure to run:pip install markupsafe==2.0.1ConfigurationsBasicprofile.ymlfor connecting to MindsDB:mindsdb:
  outputs:
    dev:
      database: 'mindsdb'
      host: '127.0.0.1'
      password: ''
      port: 47335
      schema: 'mindsdb'
      type: mindsdb
      username: 'mindsdb'
  target: devKeyRequiredDescriptionExampletype✔️The specific adapter to usemindsdbhost✔️The MindsDB (hostname) to connect tocloud.mindsdb.comport✔️The port to use3306or47335schema✔️Specify the schema (database) to build models intoThe MindsDB datasourceusername✔️The username to use to connect to the servermindsdbor mindsdb cloud userpassword✔️The password to use for authenticating to the server`passUsageCreate dbt project, choose mindsdb database and set up connectiondbt init <project_name>To create predictor add dbt model with "predictor" materialization:
Name of the model is used as name of predictor.
Parameters:integration - name of used integration to get data from and save result to.
In has to be created in mindsdb beforehandpredict - field for predictionpredict_alias [optional] - alias for predicted fieldusing [optional] - options for configure trained model{{
        config(
            materialized='predictor',
            integration='photorep',
            predict='name',
            predict_alias='name',
            using={
                'encoders.location.module': 'CategoricalAutoEncoder',
                'encoders.rental_price.module': 'NumericEncoder'
            }
        )
    }}
      select * from storesTo apply predictor add dbt model with "table" materialization.
It creates or replaces table in selected integration with results of predictor.
Name of the model is used as name of the table to store prediction results.
If you need to specify schema you can do it with dot separator: schema_name.table_name.sqlParameters:predictor_name - name of using predictor.
It has to be created in mindsdbintegration - name of used integration to get data from and save result to.
In has to be created in mindsdb beforehand{{ config(materialized='table', predictor_name='TEST_PREDICTOR_NAME', integration='int1') }}
        select a, bc from ddd where name > latestTestingInstall dev requirementspip install -r dev_requirements.txtRun pytestpython -m pytest tests/
Package: dbt-models-explorer
dbt-models-explorerDBT models explorer with output generator to check, analyze and documentFeaturesRead DBT Models Yml to objectcts and make relationships AnalysisRich Console OutputExport to CSV FormatRun LocallyClone the projectgitclonehttps://github.com/catapimbas/dbt-models-explorer.gitGo to the project directorycddbt-models-explorerInstall dependenciespoetryinstallRunpoetryrundbt-models-explorer[<path-to-yml-models><options>]Usage/ExamplesRich outupt, defaultpoetryrundbt-models-explorer<path-to-yml-models>CSV outuptpoetryrundbt-models-explorer<path-to-yml-models>--formatcsv<filename>