Saravanan666/pypi-org huggingface dataset:
Package: vdk-kerberos-auth
The plugin provides GSSAPI Kerberos authentication on data job startup. The plugin also adds Kerberos/GSSAPI support for HTTP requests.UsageTo install the plugin, run:pipinstallvdk-kerberos-authAfter it's install what happens:Upon installation and if KEYTAB_FILENAME and KEYTAB_PRINCIPAL are configured, it will try to authenticate ("kinit") at the start of very VDK command.Then when a client needs to talk to kerberos provision server they would use KerberosClient class to generate header:
With requests library, you'd usehttps://pypi.org/pypi/requests-kerberos.
The following can be used if another http library does not support kerberos to generate Authorization header:auth=KebrerosClient("http@server.fqdn.com")headers['Authorization']=auth.read_kerberos_auth_header()Known issuesThe plugin dependencyrequests-kerberos==0.12.0may fail to install on Ubuntu with the following error:src/kerberosbasic.h:17:10: fatal error: gssapi/gssapi.h: No such file or directory
     17 | #include <gssapi/gssapi.h>
        |          ^~~~~~~~~~~~~~~~~
  compilation terminated.If this is the case, installlibkrb5-devwith the command below and try reinstalling the plugin:sudoapt-getinstall-ylibkrb5-devConfigurationThe following environment variables can be used to configure this plugin:namedescriptionKRB_AUTHSpecifies the Kerberos authentication type to use. Possible values are 'minikerberos' and 'kinit'. If left empty, the authentication is disabled.KEYTAB_FILENAMESpecifies the name of the keytab file. If left empty, the name of the keytab file is assumed to be the same as the name of the data job with '.keytab' suffix.KEYTAB_FOLDERSpecifies the folder containing the keytab file. If left empty, the keytab file is expected to be located inside the data job folder.KEYTAB_PRINCIPALSpecifies the Kerberos principal. If left empty, the principal will be the job name prepended with 'pa__view_'.KEYTAB_REALMSpecifies the Kerberos realm. This value is used only with the 'minikerberos' authentication type. The default value is 'default_realm'.KERBEROS_KDC_HOSTSpecifies the name of the Kerberos KDC (Key Distribution Center) host. This value is used only with the 'minikerberos' authentication type.TestingIn order to run the tests you need pytest and docker and kerberos client (kadmin).You can use helper script../build-plugin.shto build and test locally.On Mac OS kadmin may miss some options needed. In this case you can use kadmin in docker to run the testsexportVDK_TEST_USE_KADMIN_DOCKER=truecd/source/projects/vdk-plugins/vdk-kerberos-auth
../build-plugin.sh
Package: vdk-lineage
VDK LineageVDK Lineage plugin provides lineage data (input data -> job -> output data) information and send it to a pre-configured
destination. The lineage data is send usingOpenLineage standardAt POC level currently.Currently, lineage data is collectedFor each data job run/execution both start and end events including the status of the job (failed/succeeded)For each execute query we collect input and output tables.TODOs:Collect status of the SQL query (failed, succeeded)Create parent /child relationship between sql event and job run event to track them better (single job can have multiple queries)Non-SQL lineage (ingest, load data,etc)Extend support for all queriesprovide more information using facets – op id, job version,figure out how to visualize parent/child relationships in MarquezExplore openlineage.sqlparser instead of sqllineage library as alternativeUsagepip install vdk-lineageAnd it will start collecting lineage from job and sql queries.To send data using openlineage specify VDK_OPENLINEAGE_URL. For example:export VDK_OPENLINEAGE_URL=http://localhost:5002
vdk marquez-server --start
vdk run some-job
# check UI for lineage
# stopping the server will delete any lineage data.
vdk marquez-server --stopBuild and testingIn order to build and test a plugin go to the plugin directory and use../build-plugin.shscript to build it
Package: vdk-lineage-model
VDK Lineage ModelVDK Lineage Model plugin aims to abstract emitting lineage data from VDK data jobs, so that different lineage loggers
can be configured at run time in any plugin that supports emitting lineage data.
The plugin describes the lineage data model and an interface for loggers that are responsible to send the lineage
information to the desired system.At POC level currently. Breaking changes should be expected.TODOs:Describe Non-SQL lineage (ingest, load data,etc)Extend support for all query types
Package: vdk-logging-json
This plugin is deprecated, please use vdk-logging-format instead.This is a POC level implementation of a plugin which changes the logging format of vdk-core to JSON for the purposes of structured data visualization in a log aggregator.The new format has the following fields:@timestamp: a timestamp of when the log is made;jobname: the name of the data job;level: the logging level - one of INFO, WARNING, DEBUG, ERROR;modulename: the name of the module, inside of which the logging call is made;filename: the name of the file containing the logging call being made;lineno: the number of the line of code, at which the logging call is made;funcname: the name of the function, inside which the loggin call is made;attemptid: string identifying this particular execution of the data job;message: any additional logged information.If the log record contains an error, it will also contain the following fields:error.message: the message of the exception;error.stack_trace: a stack trace of the exception;error.type: the type of exception.The label names follow the labelling recommendations found athttp://ltsv.org/.
The reason we chose the LTSV naming standard is due to the fact that this plugin was based on a previous LTSV-formatting plugin,
as well as the fact that there is no single JSON naming standard.Additionally, double quote characters within the message are escaped.UsageSwitching vdk logging can be done by simply installing the plugin:pipinstallvdk-logging-jsonAnd all logs will be automatically formatted to JSON. They will appear like this:{"@timestamp": "2021-10-14T11:37:44.703Z", "message": "Checking if job should be skipped:", "level": "INFO", "lineno": 102, "filename": "execution_skip.py", "modulename": "vdk.plugin.control_cli_plugin.execution_skip", "funcname": "_skip_job_if_necessary", "jobname": "gg-job4", "attemptid": "1634211464-1ee897-a3f0eb"}```
Package: vdk-logging-ltsv
This plugin is deprecated, please use vdk-logging-format instead.This is a POC level implementation of a plugin which changes the logging format of vdk-core to LTSV for the purposes of structured data visualization.The new format has the following fields, separated by tabs:timestamp: a timestamp of when the log is made;created: the same timestamp in the unix epoch format;jobname: the name of the data job;level: the logging level - one of INFO, WARNING, DEBUG, ERROR;modulename: the name of the module, inside of which the logging call is made;filename: the name of the file containing the logging call being made;lineno: the number of the line of code, at which the logging call is made;funcname: the name of the function, inside which the loggin call is made;attemptid: string identifying this particular execution of the data job;message: any additional logged information.The label names follow the labelling recommendations found athttp://ltsv.org/.UsageSwitching vdk logging can be done by simply installing the plugin:pipinstallvdk-logging-ltsvAnd all logs will be automatically formatted to LTSV format. They would start to look something like this:@timestamp:2021-08-04T12:51:11	created:1628070671	jobname:example-job	level:DEBUG	modulename:vdk.internal.trino_connection	filename:managed_connection_base.py	lineno:69	funcname:connect	attemptid:1628070671-452613-739749	message:Established <trino.dbapi.Connection object at 0x10b9b1d30>
@timestamp:2021-08-04T12:51:11	created:1628070671	jobname:example-job	level:DEBUG	modulename:vdk.internal.trino_connection	filename:managed_cursor.py	lineno:29	funcname:execute	attemptid:1628070671-452613-739749	message:Executing query: select 1
Package: vdk-notebook
vdk-notebookA new VDK plugin which supports running data jobs which consists of .ipynb files.
You can seeVDK Jupyter Integration VEPfor more information.UsageTo install the plugin you need to run:pip install vdk-notebookConfigurationNo specific notebook configurations are needed currently.ExampleHere is a directory structure which consist of one parent directory "parent-dir" and two subdirectories.
One of the subdirectories is a data job("example-job"). This is how the structure looks like:parent-dir
└── example-job
    ├── notebook.ipynb
    ├── requirements.txt
    └── config.ini
└── some-other-dirTo run the data job from the parent-dir we need to use:vdk run example-jobEach cell which has tag "vdk" is executedThere could be those types of VDK Cells:VDK Python Cell. Those are normal Python cells tagged with "vdk"VDK SQL Cell. Those are cells marked with%%vdksqlmagic and tagged with "vdk"VDK Ingest Cell. Those are celles marked with%%vdkingestmagic.For more information and examples seevdk-ipythondocumentationTestingTesting this plugin locally requires installing the dependencies listed in vdk-plugins/vdk-notebook/requirements.txtRunpip install -r requirements.txtArchitectureSee the architecture of the plugin in the "Detailed design" section ofthe VEP.
Package: vdk-plugin-control-cli
This plugin allows vdk-core to access vdk-control-cli functionality.Upon installing it enhances vdk with CLI commands for data job lifecycle management available from vdk-control-cli packageIt will also install properties plugin and switch Properties in vdk to use Control Service Properties API.
By default no Properties backend is used otherwise.Usagepipinstallvdk-plugin-control-cliNew CLI commandsThen on the vdk CLI you should be able to see the new commands. Runvdk--helpand you will see extra commands being added:Commands:
  ...
  login                  Authentication against the Control Service.
  logout                 Logout the user from the Data Jobs Service by...
  create                 Creates a new data job in cloud and locally.
  delete                 Deletes a data job from the cloud.
  deploy                 Deploys a data job.
  ...And you can use them from the same CLIvdkNew properties backend client registeredYou can now access remote properties from Control Service Properties API
Store state, configuration or secrets there using CLI or Python JobInput APIFor example, let's store some api-urivdkproperties--set'api-uri''http://cool.cool.api.com'and then we can use it in our data jobdefrun(job_input:IJobInput):uri=job_input.get_property('api-uri')print(requests.get(uri))BuildThe easiest way is to use the ../build-plugin.sh helper script../build-plugin.sh
Package: vdk-plugin-name
plugin-nameSimple description of my project.TODO: what the project is about, what is its purposeUsagepip install vdk-plugin-nameConfiguration(vdk config-helpis useful command to browse all config options of your installation of vdk)NameDescription(example)  Valuedummy_config_keyDummy configuration"Dummy"ExampleTODOBuild and testingpip install -r requirements.txt
pip install -e .
pytestIn VDK repo../build-plugin.shscript can be used also.Note about the CICD:.plugin-ci.yaml is needed only for plugins part ofVersatile Data Kit Plugin repo.The CI/CD is separated in two stages, a build stage and a release stage.
The build stage is made up of a few jobs, all which inherit from the same
job configuration and only differ in the Python version they use (3.7, 3.8, 3.9 and 3.10).
They run according to rules, which are ordered in a way such that changes to a
plugin's directory trigger the plugin CI, but changes to a different plugin does not.
Package: vdk-poc-anonymize
InstallationTo install requirements:python -m pip install requirements.txtTo save requirements:python -m pip list --format=freeze --exclude-editable -f https://download.pytorch.org/whl/torch_stable.html > requirements.txtNote we use Python 3.9.4 for our experimentsThe code in augerino_lib is an extension (with our modification) from the original Augerino codehttps://github.com/g-benton/learning-invariances. The code in data_augmentation/my_training.py is a modification of the pytorch example inhttps://github.com/pytorch/examples/tree/master/imagenet.Running the codeFor remaining experiments:Navigate to the corresponding directory, then execute:python run.py -mwith the correspondingconfig.yamlfile (which stores experiment configs).LicenseConsult License.md
Package: vdk-postgres
This plugin allows vdk-core to interface with and execute queries against a PostgreSQL database.UsageRunpipinstallvdk-postgresAfter this, data jobs will have access to a Postgres database connection, managed by Versatile Data Kit SDK.If it is the only database plugin installed , vdk would automatically use it.
Otherwise, users need to set VDK_DB_DEFAULT_TYPE=POSTGRES as an environment variable or set 'db_default_type' option in the data job config file (config.ini).For exampledefrun(job_input:IJobInput):job_input.execute_query("select 'Hi Postgres!'")ConfigurationRun vdk config-help - search for those prefixed with "POSTGRES_" to see what configuration options are available.TestingTesting this plugin locally requires installing the dependencies listed in vdk-plugins/vdk-postgres/requirements.txtRunpipinstall-rrequirements.txt
Package: vdk-properties-fs
This plugin allows vdk-core to read/write properties on the local FS. Mainly for development purposes,
to simplify a use-case with local Properties API usage, that would otherwise require a Control Service instance prerequisite.
For example, to quickly demo a data job that uses a secret, so that secret is quickly stored on the presenter's local FS.UsageRunpipinstallvdk-properties-fsAfter this, data jobs will be able to read and write properties stored on the local file system.For exampledefrun(job_input:IJobInput):my_props={"key":"value"}job_input.set_all_properties(my_props)assertjob_input.get_all_properties()==my_propsassertjob_input.get_property("key")=="value"ConfigurationTo enable the plugin, set "PROPERTIES_DEFAULT_TYPE" to "fs-properties-client".
Run vdk config-help - search for those prefixed with "FS_PROPERTIES_" to see what configuration options are available.TestingTesting this plugin locally requires installing the dependencies listed in vdk-plugins/vdk-properties-fs/requirements.txtRunpipinstall-rrequirements.txt
Package: vdk-singer
singerThe vdk-singer plugin provides an easy way to integrate Singer Taps as data sources within the Versatile Data Kit (VDK).
This allows you to pull data from various external systems that have Singer Taps available and use them seamlessly
within your VDK pipelines.Usagepip install vdk-singerConfiguration(vdk config-helpis useful command to browse all config options of your installation of vdk)You can configure the Singer data source via the SingerDataSourceConfiguration class. The configuration options include:tap_name: The name of the Singer Tap you are using.tap_config: A dictionary containing configuration specific to the Singer Tap.tap_auto_discover_schema: A boolean to indicate whether to auto-discover the schema.config=SingerDataSourceConfiguration(tap_name="tap-gitlab",tap_config={"api_url":"https://gitlab.com/api/v4","private_token":"your_token_here",# ... other tap specific configurations},tap_auto_discover_schema=True,ExampleThis will change as we will introduce more user frinedly way of defining sources but currently it looks like this:fromvdk.api.job_inputimportIJobInputfromvdk.internal.builtin_plugins.ingestion.source.factoryimportSingletonDataSourceFactoryfromvdk.plugin.singer.singer_data_sourceimportSingerDataSourceConfigurationdefrun(job_input:IJobInput):data_source=SingletonDataSourceFactory().create_data_source("singer-tap")config=SingerDataSourceConfiguration(tap_name="tap-gitlab",tap_config={"api_url":"https://gitlab.com/api/v4","private_token":"your_token_here",# ... other configurations},tap_auto_discover_schema=True,)data_source.connect(config,None)# ... rest of the job logicList all likely available tapsvdksinger--list-tapsBuild and testingpip install -r requirements.txt
pip install -e .
pytestIn VDK repo../build-plugin.shscript can be used also.Note about the CICD:.plugin-ci.yaml is needed only for plugins part ofVersatile Data Kit Plugin repo.The CI/CD is separated in two stages, a build stage and a release stage.
The build stage is made up of a few jobs, all which inherit from the same
job configuration and only differ in the Python version they use (3.7, 3.8, 3.9 and 3.10).
They run according to rules, which are ordered in a way such that changes to a
plugin's directory trigger the plugin CI, but changes to a different plugin does not.
Package: vdk-smarter
VDK SmarterMaking VDK smarter by employing ML/AI.Usagepip install vdk-smarterConfiguration(vdk config-helpis useful command to browse all config options of your installation of vdk)ExampleTODO# VDK SmarterMaking VDK smarter by employing ML/AI.Usagepip install vdk-smarterConfiguration(vdk config-helpand search for configuration starting with "openai")ExampleBy default reviews are disabled since they are expensive.To enable you need to setopenai_review_enabledtotruein the configuration andopenai_api_key.
You can seevdk config-help(search for configuration with openai prefix) for more information.Once enabled on vdk run each query statement executed will be also reviewed and scored.vdk run exampleQuery:
CREATETABLEIFNOTEXISTSsuper_collider.example_table(vc_idSTRING,esx_idSTRING,vm_countINT)STOREDASPARQUET;Review:{"score":5,"review":"No further changes needed. The query is efficient, readable, and follows best practices.There are no potential errors, optimization, or security vulnerabilities."}...
Query:
CREATETABLEIFNOTEXISTSsuper_collider.example_fact_snapshot(vc_idSTRING,esx_countBIGINT,vm_countBIGINT)STOREDASPARQUET;Review:{"score":4,"review":"The query is well-structured and follows best practices.However, there is an opportunity to improve its readability byadding comments to explain the purpose of the query  and the meaning of the parameters.Additionally, there is potential to optimize the query by providing more precise column types."}At the end a report is generatedqueries_reviews_report.mdwith all the queries and their reviews.Build and testingpip install -r requirements.txt
pip install -e .
pytestIn VDK repo../build-plugin.shscript can be used also.Note about the CICD:.plugin-ci.yaml is needed only for plugins part ofVersatile Data Kit Plugin repo.The CI/CD is separated in two stages, a build stage and a release stage.
The build stage is made up of a few jobs, all which inherit from the same
job configuration and only differ in the Python version they use (3.7, 3.8, 3.9 and 3.10).
They run according to rules, which are ordered in a way such that changes to a
plugin's directory trigger the plugin CI, but changes to a different plugin does not.
Package: vdk-snowflake
Versatile Data Kit Plugin for Snowflake SupportThis plugin provides functionality, used by the Versatile Data Kit
to interact with a Snowflake instance. Users of the plugin can connect to
Snowflake, and execute queries against it.InstalationTo install the plugin, open a terminal and typepipinstall--upgradevdk-snowflakeUsageIn order to use the plugin, once it has been installed, there are a few environment variables that need to be set.The following environment variables are required:\\SetdefaultdatabaseplugintobeusedVDK_DB_DEFAULT_TYPE=SNOWFLAKE\\Settheusernamethatistobeused.VDK_SNOWFLAKE_USER=<some_username>\\Setthepasswordthatistobeused.VDK_SNOWFLAKE_PASSWORD=<user_password>\\Settheaccountnamethatistobeused.\\NOTE:DoNOTincludethe'.snoflakecomputing.com'partoftheaccountname.VDK_SNOWFLAKE_ACCOUNT=<account_name_provided_by_snowflake>The following environment variables are optional, and can be overwritten from within the data job.\\Thedefaultwarehousetobeused.VDK_SNOWFLAKE_WAREHOUSE=<default_warehouse>\\Thedefaultdatabasetobeused.VDK_SNOWFLAKE_DATABASE=<default_database>\\Thedefaultdatabaseschematobeused.VDK_SNOWFLAKE_SCHEMA=<default_schema>After configuring the environment variables, to use the plugin, create a data job, and add the queries you want to execute in.sqlfiles. When the job executed, Versatile Data Kit will execute the queries in the order specified in theUser Guide.
Package: vdk-storage
storageLibrary for access to different managed storagesUsagepip install vdk-storageConfiguration(vdk config-helpis useful command to browse all config options of your installation of vdk)NameDescription(example)  Valuedummy_config_keyDummy configuration"Dummy"ExampleTODOBuild and testingpip install -r requirements.txt
pip install -e .
pytestIn VDK repo../build-plugin.shscript can be used also.Note about the CICD:.plugin-ci.yaml is needed only for plugins part ofVersatile Data Kit Plugin repo.The CI/CD is separated in two stages, a build stage and a release stage.
The build stage is made up of a few jobs, all which inherit from the same
job configuration and only differ in the Python version they use (3.7, 3.8, 3.9 and 3.10).
They run according to rules, which are ordered in a way such that changes to a
plugin's directory trigger the plugin CI, but changes to a different plugin does not.
Package: vdk-test-utils
This plugin provides utility tools used for the testing of vdk-core and vdk-core plugins.UsageCliEntryBasedTestRunnerCliEntryBasedTestRunner provides an easy way to create unit or functional tests for your plugins in isolated environment.
Similarly this can be used to unit tests data jobs.For example , let's say we want to test newly developed AtlasLineagePlugindb_plugin=AtlasLineagePlugin()# we pass the plugin to the runner that will make sure to register itrunner=CliEntryBasedTestRunner(db_plugin)# and then invoke the command we want to test -# in this case data job runresult:Result=runner.invoke(["run",util_funcs.jobs_path_from_caller_directory("simple-create-insert")])cli_assert_equal(0,result)assert'expected_schema'inresult.outputTesting pluginsSome useful plugins that can be used during testing can be seen in util_plugins.
Package: vdk-tino
tinoSimple description of my project.TODO: what the project is about, what is its purposeUsagepip install vdk-tinoConfiguration(vdk config-helpis useful command to browse all config options of your installation of vdk)NameDescription(example)  Valuedummy_config_keyDummy configuration"Dummy"ExampleTODOBuild and testingpip install -r requirements.txt
pip install -e .
pytestIn VDK repo../build-plugin.shscript can be used also.Note about the CICD:.plugin-ci.yaml is needed only for plugins part ofVersatile Data Kit Plugin repo.The CI/CD is separated in two stages, a build stage and a release stage.
The build stage is made up of a few jobs, all which inherit from the same
job configuration and only differ in the Python version they use (3.7, 3.8, 3.9 and 3.10).
They run according to rules, which are ordered in a way such that changes to a
plugin's directory trigger the plugin CI, but changes to a different plugin does not.