Saravanan666/pypi-org huggingface dataset:
Package: vdirsyncer
DocumentationSource codeVdirsyncer is a command-line tool for synchronizing calendars and addressbooks
between a variety of servers and the local filesystem. The most popular usecase
is to synchronize a server with a local folder and use a set of otherprogramsto change the local events and contacts. Vdirsyncer can then synchronize those
changes back to the server.However, vdirsyncer is not limited to synchronizing between clients and
servers. It can also be used to synchronize calendars and/or addressbooks
between two servers directly.It aims to be for calendars and contacts whatOfflineIMAPis for emails.Links of interestCheck outthe tutorialfor basic
usage.Contact informationHow to contribute to this projectDonationsLicenseLicensed under the 3-clause BSD license, seeLICENSE.
Package: vdisk
vdisk is a tool to create a debian system suitable for use by a
virtualization technology such as kvm/qemu. The output image is
is a sparse file holding a a partition table, the grub boot loader
and a filesystem containing the base system.Installationvdisk is designed to run in Debian GNU/Linux and the standard
configuration builds a Debian Squeeze image.vdisk depends on a few packages being installed on the system where
images are built. This can be accomplished by running the following
command:apt-get install kpartx parted lvm2 python-argparseFunctionalityvdisk builds sparse images and mounts them to loopback devices for convenient
access.This allows for easy manipulation of these images inside of a chroot,
effectively letting allow for setup of a virtual base system without having to
run a virtual machine.UsageAll of the following commands must be run as root on a linux host.vdisk expects a configuration file in the current working directory called
vdisk.yaml, an example should be distributed with this package.Build a fresh image and bootstrap a base system to it.bin/vdisk foo.img create
bin/vdisk foo.img bootstrapInstall required packages, and prepare image for booting.bin/vdisk foo.img install [selections]Try it out.bin/vdisk foo.img enterWhen you have a successfully installed a system, try it out using a virtual
machine or an emulator like qemu.qemu-system-x86_64 -hda foo.img -monitor stdioEC2You must include /boot/grub/menu.lst  it in your vdisk.yaml. Simple example:default 0
timeout 3
title EC2
root (hd0,0)
kernel /vmlinuz root=/dev/mapper/VolFoo-root ro
initrd  /initrd.imgDepending on your kernel you might also need to create
/etc/initramfs-tools/modules with xenblk and xennet modules, vdisk will execute
update-initramfs as the final step.Important Filesvdisk.yaml - Contains configuration for how to create your virtual disk.License & CreditsThis software is released under the GNU GPL, see COPYING for details.It is authored by John-John Tedro <udoprog@spotify.com>,
Noa Resare <noa@spotify.com> and Lars-Erik Stenholm <lasse@spotify.com>.Copyright is held by Spotify AB.
Package: vdist-solver
vdist-solverVelocity distribution function solver using Liouville's theoremvx-vz distributionvz distributionbacktraceRequirementnumpymatplotlibtqdmscipyemoutmpi4py (Install by yourself if use MPI)Installationpip install vdist-solverSimple Usage for EMSES# Use velocity distribution function solver
> gen-vdsolver 'xz'  # Create template script of vdist-solver
> vim vdist-solver.py  # Change settings
> python vdist-solver.py -d "<directory>"  # Plot distribution

# Use backtrace solver
> gen-backtrace 'xz'  # Create template script of backtrace-solver
> vim backtrace-solver.py  # Change settings
> python backtrace-solver.py -d "<directory>"  # Plot backtrace-orbit of particleA detailed description is provided belowGet Started (Japanese).Theory (Japanese)
Package: vdk-audit
Versatile Data Kit Audit PluginVisibility into the actions provides opportunities for test frameworks, logging
frameworks, and security tools to monitor and optionally limit actions taken by the
runtime.
This plugin provides an ability to audit and potentially limit user operations.
These operations are typically deep within the Python runtime or standard library, such
as dynamic code compilation, module imports or OS command invocations. In order to have a
better understanding of what exactly the job does, we will log all job operations.UsageTo use the plugin, just install it usingpipinstallvdk-audit
Package: vdk-control-api-auth
vdk-control-api-authvdk-control-api-auth is a plugin library that implements authentication
utilities used by vdk-control-cli and other plugins.UsageThis is a library plugin, not a runnable plugin, and it is intended to be
used as a dependency for other plugins, which need to authenticate users.To use the library within a plugin or another Versatile Data Kit component,
just import theAuthenticationclass, and create an instance of it. The
different authentication flows require different parameters to be specified.Once everything is done, in order to authenticate, call.authenticate()on
theAuthenticationinstance.Example Usage:fromvdk.plugin.control_api_auth.authenticationimportAuthenticationauth=Authentication(token="<oauth-api-token>",authorization_url="https://some-authorization-endpoint",auth_type="api-token",)auth.authenticate()# authenticateauth.read_access_token()# fetch the cached access tokenBuild and testingIn order to build and test a plugin go to the plugin directory and use../build-plugin.shscript to build it
Package: vdk-control-cli-NAME
control-cli-NAMESimple description of my project.TODO: what the project is about, what is its purposeUsagepip install vdk-control-cli-NAMEConfiguration(vdk config-helpis useful command to browse all config options of your installation of vdk)NameDescription(example)  Valuedummy_config_keyDummy configuration"Dummy"ExampleTODOBuild and testingpip install -r requirements.txt
pip install -e .
pytestIn VDK repo../build-plugin.shscript can be used also.Note about the CICD:.plugin-ci.yaml is needed only for plugins part ofVersatile Data Kit Plugin repo.The CI/CD is separated in two stages, a build stage and a release stage.
The build stage is made up of a few jobs, all which inherit from the same
job configuration and only differ in the Python version they use (3.7, 3.8, 3.9 and 3.10).
They run according to rules, which are ordered in a way such that changes to a
plugin's directory trigger the plugin CI, but changes to a different plugin does not.
Package: vdk-control-service-api
The Data Jobs API of Versatile Data Kit Control Service. Data Jobs allows Data Engineers to implement automated pull ingestion (E in ELT) and batch data transformation into a database (T in ELT). See also https://github.com/vmware/versatile-data-kit/wiki/Introduction The API has resource-oriented URLs, JSON-encoded responses, and uses standard HTTP response codes, authentication, and verbs. The API enables creating, deploying, managing and executing Data Jobs in the runtime environment.&lt;br&gt; &lt;br&gt; ![](https://github.com/vmware/versatile-data-kit/wiki/vdk-data-job-lifecycle-state-diagram.png) &lt;br&gt; The API reflects the usual Data Job Development lifecycle:&lt;br&gt; &lt;li&gt; Create a new data job (webhook to further configure the job, e.g authorize its creation, setup permissions, etc). &lt;li&gt; Download keytab. Develop and run the data job locally. &lt;li&gt; Deploy the data job in cloud runtime environment to run on a scheduled basis. &lt;br&gt;&lt;br&gt; If Authentication is enabled, pass OAuth2 access token in HTTP header &#39;Authorization: Bearer [access-token-here]&#39; (https://datatracker.ietf.org/doc/html/rfc6750). &lt;br The API promotes some best practices (inspired by https://12factor.net): &lt;li&gt; Explicitly declare and isolate dependencies. &lt;li&gt; Strict separation of configurations from code. Configurations vary substantially across deploys, code does not. &lt;li&gt; Separation between the build, release/deploy, and run stages. &lt;li&gt; Data Jobs are stateless and share-nothing processes. Any data that needs to be persisted must be stored in a stateful backing service (e.g IProperties). &lt;li&gt; Implementation is assumed to be atomic and idempotent - should be OK for a job to fail somewhere in the middle; subsequent restart should not cause data corruption. &lt;li&gt; Keep development, staging, and production as similar as possible. &lt;br&gt;&lt;br&gt; &lt;b&gt;API Evolution&lt;/b&gt;&lt;br&gt; In the following sections, there are some terms that have a special meaning in the context of the APIs. &lt;br&gt;&lt;br&gt; &lt;li&gt; &lt;i&gt;Stable&lt;/i&gt; - The implementation of the API has been battle-tested (has been in production for some time). The API is a subject to semantic versioning model and will follow deprecation policy. &lt;li&gt; &lt;i&gt;Experimental&lt;/i&gt; - May disappear without notice and is not a subject to semantic versioning. Implementation of the API is not considered stable nor well tested. Generally this is given to clients to experiment within testing environment. Must not be used in production. &lt;li&gt; &lt;i&gt;Deprecated&lt;/i&gt; - API is expected to be removed within next one or two major version upgrade. The deprecation notice/comment will say when the API will be removed and what alternatives should be used instead.  # noqa: E501
Package: vdk-core
Versatile Data Kit SDK CoreTo build or contribute, seeCONTRIBUTING.md.Versatile Data Kit enables data engineers, data scientists and data analysts to develop, deploy, run, and manage data processing workloads (called "Data Job").
A "Data Job" enables Data Engineers to implement automated pull ingestion (E in ELT) and batch data transformation into a database (T in ELT).BuildMake sure you have python3 installed. It's strongly recommended to use a virtual environment, and also to use a python installer like pyenv or conda. We recommend pyenv.CICD is managed through gitlab-ci.yml. You can see how to build, run tests, deploy there best.
All related CICD scripts are in /cicd/ folder.InstallationAll Versatile Data Kit SDK Core releases can be found in PyPI.Install Versatile Data Kit SDK Core with:pipinstallvdk-coreThis will install console application calledvdkThen run help to see what it can do:vdk--helpPluginsPlugins are a powerful way of extending or adapting Versatile Data Kit for all kinds of use-cases.
For more information seePlugins doc.Public interfacesAny backwards compatibility guarantees apply only to public interfaces.
Public interfaces are modules and packages defined or imported in vdk.api.*.
unless the documentation explicitly declares them to be provisional or internal interfaces.
Anything else is considered internal.
All public interfaces (classes or methods) must have documentation.
Package: vdk-csv
Versatile Data Kit CSV PluginThis plugin provides functionality to ingest and export CSV files.UsageTo use the plugin, just install it usingpipinstallvdk-csvThen run help to see what you can do.vdkingest-csv--helpandvdkexport-csv--helpFor example you can ingest CSV file into tables, or export the result of an executed query to a cvs file.The ingestion destination depends on how vdk has been configured.
See vdk config-help - search for "ingest" to check for possible ingestion configurations.
Package: vdk-data-source-git
data-source-gitExtracts content from Git repositories along with associated file metadata.Usagepip install vdk-data-source-gitExtracted Data SchemaThe extracted data is returned in aDataSourcePayloadobject with two main components:contentandmetadata.contentThecontentfield contains the actual content of the file as a string.metadataThemetadatafield contains a dictionary with the following schema:KeyDescriptionData TypeExamplesizeThe size of the file in bytesInteger12345pathThe path of the file in the repositoryString"src/main.py"num_linesThe number of lines in the fileInteger678file_extensionThe file extensionString".py"programming_languageThe detected programming language of the fileString"Python"is_likely_test_fileFlag indicating if the file is likely a test fileBooleanfalseConfiguration(vdk config-helpis useful command to browse all config options of your installation of vdk)NameDescription(example)  Valuegit_urlURL of the Git repository to be cloned."https://github.com/user/repo"Build and testingpip install -r requirements.txt
pip install -e .
pytestIn VDK repo../build-plugin.shscript can be used also.Note about the CICD:.plugin-ci.yaml is needed only for plugins part ofVersatile Data Kit Plugin repo.The CI/CD is separated in two stages, a build stage and a release stage.
The build stage is made up of a few jobs, all which inherit from the same
job configuration and only differ in the Python version they use (3.7, 3.8, 3.9 and 3.10).
They run according to rules, which are ordered in a way such that changes to a
plugin's directory trigger the plugin CI, but changes to a different plugin does not.
Package: vdk-duckdb
duckdbDuckDB plugin for the Versatile Data Kit (VDK), which enables users to connect to and interact with DuckDB databases.
The purpose is to simplify data extraction, transformation, and loading tasks when working with DuckDB as a data source or destinationUsagepip install vdk-duckdbConfigurationRunvdk config-helpto browse all available configuration options for your VDK installation.ExampleQuery ExecutionYou specify you want to use duckdb in the job config file
config.ini[vdk]
db_default_type = duckdbThen you can use itdef run(job_input: IJobInput):
        job_input.execute_query("select 'Hi Duck!'")IngestionThis plugin allows users to ingest data to a DuckDB database, which can be preferable to inserting data manually as it automatically handles serializing, packaging and sending of the data asynchronously with configurable batching and throughput.
To do so, you must set the expected variables to connect to Greenplum, plus the following environment variable:export VDK_INGEST_METHOD_DEFAULT=DUCKDBThen, from inside the run function in a Python step, you can use thesend_object_for_ingestionorsend_tabular_data_for_ingestionmethods to ingest your data.Build and testingpip install -r requirements.txt
pip install -e .
pytestIn VDK repo../build-plugin.shscript can be used also.Note about the CICD:.plugin-ci.yaml is needed only for plugins part ofVersatile Data Kit Plugin repo.The CI/CD is separated in two stages, a build stage and a release stage.
The build stage is made up of a few jobs, all which inherit from the same
job configuration and only differ in the Python version they use (3.7, 3.8, 3.9 and 3.10).
They run according to rules, which are ordered in a way such that changes to a
plugin's directory trigger the plugin CI, but changes to a different plugin does not.
Package: vdk-gdp-execution-id
An installed Generative Data Pack plugin automatically expands the data sent for ingestion.This GDP plugin detects the execution ID of a Data Job running, and decorates your data product with it. So that,
it is now possible to correlate a data record with a particular ingestion Data Job execution ID.Each ingested dataset gets automatically expanded with a Data Job execution ID micro-dimension.For example:{"product_name":"name1","product_description":"description1"}After installingvdk-gdp-execution-id, one additional field gets automatically appended to your payloads that are
sent for ingestion:{"product_name":"name1","product_description":"description1","gdp_execution_id":"product-ingestion-data-job-1628151700498"}The newly-added dimension name is configurable.UsageRunpipinstallvdk-gdp-execution-idCreate a Data Job and add to itsrequirements.txtfile:# Python jobs can specify extra library dependencies in requirements.txt file.
# See https://pip.readthedocs.io/en/stable/user_guide/#requirements-files
# The file is optional and can be deleted if no extra library dependencies are necessary.
vdk-gdp-execution-idReconfigure the ingestion pre-processing sequence to add the plugin name. For example:exportVDK_INGEST_PAYLOAD_PREPROCESS_SEQUENCE="vdk-gdp-execution-id"# orexportVDK_INGEST_PAYLOAD_PREPROCESS_SEQUENCE="[...,]vdk-gdp-execution-id"Note: The recommendation is to add this plugin last (at end-of-sequence), due prior plugins may add new data records.
For more info on configurations, seeprojects/vdk-core/src/vdk/internal/core/config.py.Example ingestion Data Job10_python_step.py:defrun(job_input:IJobInput):# objectjob_input.send_object_for_ingestion(payload={"product_name":"name1","product_description":"description1"},destination_table="product")# tabular datajob_input.send_tabular_data_for_ingestion(rows=[["name2","description2"],["name3","description3"]],column_names=["product_name","product_description"],destination_table="product")In case theVDK_INGEST_METHOD_DEFAULTwas a relational database,
then you can query the dataset and filter:# A processing Data Job then filters the ingested dataset by `vdk_gdp_execution_id` columndefrun(job_input:IJobInput):execution_ids=job_input.execute_query("SELECT DISTINCT vdk_gdp_execution_id FROM product")print(execution_ids)ConfigurationRun vdk config-help - search for those prefixed with "GDP_EXECUTION_ID_" to see what configuration options are
available.TestingTesting this plugin locally requires installing the dependencies listed in
vdk-plugins/vdk-gdp-execution-id/requirements.txtRunpipinstall-rrequirements.txtExampleFind an example data job usingvdk-gdp-execution-idplugin inexamples/gdp-execution-id-example/.
Package: vdk-greenplum
This plugin allows vdk-core to interface with and execute queries against a Greenplum database.UsageRunpipinstallvdk-greenplumAfter this, data jobs will have access to a Greenplum database connection, managed by Versatile Data Kit SDK.If it is the only database plugin installed , vdk would automatically use it.
Otherwise, users need to set VDK_DB_DEFAULT_TYPE=GREENPLUM as an environment variable or set 'db_default_type' option in the data job config file (config.ini).For exampledefrun(job_input:IJobInput):job_input.execute_query("select 'Hi Greenplum!'")IngestionThis plugin allows users toingestdata to a Greenplum database,
which can be preferable to inserting data manually as it automatically handles serializing, packaging and sending of the data asynchronously with configurable batching and throughput.
To do so, you must set the expected variables to connect to Greenplum, plus the following environment variable:exportVDK_INGEST_METHOD_DEFAULT=GREENPLUMThen, from inside the run function in a Python step, you can use thesend_object_for_ingestionorsend_tabular_data_for_ingestionmethods to ingest your data.ConfigurationRun vdk config-help - search for those prefixed with "GREENPLUM_" to see what configuration options are available.TestingTesting this plugin locally requires installing the dependencies listed in vdk-plugins/vdk-greenplum/requirements.txtRunpipinstall-rrequirements.txt
Package: vdk-huggingface
HuggingfaceVersatile Data Kit (VDK) plugin for integrating with Huggingface as both a data source and a target.
This plugin allows you to ingest data payloads into a Huggingface repository and makes it easier to work with datasets stored in Huggingface.Usagepip install vdk-huggingfaceThe functionality adds new ingestion method "huggingface" which can be used like that:job_input.send_object_for_ingestion(data,method="huggingface")Configuration(vdk config-helpis useful command to browse all config options of your installation of vdk)NameDescription(example)  ValueHUGGINGFACE_TOKENHuggingFace API token for authentication. Get one from HuggingFace Settings""HUGGINGFACE_REPO_IDHuggingFace Dataset repository ID"username/test-dataset"Build and testingpip install -r requirements.txt
pip install -e .
pytestIn VDK repo../build-plugin.shscript can be used also.Note about the CICD:.plugin-ci.yaml is needed only for plugins part ofVersatile Data Kit Plugin repo.The CI/CD is separated in two stages, a build stage and a release stage.
The build stage is made up of a few jobs, all which inherit from the same
job configuration and only differ in the Python version they use (3.7, 3.8, 3.9 and 3.10).
They run according to rules, which are ordered in a way such that changes to a
plugin's directory trigger the plugin CI, but changes to a different plugin does not.
Package: vdk-ingest-file
VDK-INGEST-FILE PluginThis plugin provides functionality to ingest data into a file. It is intended for local testing.UsageTo use the plugin, just install it, and set themethodattribute ofsend_object_for_ingestion(),
andsend_tabular_data_for_ingestion()functions to"file".Example:defrun(job_input:IJobInput):# Do something to get data for ingestionpayload=get_some_data()# Ingest the datajob_input.send_object_for_ingestion(payload=payload,destination_table="aa_test_table",method="file",target="name_of_file")The above example shows how to ingest json data. In this case, there is only one argument that is required:payload.It needs to be a json object, and can contain thedestination_tableinside, using the@tablekey. For example:{
    "@table": "destination_table_name",
    "column1": "value1",
    "column2": "value2",
    "column3": "value3",
}Thetargetattribute is being used to specify the name of the file, where the data will be ingested. If not specified, it is constructed,
using the model,table.<creation-timestamp>.json.
Package: vdk-ingest-http
VDK-INGEST-HTTP PluginThis plugin provides functionality to ingest data over http.UsageTo use the plugin, just install it, and set themethodattribute ofsend_object_for_ingestion(),
andsend_tabular_data_for_ingestion()functions to"http".Example:defrun(job_input:IJobInput):# Do something to get data for ingestionpayload=get_some_data()# Ingest the datajob_input.send_object_for_ingestion(payload=payload,destination_table="aa_test_table",method="http",target="http://example.com/data-source")The above example shows how to ingest json data. In this case, there are three arguments that are required:payload,methodandtarget. This would be fixed in the future, so that onlypayloadwould be required.The payload needs to be a json object, and should contain thedestination_tableinside, using the@tablekey. For example:{
    "@table": "destination_table_name",
    "column1": "value1",
    "column2": "value2",
    "column3": "value3",
}Themethodattribute needs to be provided for the time being.Thetargetattribute should specify the url endpoint, where the data will be ingested.TestingTo develop or test locally (from the current directory)# install dev and test dependenciespipinstall-rrequirements# install the plugin in editable modepipinstall-e.# run the testspytest