Saravanan666/pypi-org huggingface dataset:
Package: datakit-github
This plugin provides light-weight GitHub integration for theDataKit command-line tool.It’s intended to streamline the process of generating a GitHub project and linking it to a local
directory containing a newly created project skeleton.DataKit is a project by The Associated Press designed to streamline data analysis workflows.
To learn more about the wider DataKit ecosystem and how AP uses it, check out theofficial documentation for DataKit.More information on the datakit-github plugin can be found below:Documentation:http://datakit-github.readthedocs.io/en/latest/GitHub:https://github.com/associatedpress/datakit-githubPyPI:https://pypi.python.org/pypi/datakit-githubFree and open source software:ISC license
Package: dataknead
datakneadFluent conversion between data formats like JSON, XML and CSVRead the docsEver sighed when you wrote code to convert CSV to JSON for the thousandth time?importcsvimportjsondata=[]withopen("cities.csv")asf:reader=csv.DictReader(f)forrowinreader:data.append(row)withopen("cities.json","w")asf:json.dump(data,f)Stop sighing and usedataknead. Fetch it withpip:$pipinstalldatakneadAnd use it like this:fromdatakneadimportKneadKnead("cities.csv").write("cities.json")Or make it even easier on the command line:kneadcities.csvcities.jsondatakneadhas inbuilt loaders for CSV, Excel, JSON, TOML and XML and you can easily write your own.Piqued your interest?Read the docs!.
Package: dataknobs
DataKnobsDescriptionUseful implementations of data structures and design patterns for knowledge bases and AI, or the knobs and levers for fine-tuning and leveraging your data.This repo also serves as a template or sandbox for development, experimentation, and testing of general data structures, algorithms, and utilities for DS, AI, ML, and NLP.Provides connectors for other popular text and data processing packages like:numpy and pandasnltkwordnetpostgreselasticsearchGeneral project informationThe purpose of this project is:To provide dependable implementations of useful data structures.To show examples of design patterns and ways to apply AI concepts.To prototype tools for delivering a robust DS/AI/ML/NLP utilities library package.To facilitate interactive development, demonstration, visualization, and testing of the library components via jupter notebooks and/or scripts.Installation and Usage% pip install dataknobs
% python
>>> import dataknobs as dk
>>> ...DevelopmentDevelopment machine prerequisitesThe following minimum configuration should exist for development:toxpyenvpyenv install 3.9poetryWith optional:dockerbashBy convention, a data directory can be leveraged for development that is mounted as a shared volumne in Docker as /data. This has the default of $HOME/data, but can be overridden with the DATADIR environment variable.Development quickstart guideIn a terminal, clone the repo and cd into the project directory.TestingTests and Lint: "tox"Just unit tests: "tox -e tests"Just lint: "tox -e lint"Using dockerDevelopment:% tox -e dev
# poetry shell
# pythonNotebook:execute "tox -e nb"copy/paste url into browserUsing virtual environmentsDevelopment:Manual: source ".project_vars", poetry install, poetry shellAutomated: execute "bin/start_dev.sh"  (requires "/bin/bash" on your machine)Notebook:execute "bin/start_notebook.sh"copy/paste url into browser
Package: datalab
Datalab is deprecated.Vertex AI Workbenchprovides a notebook-based environment that offers capabilities beyond Datalab.
We recommend that you use Vertex AI Workbench for new projects andmigrate your Datalab notebooks to Vertex AI Workbench.
For more information, seeDeprecation information.
To get help migrating Datalab projects to Vertex AI Workbench seeGet help.
Package: datalab-ltiauthenticator
LTI dataLAB AuthenticatorThis package contains LTI authenticators for JupyterHub with some custom config for username_key.Implements theLTI 1.3and theLTI v1.1authenticators for use with JupyterHub.This converts JupyterHub into an LTITool Provider, which can be then easily be used with variousTool Consumers, such as Canvas, Open EdX, Moodle, Blackboard, etc.So far,ltiauthenticatorhas been tested withOpen edX,Canvas, andMoodle. Documentation contributions are highly welcome!Note that with theseLTIauthenticators going directly to the hub URL will no longer allow you to log in. Youmustvisit the hub through an appropriate LTI 1.1 compliant Tool Consumer or LTI 1.3 compliant Platform (such as Canvas, Moodle, Open edX, etc.) to be able to log in.Note: LTI 1.1 identifies the LMS as theTool Consumerand LTI 1.3 identifies the LMS as thePlatformfor all practical purposes these terms are equivalent.InstallationYou can install the authenticator from PyPI:pipinstalldatalab-ltiauthenticatorUsing the LTI AuthenticatorsFor detailed instructions on how to configure theLTI13AuthenticatororLTI11Authenticatorand integrate it with an LMS, such as Canvas, Open EdX, Moodle, Blackboard, etc., please take a look at thedocumentation.
Package: datalad-cds
DataLad extension for the Copernicus Climate Data StoreWhat?A DataLad extension to integrate with the Copernicus Climate Data Store (CDS).
So far this just implements adatalad download-cdscommand that can be used to fetch data from the CDS
and record this action in a way so thatdatalad get(or justgit annex get) can redo the download in the future.Why?This extension enables automated provenance tracking for fetching data from the CDS.
In a dataset that retrieves data from the CDS using this extension it will become visible how this data was initially fetched
and how it can be retrieved again in the future.How?You will first have to create an account with the CDS,
if you don't have one already.
You can do so here:https://cds.climate.copernicus.eu/user/register?destination=%2F%23!%2FhomeNext,
you will need to create the "~/.cdsapirc" file as described here:https://cds.climate.copernicus.eu/api-how-to#install-the-cds-api-key.
This file is required since the datalad-cds extension internally uses the cdsapi package
and therefore uses its authentication mechanism.Also,
you need to install datalad and the datalad-cds extension.
Both can be had through pip.Now you are ready to use the extension.
When you look through the CDS you will notice that for any given dataset you can select a subset of the data using the "Download data" tab.
After you do that you can use the "Show API request" button at the bottom to get a short python script that would fetch the chosen subset using the cdsapi.
The following is an example of that:#!/usr/bin/env pythonimportcdsapic=cdsapi.Client()c.retrieve("reanalysis-era5-pressure-levels",{"variable":"temperature","pressure_level":"1000","product_type":"reanalysis","year":"2008","month":"01","day":"01","time":"12:00","format":"grib"},"download.grib",)To fetch the same data to the same local file using datalad-cds we just need to adapt this a little:$dataladdownload-cds--pathdownload.grib'{"dataset": "reanalysis-era5-pressure-levels","sub-selection": {"variable": "temperature","pressure_level": "1000","product_type": "reanalysis","year": "2008","month": "01","day": "01","time": "12:00","format": "grib"}}'The local path to save to ("download.grib") becomes the--pathargument.
The dataset name ("reanalysis-era5-pressure-levels" in this case) becomes the value of thedatasetkey in a json object that describes the data to be downloaded.
The sub-selection of the dataset becomes the value of thesub-selectionkey.After executing the abovedatalad download-cdscommand in a DataLad dataset a file called "download.grib" should be newly created.
This file will have its origin tracked in git-annex (you can see that by runninggit annex whereis download.grib).
If you nowdatalad dropthe file
and thendatalad getit you'll see that git-annex will automatically re-retrieve the file from the CDS
as if it was just another location to get data from.To see more possible usage options take a look at the help page of the command (datalad download-cds --help)
or the documentation athttps://matrss.github.io/datalad-cds/.
Package: datalad-container
____          _           _                 _
|  _ \   __ _ | |_   __ _ | |      __ _   __| |
| | | | / _` || __| / _` || |     / _` | / _` |
| |_| || (_| || |_ | (_| || |___ | (_| || (_| |
|____/  \__,_| \__| \__,_||_____| \__,_| \__,_|
                                   ContainerThis extension enhances DataLad (http://datalad.org) for working with
computational containers. Please see theextension
documentationfor a description on additional commands and functionality.For general information on how to use or contribute to DataLad (and this
extension), please see theDataLad websiteor themain GitHub project page.InstallationBefore you install this package, please make sure that youinstall a recent
version of git-annex.  Afterwards,
install the latest version ofdatalad-containerfromPyPi. It is recommended to use
a dedicatedvirtualenv:# create and enter a new virtual environment (optional)
virtualenv --system-site-packages --python=python3 ~/env/datalad
. ~/env/datalad/bin/activate

# install from PyPi
pip install datalad_containerIt is also available for conda package manager from conda-forge:conda install -c conda-forge datalad-containerSupportThe documentation of this project is found here:http://docs.datalad.org/projects/containerAll bugs, concerns and enhancement requests for this software can be submitted here:https://github.com/datalad/datalad-container/issuesIf you have a problem or would like to ask a question about how to use DataLad,
pleasesubmit a question to
NeuroStars.orgwith adataladtag.
NeuroStars.org is a platform similar to StackOverflow but dedicated to
neuroinformatics.All previous DataLad questions are available here:http://neurostars.org/tags/datalad/AcknowledgementsDataLad development is supported by a US-German collaboration in computational
neuroscience (CRCNS) project "DataGit: converging catalogues, warehouses, and
deployment logistics into a federated 'data distribution'" (Halchenko/Hanke),
co-funded by the US National Science Foundation (NSF 1429999) and the German
Federal Ministry of Education and Research (BMBF 01GQ1411). Additional support
is provided by the German federal state of Saxony-Anhalt and the European
Regional Development Fund (ERDF), Project: Center for Behavioral Brain
Sciences, Imaging Platform.  This work is further facilitated by the ReproNim
project (NIH 1P41EB019936-01A1).
Package: datalad-crawler
____          _           _                 _
|  _ \   __ _ | |_   __ _ | |      __ _   __| |
| | | | / _` || __| / _` || |     / _` | / _` |
| |_| || (_| || |_ | (_| || |___ | (_| || (_| |
|____/  \__,_| \__| \__,_||_____| \__,_| \__,_|
                                   CrawlerThis extension enhances DataLad (http://datalad.org) for crawling external
web resources into an automated data distribution. Please see theextension
documentationfor a description on additional commands and functionality.For general information on how to use or contribute to DataLad (and this
extension), please see theDataLad websiteor themain GitHub project page.InstallationBefore you install this package, please make sure that youinstall a recent
version of git-annex.  Afterwards,
install the latest version ofdatalad-crawlerfromPyPi. It is recommended to use
a dedicatedvirtualenv:# create and enter a new virtual environment (optional)
virtualenv --system-site-packages --python=python3 ~/env/datalad
. ~/env/datalad/bin/activate

# install from PyPi
pip install datalad_crawlerSupportThe documentation of this project is found here:http://docs.datalad.org/projects/crawlerAll bugs, concerns and enhancement requests for this software can be submitted here:https://github.com/datalad/datalad-crawler/issuesIf you have a problem or would like to ask a question about how to use DataLad,
pleasesubmit a question to
NeuroStars.orgwith adataladtag.
NeuroStars.org is a platform similar to StackOverflow but dedicated to
neuroinformatics.All previous DataLad questions are available here:http://neurostars.org/tags/datalad/AcknowledgementsDataLad development is supported by a US-German collaboration in computational
neuroscience (CRCNS) project "DataGit: converging catalogues, warehouses, and
deployment logistics into a federated 'data distribution'" (Halchenko/Hanke),
co-funded by the US National Science Foundation (NSF 1429999) and the German
Federal Ministry of Education and Research (BMBF 01GQ1411). Additional support
is provided by the German federal state of Saxony-Anhalt and the European
Regional Development Fund (ERDF), Project: Center for Behavioral Brain
Sciences, Imaging Platform.  This work is further facilitated by the ReproNim
project (NIH 1P41EB019936-01A1).
Package: datalad-dataverse
DataLad extension for working DataverseDataverseis open source research data repository software that is deployed all over the world as data or metadata repositories.
It supports sharing, preserving, citing, exploring, and analyzing research data with descriptive metadata, and thus contributes greatly to open, reproducible, and FAIR science.DataLad, on the other hand, is a data management and data publication tool build on Git and git-annex.
Its core data structure, DataLad datasets, can version control files of any size, and streamline data sharing, updating, and collaboration.
This DataLad extension package provides interoperablity with Dataverse to support dataset transport to and from Dataverse instances.Installation# create and enter a new virtual environment (optional)
$ virtualenv --python=python3 ~/env/dl-dataverse
$ . ~/env/dl-dataverse/bin/activate
# install from PyPi
$ python -m pip install datalad-dataverseHow to useAdditional commands provided by this extension are immediately available
after installation. However, in order to fully benefit from all improvements,
the extension has to be enabled for auto-loading by executing:git config --global --add datalad.extensions.load dataverseDoing so will enable the extension to also alter the behavior the core DataLad
package and its commands, from example to be able to directly clone from
a Dataverse dataset landing page.Summary of functionality provided by this extensionInteroperability between DataLad and Dataverse version 5 (or later).Aadd-sibling-dataversecommand to register a Dataverse dataset as remote sibling for a DataLad dataset.Agit-annex-remote-dataversespecial remote implementation for storage and retrieval of data in Dataverse dataset via git-annex.These two features combined enable the deposition and retrieveal of complete DataLad dataset on Dataverse, including version history and metadata. A directdatalad clonefrom a Dataverse dataset landing page is supported, and yields a fully functional DataLad dataset clone (Git repository).Contributors ✨Thanks goes to these wonderful people (emoji key):Johanna Bayer📖Nadine Spychala🚇📖Benjamin Poldrack🚇💻📖🚧👀🤔🔧Adina Wagner💻🤔🚇📖🚧👀Michael Hanke💻🤔🚧🚇👀🔧enicolaisen📖Roza📖Kelvin Sarink💻Jan Ernsting💻Chris Markiewicz💻Alex Waite🚇💻🚧🔧Shammi270787💻Wu Jianxiao💻👀📓Laura Waite📖Michał Szczepanik🚇This project follows theall-contributorsspecification. Contributions of any kind welcome!AcknowledgementsThis DataLad extension was developed with support from the German Federal
Ministry of Education and Research (BMBF 01GQ1905), the US National Science
Foundation (NSF 1912266), the Helmholtz research center Jülich (RDM challenge
2022), and the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) under grant SFB 1451
(431549029, INF project).
Package: datalad-deprecated
DataLad extension for functionality that was phased out of the core packageCommands provided by this extensionannotate-paths-- analyze and act upon input pathsdiff-- old'ish implementation for differences between two statesls-- list summary information about URLs and datasets
of a dataset (hierarchy)publish-- original implementation, replaced bypushin coremetadata commandssearch,metadata,extract-metadataandaggregate-metadata; moved out of core in favor of thedatalad-metaladextensionProcedures provided by this extensioncfg_metadatatypesIn addition, this extension provides theAutomagicIOfeature that patches Python'sopen()function to automatically obtain file content, if needed.It also provides the web user interface that is used athttp://datasets.datalad.org(yet to be migrated tohttps://github.com/datalad/datalad-webuifully).AcknowledgementsThis work was performed by a US-German collaboration in computational
neuroscience (CRCNS) co-funded by the US National Science Foundation (NSF
1429999) and the German Federal Ministry of Education and Research (BMBF
01GQ1411).
Package: datalad-fuse
DataLad FUSE extension packagedatalad-fuseprovides commands for reading files in aDataLaddataset from their remote web URLs without having
to download them in their entirety first.  Instead,fsspecis used to sparsely download
and locally cache the files as needed.InstallationCurrent version ofdatalad-fuserequires Python 3.7 or higher.  Just usepipfor Python 3 (You have pip, right?) to install it:python3 -m pip install datalad-fuseIn addition, use of thedatalad fusefscommand requires FUSE to be installed;
on Debian-based systems, this can be done with:sudo apt-get install fuseCommandsdatalad fsspec-cache-clear [<options>]Clears the local download cache for a dataset.Options-d <DATASET>,--dataset <DATASET>— Specify the dataset to operate on.
If no dataset is given, an attempt is made to identify the dataset based on
the current working directory.-r,--recursive— Clear the caches of subdatasets as well.datalad fsspec-head [<options>] <path>Shows leading lines/bytes of an annexed file by fetching its data from a remote
URL.Options-d <DATASET>,--dataset <DATASET>— Specify the dataset to operate on.
If no dataset is given, an attempt is made to identify the dataset based on
the current working directory.-n <INT>,--lines <INT>— How many lines to show (default: 10)-c <INT>,--bytes <INT>— How many bytes to showdatalad fusefs [<options>] <mount-path>Create a read-only FUSE mount at<mount-path>that exposes the files in the
given dataset.  Opening a file under the mount that is not locally present in
the dataset will cause its contents to be downloaded from the file's web URL as
needed.When the command finishes,fsspec-cache-clearmay be run depending on the
value of thedatalad.fusefs.cache-clearconfiguration option.  If it is set
to "visited", then any (sub)datasets that were accessed in the FUSE mount
will have their caches cleared; if it is instead set to "recursive", then all
(sub)datasets in the dataset being operated on will have their caches cleared.Options--allow-other— Allow all users to access files in the mount.  This
requires settinguser_allow_otherin/etc/fuse.conf.-d <DATASET>,--dataset <DATASET>— Specify the dataset to operate on.
If no dataset is given, an attempt is made to identify the dataset based on
the current working directory.-f,--foreground— Run the FUSE process in the foreground; use Ctrl-C to
exit.  This option is currently required.--mode-transparent— Expose the dataset's.gitdirectory in the mount
Package: datalad-gooey
DataLad Gooey (pronounce "GUI")This package provides a graphical user interface (GUI) forDataLad. It is specifically aiming at making key data
management tasks more accessible and more convenient, without requiring to
become familiar with the command line.This simplified interface to DataLad is built on a foundation that is capable
of providing graphical user interfaces for any DataLad command, including those
provided by extension packages. Moreover, extension packages can even provide
their own GUI suites, by mixing and tuning a custom set of commands and
parameters.To try it out, install this package, and rundatalad gooey.AcknowledgementsDataLad development is supported by a US-German collaboration in computational
neuroscience (CRCNS) project "DataGit: converging catalogues, warehouses, and
deployment logistics into a federated 'data distribution'" (Halchenko/Hanke),
co-funded by the US National Science Foundation (NSF 1429999) and the German
Federal Ministry of Education and Research (BMBF 01GQ1411).This DataLad extension was developed with additional funding from the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation) under grant SFB 1451Contributors ✨Thanks goes to these wonderful people (emoji key):Michael Hanke💻🤔📆🧑‍🏫Yaroslav Halchenko💻Chris Markiewicz💻Adina Wagner💻John T. Wodder II💻Benjamin Poldrack💻Stephan Heunis💻Michał Szczepanik💻Alex Waite📓🤔Leonardo Muller-Rodriguez📓💻Laura Waite💻Christian Mönch💻This project follows theall-contributorsspecification. Contributions of any kind welcome!
Package: datalad-lgpdextension
DataLad extension templateThis repository contains an extension template that can serve as a starting point
for implementing aDataLadextension. An extension can
provide any number of additional DataLad commands that are automatically
included in DataLad's command line and Python API.For a version, clone this repository and install the demo extension viapip install -e . or pip install lgpdextension-dataladDataLad will now expose a new command suite with ahello...command.% datalad --help |grep -B2 -A2 hello
*DataLad command suite*

  lgpd-extension
      Apply lgpd patterns

    -p | --pathfile
      Full address to configuration file
    -c | --createbase
      Create the default configuration fileTo start implementing your own extension,use this
template, and
adjust as necessary. A good approach is toPick a name for the new extension.Look through the sources and replacedatalad_lgpdextensionwithdatalad_<newname>(hint:git grep datalad_lgpdextensionshould find all
spots).Delete the example command implementation indatalad_lgpdextension/__init__.pyby (re)moving thelgpdextensionclass.Implement a new command, and adjust thecommand_suiteindatalad_lgpdextension/__init__.pyto point to it.Replacelgpd_extensionwith the name of the new command indatalad_lgpdextension/tests/test_register.pyto automatically test whether the
new extension installs correctly.Adjust the documentation indocs/source/index.rst.Replace this README.Updatesetup.cfgwith appropriate metadata on the new extension.You can consider filling in the provided.zenodo.jsonfile with
contributor information andmeta datato acknowledge contributors and describe the publication record that is created whenyou make your code citeableby archiving it usingzenodo.org. You may also want to
consider acknowledging contributors with theallcontributors bot.