Saravanan666/pypi-org huggingface dataset:
Package: datalad-metadata-model
Datalad Metadata ModelThis software implements the metadata model that datalad and datalad-metalad
(from version 0.3.0) use to store metadata.Model Elements (the model layer)The metadata model is defined by the API of the top-level
classes. Those are:MetadataRootRecord-- holds top-level metadata information for a single
version of a datalad datasetUUIDSet-- holds metadata root records for a set of datasets that are
identified by their UUIDs and their version.TreeVersionList-- holds metadata root records and a sub-dataset tree for a
dataset version and its sub-datasetsMetadata-- represents metadata for a single item, i.e. dataset or file.
Metadata is associated with extractor names and extraction parameters.DatasetTree-- a representation of the sub-dataset hierarchy of a datasetFileTree-- a representation of the file-tree of a dataset...Because of the large size of some datalad-datasets, e.g. tens of thousands of
sub-datasets and hundres of millions of files, the implementation allows
focus-based operations on individual parts of the potentially very large
metadata model. The implementation uses the proxy-pattern, that means, it
loads, modifies, and saves only the minimal necessary model elements that are
necessary to operate on the metadata-information that
the user is interested in.Storage layerThe model elements have to be persisted on a storage backend.
How the model is mapped on storage backends is defined by the
storage layer, that is to a large degree independent of the model layer.
The intention is to support multiple storage backends in the past.Currently, only one storage backend is supported:git-mapping-- a storage backend that stores a metadata model in a
git repository. The model objects are stored outside of existing branches.
They are referenced bydatalad-specific git-references underrefs/datalad/*AcknowledgementsThis DataLad extension was developed with support from the German Federal Ministry of Education and Research (BMBF 01GQ1905), and the US National Science Foundation (NSF 1912266).
Package: datalad-mihextras
MIH's extra DataLad functionalityDataLad extension with special interest functionality or drafts of future
additions to DataLad proper.This is a staging area to test-drive new functionality in the wild without
making any promises on longevity or stability. Once a certain level of maturity
has been reached, most functionality will move to more appropriate places, like
topical extensions or even DataLad proper.Command(s) provided by this extensionx-export-bagit-- export datasets (recursively) to an RFC8493-compliant
BagIt "bag"x-snakemake-- thin wrapper aroundSnakeMaketo obtain file content prior processingDataset procedure(s) provided by this extensioncfg_kdenlive-- configures a dataset for use as a KDENLIVE project.
Manages commits of the project file to be more portable, and configures
a standard set of temporary directories for KDENLIVE.Once available functionalityx-configurationhas been migrated to the core DataLad package under the nameconfiguration.git-remote-datalad-annex-- has been migrated to an improved implementation
provided byhttps://github.com/datalad/datalad-nextgit-annex-backend-XDLRA, and the base class to facilitate development of
external backends in Python -- has been migrated to an improved implementation
provided byhttps://github.com/datalad/datalad-nextx-export-to-webdav-- is discontinued.https://github.com/datalad/datalad-nextprovidescreate-sibling-webdav(supports export remotes) and enhances the
standardpushcommand to be able to handle exports automatically. This
removed the need for a custom implementation.
Package: datalad-neuroimaging
____          _           _                 _
|  _ \   __ _ | |_   __ _ | |      __ _   __| |
| | | | / _` || __| / _` || |     / _` | / _` |
| |_| || (_| || |_ | (_| || |___ | (_| || (_| |
|____/  \__,_| \__| \__,_||_____| \__,_| \__,_|
                                   NeuroimagingThis extension enhances DataLad (http://datalad.org) for working with
neuroimaging data and workflows. Please see theextension
documentationfor a description on additional commands and functionality.For general information on how to use or contribute to DataLad (and this
extension), please see theDataLad websiteor themain GitHub project page.InstallationBefore you install this package, please make sure that youinstall a recent
version of git-annex.  Afterwards,
install the latest version ofdatalad-neuroimagingfromPyPi. It is recommended to use
a dedicatedvirtualenv:# create and enter a new virtual environment (optional)
virtualenv --system-site-packages --python=python3 ~/env/dataladni
. ~/env/dataladni/bin/activate

# install from PyPi
pip install datalad_neuroimagingThere is also aSingularity containeravailable.
The latest release version can be obtained by running:singularity pull shub://datalad/datalad-neuroimagingSupportThe documentation of this project is found here:http://docs.datalad.org/projects/neuroimagingAll bugs, concerns and enhancement requests for this software can be submitted here:https://github.com/datalad/datalad-neuroimaging/issuesIf you have a problem or would like to ask a question about how to use DataLad,
pleasesubmit a question to
NeuroStars.orgwith adataladtag.
NeuroStars.org is a platform similar to StackOverflow but dedicated to
neuroinformatics.All previous DataLad questions are available here:http://neurostars.org/tags/datalad/AcknowledgementsDataLad development is supported by a US-German collaboration in computational
neuroscience (CRCNS) project "DataGit: converging catalogues, warehouses, and
deployment logistics into a federated 'data distribution'" (Halchenko/Hanke),
co-funded by the US National Science Foundation (NSF 1429999) and the German
Federal Ministry of Education and Research (BMBF 01GQ1411). Additional support
is provided by the German federal state of Saxony-Anhalt and the European
Regional Development Fund (ERDF), Project: Center for Behavioral Brain
Sciences, Imaging Platform.  This work is further facilitated by the ReproNim
project (NIH 1P41EB019936-01A1).
Package: datalad-osf
DataLad-OSF: Opening up the Open Science Framework for DataLadWelcome! This repository contains aDataLadextension that enables DataLad to work with the Open Science Framework (OSF). Use it to share, retrieve and collaborate on DataLad datasets via the OSF.The development of this tool started atOHBM Brainhack 2020in June 2020, coordinated inthis repository. See ourdocumentationfor more extensive information.RequirementsDatalad: relies ongit-annex,Gitand Python. If you don’t have DataLad installed yet, please follow the instructionshere.Account on theOpen Science Framework (OSF)Installation# create and enter a new virtual environment (optional)
$ virtualenv --python=python3 ~/env/dl-osf
$ . ~/env/dl-osf/bin/activate
# install from PyPi
$ pip install datalad-osfHow to useSee ourdocumentationfor more info on how to use this tool and a tutorial on major use cases.How to contributeYou are very welcome to help out developing this tool further. You can contribute by:Creating an issue for bugs or tips for further developmentMaking a pull request for any changes suggested by yourselfTesting out the software and communicating your feedback to usPlease see our contributing guidelines for more information.Contributors ✨Thanks goes to these wonderful people (emoji key):Michael Hanke🚧💻🐛🤔Dorien Huijser📖📆🤔📓Ashish Sahoo📖🚧Simon Steinkamp⚠️📖📆🤔📓🚧Benjamin Poldrack📆🤔💻🚧Nick📆🤔💻🚧Nikita Beliy🤔📓Moritz J. Boos💻📓🤔🚧Adina Wagner📆🤔💻📖🚧Stefan Appelhoff📖📓Tom Hu🚇This project follows theall-contributorsspecification. Contributions of any kind welcome!AcknowledgementsThis DataLad extension was developed with support from the German Federal
Ministry of Education and Research (BMBF 01GQ1905), and the US National Science
Foundation (NSF 1912266).
Package: datalad-revolution
Travis tests
statusBuild
statuscodecov.ioGitHub
releasePyPI version
fury.ioDocumentationThis software is aDataLadextension that
equips DataLad with alternative and additional core commands that are
leaner and written specifically with enhanced cross-platform
compatibility and speed in mind. Please see theextension
documentationfor a
description on additional commands and functionality.Note:There isno supportforgit-annex direct mode
repositories. Users
that previously relied on this mode, and Windows users in particular,
are recommended to usegit-annex V7
mode. DataLad can be
instructed to always use this mode by running:git config --global --add datalad.repo.version 6Command(s) currently provided by this extensionrev-extract-metadata– new and improved dedicated command to run
any and all of DataLad’s metadata extractors. Much improved versions
of the core extractor implementations also ship with this extension.rev-aggregate-metadata– complete reimplementation of metadata
aggregation, with stellar performance benefits, in particular on
large dataset hierarchies.query-metadata– new command to specifically access the
aggregated metadata present in a dataset, much faster and more
predictable behavior than themetadatacommand in datalad-core.InstallationBefore you install this package, please make sure that youinstall a
recent version of
git-annex. Afterwards,
install the latest version ofdatalad-revolutionfromPyPi. It is
recommended to use a dedicatedvirtualenv:# create and enter a new virtual environment (optional)
virtualenv --system-site-packages --python=python3 ~/env/datalad
. ~/env/datalad/bin/activate

# install from PyPi
pip install datalad_revolutionSupportFor general information on how to use or contribute to DataLad (and this
extension), please see theDataLad websiteor
themain GitHub project page. The documentation
is found here:http://docs.datalad.org/projects/revolutionAll bugs, concerns and enhancement requests for this software can be
submitted here:https://github.com/datalad/datalad-revolution/issuesIf you have a problem or would like to ask a question about how to use
DataLad, pleasesubmit a question to
NeuroStars.orgwith adataladtag. NeuroStars.org is a platform similar to StackOverflow
but dedicated to neuroinformatics.All previous DataLad questions are available here:http://neurostars.org/tags/datalad/AcknowledgementsDataLad development is supported by a US-German collaboration in
computational neuroscience (CRCNS) project “DataGit: converging
catalogues, warehouses, and deployment logistics into a federated ‘data
distribution’” (Halchenko/Hanke), co-funded by the US National Science
Foundation (NSF 1429999) and the German Federal Ministry of Education
and Research (BMBF 01GQ1411). Additional support is provided by the
German federal state of Saxony-Anhalt and the European Regional
Development Fund (ERDF), Project: Center for Behavioral Brain Sciences,
Imaging Platform. This work is further facilitated by the ReproNim
project (NIH 1P41EB019936-01A1).
Package: datalad-xnat
DataLad extension tracking data in an XNAT serverThis software is aDataLadextension that equips DataLad
with a set of commands to track XNAT projects.XNAT is an open source imaging informatics platform developed by the
Neuroinformatics Research Group at Washington University. It facilitates common
management, productivity, and quality assurance tasks for imaging and
associated data. XNAT can be used to support a wide range of neuro/medical
imaging-based projects.Command(s) provided by this extensionxnat-init-- Initialize an existing dataset to track an XNAT projectxnat-update-- Update an existing dataset of an XNAT project; retrieve data from the tracked projectxnat-query-files-- Query available files on an XNAT instance or projectInstallationBefore you install this package, please make sure that youinstall a recent
version of git-annex.  Afterwards,
install the latest version ofdatalad-xnatfromPyPi. It is recommended to use
a dedicatedvirtualenv:# create and enter a new virtual environment (optional)
virtualenv --system-site-packages --python=python3 ~/env/datalad
. ~/env/datalad/bin/activate

# install from PyPi
pip install datalad_xnatSupportFor general information on how to use or contribute to DataLad (and this
extension), please see theDataLad websiteor themain GitHub project page.All bugs, concerns and enhancement requests for this software can be submitted here:https://github.com/datalad/datalad-xnat/issuesIf you have a problem or would like to ask a question about how to use DataLad,
pleasesubmit a question to
NeuroStars.orgwith adataladtag.
NeuroStars.org is a platform similar to StackOverflow but dedicated to
neuroinformatics.All previous DataLad questions are available here:http://neurostars.org/tags/datalad/AcknowledgementsThis development was supported by European Union’s Horizon 2020 research and
innovation programme under grant agreementVirtualBrainCloud
(H2020-EU.3.1.5.3, grant no.
826421).Contributors ✨Thanks goes to these wonderful people (emoji key):Michael Hanke💻🐛📖🤔🚧Laura Waite💻🐛🤔🚧Adina Wagner💻🐛📖🤔🚧John T. Wodder II💻🤔🚧Yaroslav Halchenko💻Janvi Raina📖Stephan Heunis🚇🎨🧑‍🏫tsankeuodelfa📖Michał Szczepanik📖💻Benjamin Poldrack💻Alex Waite💬📖✅oportoles📓📖🐛✅This project follows theall-contributorsspecification. Contributions of any kind welcome!
Package: datalake-catalog
Setup and start the APIConfigure the parameters with apython fileFor example,catalog.conf.pySECRET_KEY=b"changemenow"DB_STRING="sqlite://localhost/catalog.sqlite"Start the catalogdockerrun-d\-p'8080:8080'\-v'catalog.conf.py:/etc/datacatalog/catalog.conf.py'\-e'CATALOG_SETTINGS=/etc/datacatalog/catalog.conf.py'\public.ecr.aws/equancy-tech/datalake-catalogGenerate an API tokenThecatalog create-api-keygenerated tokens that can be used with restricted endpoint.adminrole can access all restricted endpointsauthorrole can only access the restricted endpoints in/catalogUsage:catalogcreate-api-key[OPTIONS]NAME

Options:-e,--expiresINTEGERnumberofdaysbeforetokenexpires-r,--role[author|admin]theroleassociatedwiththename--helpShowthismessageandexit.Update the storages configurationStorage configures aliases to help resolve actual buckets (S3, Azure, local fs, etc.).
It consists of a bucket name and an optional prefix.For example:{"landing":{"bucket":"my-raw-bucket","prefix":"landing"},"archive":{"bucket":"my-raw-bucket","prefix":"archives"},"bronze":{"bucket":"my-bucket-bronze"},"silver":{"bucket":"my-bucket-silver"},"gold":{"bucket":"my-bucket-gold",}}curl-XPUThttp://localhost:8080/storage\-H"Content-Type: application/json"\-H"Authorization: Bearer${CATALOG_ADMIN_TOKEN}"\--data"@${STORAGE_CONFIG_FILE}"
Package: datalake-framework
Datalake FrameworkA Python framework for working with a datalake's cloud assets likedata catalog,storage,monitoringorsecretsThe framework helps making abstraction of cloud resources when developing use-cases for a datalake. Supported cloud providers areAmazon Web ServicesGoogle Cloud PlaformMicrosoft AzureAdditionnaly, most cloud features in the framework are available forlocal development. Hence developers can work on use-cases without requiring a full cloud insfrastructure or even work offline.InstallationUsing pippipinstalldatalake-framework
Package: datalake-scripts
____        _        _       _          ____            _       _ 
|  _ \  __ _| |_ __ _| | __ _| | _____  / ___|  ___ _ __(_)_ __ | |_ ___
| | | |/ _` | __/ _` | |/ _` | |/ / _ \ \___ \ / __| '__| | '_ \| __/ __|
| |_| | (_| | || (_| | | (_| |   <  __/  ___) | (__| |  | | |_) | |_\__ \
|____/ \__,_|\__\__,_|_|\__,_|_|\_\___| |____/ \___|_|  |_| .__/ \__|___/
                                                          |_|datalakeDatalake scriptsHow to useInstallationWith Python 3.6+:$ pip install datalake-scripts
$ pip3 install datalake-scriptsUsing as a libraryseethe following linkUsing as a CLIThe cli can be used with:$ocd-dtl<command><parameter>Checkocd-dtl -hfor help, including the list of commands available.You can also use a script directly by using the following command:<script_name> <script_options>./!\ Make sure to use utf-8without BOMwhen providing a file (-i option)Environment variablesAuthenticationIn case you don't want to enter credential for each commands and you are on a secured terminal, set those variables:OCD_DTL_USERNAMEemail address used to login on Datalake API/GUI.OCD_DTL_PASSWORDpassword used to login on Datalake API/GUI.They are independent and one can be used without the other if you wish.ThrottlingFor throttling the request, those two environment variable can be used:OCD_DTL_QUOTA_TIMEdefine, in seconds, the time before resetting the requests limit,default is 1 second.OCD_DTL_REQUESTS_PER_QUOTA_TIMEdefine the number of request to do at maximum for the given time,default is 5 queries.Please don't exceed the quota markedherefor each endpointCli parametersParameters common and optional for all commands:--debug  display more information for debugging purposes-e to change the environment {preprod, prod},  default isprod-o will set the output file as the API gives it.-q will quiet the verbosity of the program (but still show errors / warnings)For information about each command and more, please checkthe documentation directoryContributingTo develop on this repository, please refer tothis file
Package: datalake-utils
Data Lake UtilityPackage to manipulate data from/into Amazon S3 using partitions compatible with Apache Hadoop filesystem.
At this moment, this package was conceived to handle JSON and Parquet formats. That being said, it expects a Pandas DataFrame.Data will be written into Amazon S3 as a multi-line JSON or Apache Parquet, compressed as GZIP.FeaturesConvert list of dictionaries...[{"brand":"Ford","model":"Mustang","year":1965},{"brand":"Pontiac","model":"GTO","year":1964},{"brand":"Lamborghini","model":"Miura","year":1966}]...to multi-line JSON compressed as GZIP{"brand": "Ford","model": "Mustang","year": 1965}
{"brand": "Pontiac","model": "GTO","year": 1964}
{"brand": "Lamborghini","model": "Miura","year": 1966}...or to Apache Parquet compressed as GZIPHow to useThis package does not slice data into partitions at the moment. You must handle slicing of data to write into partitions desired.Check this example below:fromdatalake_utils.utilsimportDataLakeimportpandasdata=[{"brand":"Ford","model":"Mustang","year":1965},{"brand":"Pontiac","model":"GTO","year":1964},{"brand":"Lamborghini","model":"Miura","year":1966}]datalake=DataLake(bucket_name="vehicles",schema="motor_vehicles",table="cars",partitions=[{"key":"decade","value":"1960s"}],)datalake.append_to_s3(data=pandas.DataFrame(data),file_format="json")It will create an object into Amazon S3 with the following structure:motor_vehicles/
`-- cars/
    `-- decade=1960s/
        `-- 2c0fea6c-444e-11ed-969f-acde48001122.json.gzTo read all files from partition, do:retrieved_data=datalake.read_from_s3(file_format="json")retrieved_datais a Pandas DataFrame object. It is not possible to read a specific file using this function.To convert a Pandas DataFrame into a list of tuples containing a single JSON:datalake.df_to_tuples(data=retrieved_data)To delete all files from partition, do:datalake.delete_from_s3()
Package: datalang
DatalangDatalang it's an easy to use language with a similar syntax to YAML.How-to InstallYou will needPIPgitclonehttps://github.com/ZSendokame/datalang# OrpipinstalldatalangHow-to Useimportdatalangload=datalang.load('''str string: valueint integer: 1list listVariable:- value0- value1dict dictVariable:- variable0: value0- variable1: value1''')print(load){'string':'value','integer':1,'listVariable':['value0','value1'],'dictVariable':{'variable0':'value0','variable1':'value1'}}# And vice-versa withprint(datalang.dump(load))
Package: datalayer
Ξ DatalayerDatalayer core.Datalayer is the base foundation package used by many other Datalayer packages. It contains base application classes and configuration inherited by other projects.It is also the meta package to get the other Datalayer packages installed.It also ships a JupyterLab extension to inspect the JupyterLab's internal strutures like plugin graph, file types, models and widgets.Developyarn
yarnbuild# open http://localhost:3063# open http://localhost:8888/api/jupyter/lab?token=60c1661cc408f978c309d04157af55c9588ff9557c9380e4fb50785750703da6yarnstartpipinstall-e.[test]jupyterlabextensiondevelop.--overwrite
jupyterlabextensionlist# jupyter server extension enable datalayerjupyterserverextensionlist# open http://localhost:8888/api/jupyter/lab?token=60c1661cc408f978c309d04157af55c9588ff9557c9380e4fb50785750703da6yarnjupyterlabReleasesJupyter Viewer is released inPyPI.
Package: data-layer-lib
data-layer-libDatabase Layer LibraryintroductionThis is a library package for handling database models.Getting Started1- Clone the repository2- To manually create a virtualenv$ python -m venv .venv3- To install all the dependencies$ poetry install4- To create a migration for the first time, execute the following command$ alembic revision --autogenerate -m "migration name"The same command can be used to generate a new migrations5- Verify the alembic autogenerated db scripts6- To apply migrations, execute the following command. This will to create/update tables in the database$ alembic upgrade head
Package: datalens
Data LensVisualise your dataset before training the model in one line!Made changes to the bounding boxes or images?
Save time by visualising the data and avoid mistakes before starting the training processUse Case:Sanity check for the right dataset and annotationsDont forget to resize the bounding boxesVisualise what augmentation functions do to your dataUsing simple GUI, visualise random images for your tasksInstallation (Python 3.6+):pip install datalensUsage:import datalens
datalens.Visualise(image_dir_path = image_dir_path, annotations_dict = annotations_dict, count = count)Current version supports Object Detection in 2D images.Data Formatting:image_dir_path = #PATH TO IMAGE DIRECTORY
annotations_dict = #OBJECT DETECTION - {image_filename: [{"bbox": [x, y, width, height], "category_id": <int>}, ...], ...}
count = 10 #NUMBER OF IMAGES TO VISUALISEContributions are welcome for different machine learning tasks for text, images and 3D Point cloud data.
Package: datalib
==================================================This library is now part of the ambhas(http://pypi.python.org/pypi/ambhas/) library.Updates will only be made in the ambhas library.==============================================================datalib============This is a package/library in python to work with time series data from xls file.This is being developed under the project `AMBHAS <http://www.ambhas.com/>`_.Installing datalib======================Installing datalib can be done by downloading source file (datalib--<version>.tar.gz),and after unpacking issuing the command::python setup.py installThis requires the usual Distutils options available.Or, download the datalib--<version>.tar.gz file and issue the command::pip /path/to/datalib--<version>.tar.gzOr, directly using the pip::pip install datalibUsage=========Import required modules::import numpy as npfrom datalib import time_dataRead the time series data::data = time_data('/pata/to/file.xls')year = data.get('year')month = data.get('month')gw_observed = 839-data.get('gw')Changelog=========0.0.0 (July 2011)------------------- Initial releaseAny questions/comments=========================If you have any comment/suggestion/question,please feel free to write me at satkumartomer@gmail.comSo far, the documents are not well explained,I will be updating them soon.