Saravanan666/pypi-org huggingface dataset:
Package: dataset-convertor
👉 Convert object detection dataset format 👈Dataset typesPASCAL VOC: Pascal voc dataset have a xml file for each image.YOLO: YOLO dataset have a txt file for each image.COCO: COCO dataset have a json file for each image.Current support formatCurrently, the following formats are supported:fromtoimplementedPASCAL VOCYOLO(TXT files)YesYOLOPASCAL VOC (XML files)YesUpcoming support formatfromtoIssue/PR(if any)PASCAL VOCCOCO (JSON files)NoPASCAL VOCTFRecord (TFRecord files)NoCOCOPASCAL VOC (XML files)NoCOCOYOLO (TXT files)NoCOCOTFRecord (TFRecord files)NoYOLOCOCO (JSON files)NoYOLOTFRecord (TFRecord files)NoInstallationInstallation from source codegitclonehttps://github.com/codePerfectPlus/dataset-convertor/cddataset-convertor
python-mvenvvenvsourcevenv/bin/activate
pipinstallrequirements.txtInstallation from PyPIpipinstalldataset-convertorUsageconvert annotations from one format to another format.dataset formatting example:- data/pascal_voc/JPEGImages/*.jpg
- data/pascal_voc/Annotations/*.xml

- data/yolo5/JPEGImages/*.jpg
- data/yolo5/labels/*.txtPascal VOC(xml) to yolo(txt)fromconvertimportConvertorcon=Convertor(input_folder='/home/user/data/pascal_voc',output_folder='/home/user/data/yolo5')con.voc2yolo()from yolo(txt) to Pascal VOC(xml)fromconvertimportConvertorcon=Convertor(input_folder='/home/user/data/yolo5',output_folder='/home/user/data/pascal_voc')con.yolo2voc()Contributingcreate an issue/PR if any format is missing.Open-source contribution is welcome.check thecontributing guidefor details.ReferencePASCAL VOC -http://host.robots.ox.ac.uk/pascal/VOC/COCO -http://cocodataset.org/YOLO9000 -https://arxiv.org/abs/1612.08242YOLO4 -https://arxiv.org/abs/2004.10934v1LicenseMIT LicenseAuthorsCodePerfectPlus
Package: datasetcrux
datasetcruxDocumentation will be added over here.0.0.1
blank0.0.2random_string(length)can return random string0.0.3
OS classifiers added0.0.31
python minimum 3.60.0.32
csv_printer, csv_reader, csv_reader_dict
csv_writer, csv_writer_append, csv_writer_append_dict, csv_writer_append_dict_header, csv_writer_append_dict_header_if_not_exist
csv functions  added0.0.33
folder issue solved
Package: dataset-downloader
Dataset DownloaderPreviewDataset_downloader allow you to download large dataset from multiple list of url, fromimage-netfor example.
You can split the download into 2 folders, one for the training and one for the testing.
File are save into their class name, perfect for model training. It looks something like that:root:.
|
â”œâ”€â”€â”€test
â”‚   â”œâ”€â”€â”€accerola
â”‚   â”œâ”€â”€â”€apple
â”‚   â””â”€â”€â”€lemon
â”œâ”€â”€â”€train
â”‚   â”œâ”€â”€â”€accerola
â”‚   â”œâ”€â”€â”€apple
â”‚   â””â”€â”€â”€lemonInstallationSimply install from pip:pip install dataset_downloaderConfigCreate adataset.jsonfile with the following content:{"outputTrain":"...","outputTest":"...","ratio":...,"classes":{"class1":["http://url1","http://url2"],"class2":["http://url1","http://url2"],"class3":"list_images.txt"}}outputTrain: Output folder of the training imagesoutputTest: Output folder of the testing imagesratio: The ratio of training/testing images. 0.8 correspond of 80% of training images.classes: List of classes with their urls. Urls can be a list of url, a file containing a list of urls or an url containing a list of urlsAn exemple of file on a windows computer:"outputTrain":"D:/dataset/train","outputTest":"D:/dataset/test","ratio":0.8,"classes":{"accerola":["http://tiachea.files.wordpress.com/2008/10/acerolas.jpg","http://www.jardimdeflores.com.br/floresefolhas/JPEGS/A56acerola5.JPG","http://farm2.staticflickr.com/1353/4602150961_177e096984_z.jpg",],"apple":["http://www.naturalhealth365.com/images/apple.jpg","http://urbanext.illinois.edu/fruit/images/apple1.jpg","https://www.aroma-zone.com/cms//sites/default/files/plante-acerola.jpg"],"lemon":"list_images.txt","watermelon":"https://gist.githubusercontent.com/johnrazeur/645787bc08a5aedd82da9573fbfa169a/raw/49cea1ee1438cecef8ac213b20f24e5ae02d4d78/watermelon.txt"}RunSimple call the dataset_downloader command:cdyourdirectory# You must create the dataset.json file beforedataset_downloader
Package: dataset-fetcher
Welcome to DataSet-Fetcher documentation!Introduction:This package has one mission: to make fetching diverse dataset stats a breeze, perfect for insightful Analytics.QuickStart:To start with dataset_fetcher run:pipinstalldataset_fetcherUsage:This package is specifically designed for fetching datasets from public GitHub repositories, which are shared within my own project's dataset purposes. Users of this package will no longer need to manually copy and paste datasets from various sources. In the future, I plan to integrate functionalities such as the Kaggle API, web scraping API, and connections to different databases. Many more functionalities will be added to this package.To Fetch the data, use the datafetcher class as below:importDataFetcher# Fetch a dataset from the list of available datasets.print(DataFetcher.dataset())importDataFetcher# Fetch a dataset by specifying its name# print(DataFetcher.dataset('datset_name'))dataframe=DataFetcher.dataset('AustralianElections')print(dataframe)Check out my YouTube channel for more innovative projects: https://www.youtube.com/@dataisenoughReach out to us for any queries atdataisenough@gmail.comThank you for choosing dataset_fetcher!
Package: dataset-format-benchmark
This package runs different image format benchmarks for dataset ML tasksInstallationMake sure you have some system deps installed:sudoaptinstallpkg-configlibhdf5-devpython3.11-mvenvvenv--upgrade-depssourcevenv/bin/activatepython-mpipinstall-U-rrequirements_dev.txt# For running on Nvidia GPU:python-mpipinstall-Utorchtorchvision# For running on CPU:python-mpipinstall-Utorchtorchvisiontorchaudio--extra-index-urlhttps://download.pytorch.org/whl/cpu# For some reason h5py fails to install Cython while it needs itpython-mpipinstall-UCythonpythonsetup.pydevelopRunning dataset format benchmarkpython-mdataset_format_benchmark--data-root/path/to/datasets/
Package: dataset-grouper
Dataset GrouperDataset Grouper is a library for creating, writing, and iterating over datasets
with group-level structure. It is primarily intended for creating datasets useful
for federated learning research, and prioritizes flexibility and scalability.DisclaimersThis is a utility library that downloads and prepares public datasets. We do
not host or distribute these datasets, vouch for their quality or fairness, or
claim that you have license to use the dataset. It is your responsibility to
determine whether you have permission to use the dataset under the dataset's
license.If you're interested in learning more about responsible AI practices, please
see Google AI'sResponsible AI Practices.Dataset Grouper is Apache 2.0 licensed. See theLICENSEfile.This is not an officially supported Google product.
Package: dataset-iterator
Dataset IteratorThis repo contains keras iterator classes for multi-channel (time-lapse) images contained in dataset files such as hdf5.Dataset structure:One dataset file can contain several sub-datasets (dataset_name0, dataset_name1, etc...), the iterator will iterate through all of them as if they were concatenated..
├── ...
├── dataset_name0                    
│   ├── channel0          
│   └── channel1   
│   └── ...
├── dataset_name1                    
│   ├── channel0          
│   └── channel1   
│   └── ...
└── ...Each dataset contain channels (channel0, channel1 ...) that must have same shape. All datasets must have the same number of channels, and shape (except batch size) must be equal among datasets.GroupsThere can be more folder level, for instance to have train and test sets in the same file:.
├── ...
├── experiment1                    
│   ├── train          
│   │   ├── raw
│   │   └── labels
│   └── test   
│       ├── raw
│       └── labels
├── experiment2                    
│   ├── train          
│   │   ├── raw
│   │   └── labels
│   └── test   
│       ├── raw
│       └── labels
└── ...train_it=MultiChannelIterator(dataset_file_path=file_path,channel_keywords=["/raw","/labels"],group_keyword="train")test_it=MultiChannelIterator(dataset_file_path=file_path,channel_keywords=["/raw","/labels"],group_keyword="test")Image formatsThose iterators are using an object of classDatasetIOto access the data.There is currently an implementation of DatasetIO for .h5 files (H5pyIO), as well as dataset composed of multiple images files supported by PILLOW (MultipleFileIO).one can also concatenate datasets from different files:if a dataset is split into several files that contain the same channels: useConcatenateDatasetIOif a dataset contains channels in different files, use:MultipleDatasetIODemoSee this notebook for a demo:
Package: dataset-librarian
Dataset LibrarianInstallation Processpython-mpipinstalldataset-librarianFor more information check thedataset-librarian PyPI packageDatasetsDataset nameDescriptionDownloadPreprocessingcommandbrcaBreast Cancer datasetthat contains categorized contrast enhanced mammography data and radiologists’ notes.supportedA prerequisite:Use a browser, downloadthe Low Energy and Subtracted images, then provide the path to the directory that contains the downloaded images using--directoryargument.python -m dataset_librarian.dataset -n brca --download --preprocess -d <path to the dataset directory> --split_ratio 0.1tabformerCredit card datafor TabFormersupportednot supportedpython -m dataset_librarian.dataset -n tabformer --downloaddureader-visDuReader-visfor document automation. Chinese Open-domain Document Visual Question Answering (Open-Domain DocVQA) dataset, containing about 15K question-answering pairs and 158K document images from the Baidu search engine.supportednot supportedpython -m dataset_librarian.dataset -n dureader-vis --downloadmsmarcoMS MARCOis a collection of datasets focused on deep learning in searchsupportednot supportedpython -m dataset_librarian.dataset -n msmarco --downloadmvtec-adMVTEC Anomaly Detection DATASETfor industrial inspection. It contains over 5000 high-resolution images divided into fifteen different object and texture categories.supportedsupportedpython -m dataset_librarian.dataset -n mvtec-ad --download --preprocess -d <path to the dataset directory>Command-line InterfaceInput ArgumentsDescription--list (-l)list the supported datasets.--name (-n)dataset name--directory (-d)directory location where the raw dataset will be saved on your system. It's also where the preprocessed dataset files will be written. If not set, a directory with the dataset name will be created.--downloaddownload the dataset specified.--preprocesspreprocess the dataset if supported.--split_ratiosplit ratio of the test data, the default value is 0.1.Python APIfrom dataset_librarian.dataset_api.download import download_dataset
from dataset_librarian.dataset_api.preprocess import preprocess_dataset

# Download the datasets
download_dataset('brca', <path to the raw dataset directory>)

# Preprocess the datasets
preprocess_dataset('brca', <path to the raw dataset directory>)Building for sourceClone theModel Zoo for Intel® Architecturerepository and navigate to thedataset_apidirectory.gitclonehttps://github.com/IntelAI/models.gitcdmodels/datasets/dataset_api
python-mpipinstall--upgradepipbuildsetuptoolswheel
python-mpipinstall.
Package: dataset-manager
Dataset ManagerManage and automatize your datasets for your project with YAML files.Current Support:How it WorksThis project create a file calledidentifier.yamlin your dataset directory with these fields:source: https://raw.githubusercontent.com/pcsanwald/kaggle-titanic/master/train.csv

description: this dataset is a test datasetidentifier: is the identifier for dataset reference is the file name withyamlextension.source: is location from dataset.description: describe your dataset to remember later.Each dataset is a YAML file inside dataset directory.InstallingWith pip just:pip install dataset_managerWith conda:conda install dataset_managerUsingYou can manage your datasets with a list of commands and integrate withPandasor other data analysis tool.Manager functionsShow all DatasetsReturn a table with all Datasets from dataset pathfrom dataset_manager import DatasetManager

manager = DatasetManager(dataset_path, local_path_to_download)

manager.show_datasets()Create a DatasetCreate a Dataset with every information you want inside dataset_path defined.from dataset_manager import DatasetManager

manager = DatasetManager(dataset_path, local_path_to_download)

manager.create_dataset(identifier, source, description, **kwargs)Remove a DatasetRemove Dataset from dataset_pathfrom dataset_manager import DatasetManager

manager = DatasetManager(dataset_path, local_path_to_download)

manager.remove_dataset(identifier)Prepare DatasetsDownload and Unzip all Datasetsfrom dataset_manager import DatasetManager

manager = DatasetManager(dataset_path, local_path_to_download)

manager.prepare_datasets()Using Multiple FilesystemsThis manager is integrated withPyfilesystem2and you can use allbuiltinfilesystems or withthird-partyextensions orcreating your ownextension.With Pyfilesystem2, you can download, extract and manage datasets in any place.from fs.tempfs import TempFS
from dataset_manager import DatasetManager

manager = DatasetManager(dataset_path, local_path_to_download, TempFS())

manager.prepare_datasets() # all datasets will be downloaded and extracted on temporary files respecting your local_path_to_download hierarchyGet one DatasetGet Dataset line as dictimport pandas as pd
from dataset_manager import DatasetManager

manager = DatasetManager(dataset_path, local_path_to_download)

dataset = manager.get_dataset(identifier)

df = pd.read_csv(dataset.uri)Dataset functionsDownload DatasetDownload Dataset based on source. This only download once because validates cache.
It works both with HTTP, HTTPS and FTP protocols.dataset = manager.get_dataset(identifier)

dataset.download()Unzip DatasetUnzip Dataset based on dataset uri. It works with zip files and others from supported library:fs.archivedataset = manager.get_dataset(identifier)

dataset.unzip()Prepare DatasetPrepare Dataset combine these two before.dataset = manager.get_dataset(identifier)

dataset.prepare()ContributingJust make pull request and be happy!Let's grow together ;)
Package: dataset-manager-py
dataset-manager-pyPython wrapper over the HuggingFace datasets library that makes it easier to load and convert datasets.# Import the DatasetManager classfromdataset.managerimportDatasetManager# Instantiate a new DataManager objectmanager=DatasetManager()# Download a dataset from the HuggingFace Hubdataset=manager.load_from_hub(dataset_name="cuad")# Calling dataset will print out the top-level detail about the datasetdatasetDatasetDict({train:Dataset({features:['id','title','context','question','answers'],num_rows:22450})test:Dataset({features:['id','title','context','question','answers'],num_rows:4182})})# You can also save the dataset to diskmanager.save_to_disk(path="cuad-dataset")# And reload the dataset from diskreloaded_dataset=manager.load_from_disk(path="cuad-dataset")# It's also possible to compress the dataset into either a zip file or a tarball# Defaults to the 'zip' formatmanager.archive_dataset(dataset_dir="cuad-dataset",archive_path=".",archive_format="zip")
Package: dataset-orm
Fast Active Record ORM for the dataset libraryWhy?The dataset library is a great and easy tool to work with any SQL database. Unfortunately, it lacks an object mapper (
ORM) - if you need one you are left with the complexity that is sqlalchemy.Enter dataset-ormHow?Installation$ pip install dataset-ormDefine classes that define a dataset.Table:from dataset_orm import Model, connect

class User(Model):
    username = Column(types.string, unique=True)
    data = Column(types.json)

connect('sqlite:///mydb.sqlite')Alternatively, use the functional API, e.g. to create models dynamically:User = ds.Model.from_spec(name='User',
                          columns=[ds.Column(ds.types.string, 'name', unique=True),
                                   ds.Column(ds.types.json, 'data')],
                          db=db)Then create rows directly from Python objects:user = User(username='dave', data={'sports': ['football', 'tennis']'})
user.save()

user = User.objects.find_one(username='dave')
user.data
=> 
{'sports': ['football', 'tennis']'}Query exiting tables, ORM-style:User = Model.from_table(db['customer'])
User.objects.all()
=>
[ User(pk=1), User(pk=2), User(pk=3)]

user = User.objects.find_one(name='John Walker')
print(user.pk, user.name)
=> 
1 John WalkerUpdate and deleteuser = User.objects.find_one(name='John Walker')
user.place = 'New York'
user.save()

users = User.objects.find(place='London')
users.delete()Store and access any data types, including json and binary valuesclass User(Model):
    # in some dbs, unique strings must be limited in length
    username = Column(types.string(length=100), unique=True)
    picture = Column(types.binary)

user = User.objects.get(name='Dave')
with open('image.png', 'rb') as fimg:
    user.picture = fimg.read()  
    user.save()Use the file column type for transparently storing binary data:class Image(Model):
    imagefile = Column(types.file)Usage:img = Image()
with open('/path/to/image') as f:
   img.imagefile.write(f)
img.save()
data = img.imagefile.read()Here the imagefile field provides a file-like API. This is an efficient way to store binary data in the database. The
file's data is split in chunks and written to the database in multiple parts. On reading back, the chunks are retrieved
from the db in parallel, in order to improve performance for large files. Tests indicate a 25% speed up is possible v.v.
a binary field.You may use thedataset.filesAPI to get a filesystem-like API to binary data stored in the database, without the need
to use a model:from dataset_orm import files

connect('sqlite:///test.sqlite')

files.write('myfile', b'some data')
files.read('myfile')
=> b'some data'

files.exists('myfile') 
=> True

files.list()
=> ['myfile']

files.find('*file*')
=> ['myfile']

files.remove('myfile')The convenience methodsput()andget()allow for an even simpler use of the files api:files.put(b'some data', 'myfile')
data = files.get('myfile').read()
=> b'some data'
Package: datasets2
datasets2provides add-ons to the huggingfacedatasets.Example usage:# datasets is just the huggingface datasets
# load_dataset and save_to_disk adds parquet support to datasets.load_dataset and datasets.Dataset.save_to_disk
from datasets2 import datasets, load_dataset, save_to_disk

my_dataset = prepare_my_huggingface_dataset()
output_dir = "my_dataset_dir"
save_to_disk(my_dataset, output_dir, parquet=True)

load_dataset(output_dir)  # automatically infers if the dataset uses parquet format.
Package: datasetscraper
DatasetScraperTool to create image datasets for machine learning problems by scraping search engines like Google, Bing and Baidu.Features:Search engine support: Google, Bing, Baidu. (in-production): Yahoo, Yandex, DuckduckgoImage format support: jpg, png, svg, gif, jpegFast multiprocessing enabled scraperVery fast multithreaded downloaderData verification after download for assertion of image filesInstallationCOMING SOON on pypiUsage:Importfrom datasetscraper import ScraperDefaultsobj=Scraper()urls=obj.fetch_urls('kiniro mosaic')obj.download(urls,directory='kiniro_mosaic/')Specify a search engineobj=Scraper()urls=obj.fetch_urls('kiniro mosaic',engine=['google'])obj.download(urls,directory='kiniro_mosaic/')Specify a list of search enginesobj=Scraper()urls=obj.fetch_urls('kiniro mosaic',engine=['google','bing'])obj.download(urls,directory='kiniro_mosaic/')Specify max images (default was 200)obj=Scraper()urls=obj.fetch_urls('kiniro mosaic',engine=['google','bing'],maxlist=[500,300])obj.download(urls,directory='kiniro_mosaic/')FAQsWhy aren't yandex, yahoo, duckduckgo and other search engines supported?
They are hard to scrape, I am working on them and will update as soon as I can.I set maxlist=[500] why are only (x<500) images downloaded?
There can be several reasons for this:Search ran out: This happens very often, google/bing might not have enough images for your querySlow internet: Increase the timeout (default is 60 seconds) as follows:obj.download(urls, directory='kiniro_mosaic/', timeout=100)How to debug?
You can change the logging level while making the scraper object :obj = Scraper(logger.INFO)TODO:More search enginesBetter debugWrite documentationText data? Audio data?
Package: datasetsDynamic
datasetsDynamicInstallpipinstalldatasetsDynamicHow to useFor every dataset a load function is implemented which computes training
and test data for the corresponding dataset including all preprocessing
and basic feature engineering steps. For most datasets the test period
can be chosen dynamically using the parametertestDays. While doing
so, it is ensured that all features that depend on the train and test
structure are computed only based on the training data.fromdatasetsDynamic.loadDataYazimportloadDataYazdata,XTrain,yTrain,XTest,yTest=loadDataYaz(testDays=28,returnXY=True,daysToCut=0,disable_progressbar=False)Rolling: 100%|██████████| 30/30 [00:00<00:00, 36.35it/s]
Feature Extraction: 100%|██████████| 30/30 [00:02<00:00, 13.59it/s]
Rolling: 100%|██████████| 30/30 [00:00<00:00, 35.29it/s]
Feature Extraction: 100%|██████████| 30/30 [00:02<00:00, 12.19it/s]
Rolling: 100%|██████████| 30/30 [00:00<00:00, 37.20it/s]
Feature Extraction: 100%|██████████| 30/30 [00:02<00:00, 14.39it/s]fromdatasetsDynamic.loadDataBakeryimportloadDataBakerydata,XTrain,yTrain,XTest,yTest=loadDataBakery(testDays=28,returnXY=True,daysToCut=0,disable_progressbar=False)Rolling: 100%|██████████| 152/152 [00:11<00:00, 13.25it/s]
Feature Extraction: 100%|██████████| 160/160 [00:43<00:00,  3.70it/s]
Rolling: 100%|██████████| 152/152 [00:12<00:00, 11.84it/s]
Feature Extraction: 100%|██████████| 160/160 [00:44<00:00,  3.59it/s]
Rolling: 100%|██████████| 152/152 [00:11<00:00, 13.53it/s]
Feature Extraction: 100%|██████████| 160/160 [00:44<00:00,  3.57it/s]
Package: dataset.sh
Getting Starteddataset.shis a dataset manager designed to simplify the process of installing, managing, and publishing datasets.
We hope to make working with datasets as straightforward as using package managers like npm or pip for programming
libraries.QuickstartHost your own server (Optional)dockerrun-v./:/var/lib/dsh-serverdatasetsh/server:latestdataset-sh-server-cliinitdockerrun-v./:/var/lib/dsh-serverdatasetsh/server:latest-d