Package: datasets
🤗 Datasets is a lightweight library providingtwomain features:one-line dataloaders for many public datasets: one-liners to download and pre-process any of themajor public datasets (image datasets, audio datasets, text datasets in 467 languages and dialects, etc.) provided on theHuggingFace Datasets Hub. With a simple command likesquad_dataset = load_dataset("squad"), get any of these datasets ready to use in a dataloader for training/evaluating a ML model (Numpy/Pandas/PyTorch/TensorFlow/JAX),efficient data pre-processing: simple, fast and reproducible data pre-processing for the public datasets as well as your own local datasets in CSV, JSON, text, PNG, JPEG, WAV, MP3, Parquet, etc. With simple commands likeprocessed_dataset = dataset.map(process_example), efficiently prepare the dataset for inspection and ML model evaluation and training.🎓Documentation🔎Find a dataset in the Hub🌟Share a dataset on the Hub🤗 Datasets is designed to let the community easily add and share new datasets.🤗 Datasets has many additional interesting features:Thrive on large datasets: 🤗 Datasets naturally frees the user from RAM memory limitation, all datasets are memory-mapped using an efficient zero-serialization cost backend (Apache Arrow).Smart caching: never wait for your data to process several times.Lightweight and fast with a transparent and pythonic API (multi-processing/caching/memory-mapping).Built-in interoperability with NumPy, pandas, PyTorch, TensorFlow 2 and JAX.Native support for audio and image data.Enable streaming mode to save disk space and start iterating over the dataset immediately.🤗 Datasets originated from a fork of the awesomeTensorFlow Datasetsand the HuggingFace team want to deeply thank the TensorFlow Datasets team for building this amazing library. More details on the differences between 🤗 Datasets andtfdscan be found in the sectionMain differences between 🤗 Datasets andtfds.InstallationWith pip🤗 Datasets can be installed from PyPi and has to be installed in a virtual environment (venv or conda for instance)pipinstalldatasetsWith conda🤗 Datasets can be installed using conda as follows:condainstall-chuggingface-cconda-forgedatasetsFollow the installation pages of TensorFlow and PyTorch to see how to install them with conda.For more details on installation, check the installation page in the documentation:https://huggingface.co/docs/datasets/installationInstallation to use with PyTorch/TensorFlow/pandasIf you plan to use 🤗 Datasets with PyTorch (1.0+), TensorFlow (2.2+) or pandas, you should also install PyTorch, TensorFlow or pandas.For more details on using the library with NumPy, pandas, PyTorch or TensorFlow, check the quick start page in the documentation:https://huggingface.co/docs/datasets/quickstartUsage🤗 Datasets is made to be very simple to use - the API is centered around a single function,datasets.load_dataset(dataset_name, **kwargs), that instantiates a dataset.This library can be used for text/image/audio/etc. datasets. Here is an example to load a text dataset:Here is a quick example:fromdatasetsimportload_dataset# Print all the available datasetsfromhuggingface_hubimportlist_datasetsprint([dataset.idfordatasetinlist_datasets()])# Load a dataset and print the first example in the training setsquad_dataset=load_dataset('squad')print(squad_dataset['train'][0])# Process the dataset - add a column with the length of the context textsdataset_with_length=squad_dataset.map(lambdax:{"length":len(x["context"])})# Process the dataset - tokenize the context texts (using a tokenizer from the 🤗 Transformers library)fromtransformersimportAutoTokenizertokenizer=AutoTokenizer.from_pretrained('bert-base-cased')tokenized_dataset=squad_dataset.map(lambdax:tokenizer(x['context']),batched=True)If your dataset is bigger than your disk or if you don't want to wait to download the data, you can use streaming:# If you want to use the dataset immediately and efficiently stream the data as you iterate over the datasetimage_dataset=load_dataset('cifar100',streaming=True)forexampleinimage_dataset["train"]:breakFor more details on using the library, check the quick start page in the documentation:https://huggingface.co/docs/datasets/quickstartand the specific pages on:Loading a dataset:https://huggingface.co/docs/datasets/loadingWhat's in a Dataset:https://huggingface.co/docs/datasets/accessProcessing data with 🤗 Datasets:https://huggingface.co/docs/datasets/processProcessing audio data:https://huggingface.co/docs/datasets/audio_processProcessing image data:https://huggingface.co/docs/datasets/image_processProcessing text data:https://huggingface.co/docs/datasets/nlp_processStreaming a dataset:https://huggingface.co/docs/datasets/streamWriting your own dataset loading script:https://huggingface.co/docs/datasets/dataset_scriptetc.Add a new dataset to the HubWe have a very detailed step-by-step guide to add a new dataset to thedatasets already provided on theHuggingFace Datasets Hub.You can find:how to upload a dataset to the Hub using your web browser or Pythonand alsohow to upload it using Git.Main differences between 🤗 Datasets andtfdsIf you are familiar with the great TensorFlow Datasets, here are the main differences between 🤗 Datasets andtfds:the scripts in 🤗 Datasets are not provided within the library but are queried, downloaded/cached and dynamically loaded upon requestthe backend serialization of 🤗 Datasets is based onApache Arrowinstead of TF Records and leverage python dataclasses for info and features with some diverging features (we mostly don't do encoding and store the raw data as much as possible in the backend serialization cache).the user-facing dataset object of 🤗 Datasets is not atf.data.Datasetbut a built-in framework-agnostic dataset class with methods inspired by what we like intf.data(like amap()method). It basically wraps a memory-mapped Arrow table cache.Disclaimers🤗 Datasets may run Python code defined by the dataset authors to parse certain data formats or structures. For security reasons, we ask users to:check the dataset scripts they're going to run beforehand andpin therevisionof the repositories they use.If you're a dataset owner and wish to update any part of it (description, citation, license, etc.), or do not want your dataset to be included in the Hugging Face Hub, please get in touch by opening a discussion or a pull request in the Community tab of the dataset page. Thanks for your contribution to the ML community!BibTeXIf you want to cite our 🤗 Datasets library, you can use ourpaper:@inproceedings{lhoest-etal-2021-datasets,title="Datasets: A Community Library for Natural Language Processing",author="Lhoest, Quentin  andVillanova del Moral, Albert  andJernite, Yacine  andThakur, Abhishek  andvon Platen, Patrick  andPatil, Suraj  andChaumond, Julien  andDrame, Mariama  andPlu, Julien  andTunstall, Lewis  andDavison, Joe  and{\v{S}}a{\v{s}}ko, Mario  andChhablani, Gunjan  andMalik, Bhavitvya  andBrandeis, Simon  andLe Scao, Teven  andSanh, Victor  andXu, Canwen  andPatry, Nicolas  andMcMillan-Major, Angelina  andSchmid, Philipp  andGugger, Sylvain  andDelangue, Cl{\'e}ment  andMatussi{\`e}re, Th{\'e}o  andDebut, Lysandre  andBekman, Stas  andCistac, Pierric  andGoehringer, Thibault  andMustar, Victor  andLagunas, Fran{\c{c}}ois  andRush, Alexander  andWolf, Thomas",booktitle="Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",month=nov,year="2021",address="Online and Punta Cana, Dominican Republic",publisher="Association for Computational Linguistics",url="https://aclanthology.org/2021.emnlp-demo.21",pages="175--184",abstract="The scale, variety, and quantity of publicly-available NLP datasets has grown rapidly as researchers propose new tasks, larger models, and novel benchmarks. Datasets is a community library for contemporary NLP designed to support this ecosystem. Datasets aims to standardize end-user interfaces, versioning, and documentation, while providing a lightweight front-end that behaves similarly for small datasets as for internet-scale corpora. The design of the library incorporates a distributed, community-driven approach to adding datasets and documenting usage. After a year of development, the library now includes more than 650 unique datasets, has more than 250 contributors, and has helped support a variety of novel cross-dataset research projects and shared tasks. The library is available at https://github.com/huggingface/datasets.",eprint={2109.02846},archivePrefix={arXiv},primaryClass={cs.CL},}If you need to cite a specific version of our 🤗 Datasets library for reproducibility, you can use the corresponding version Zenodo DOI from thislist.