Package: datalabs
DataLabis a unified platform that allows for NLP researchers to perform a number of data-related tasks in an efficient and easy-to-use manner. In particular, DataLab supports the following functionalities:Data Diagnostics: DataLab allows for analysis and understanding of data to uncover undesirable traits such as hate speech, gender bias, or label imbalance.Operation Standardization: DataLab provides and standardizes a large number of data processing operations, including aggregating, preprocessing, featurizing, editing and prompting operations.Data Search: DataLab provides a semantic dataset search tool to help identify appropriate datasets given a textual description of an idea.Global Analysis: DataLab provides tools to perform global analyses over a variety of datasets.InstallationDataLab can be installed from PyPipipinstall--upgradepip
pipinstalldatalabs
python-mnltk.downloaderomw-1.4# to support more feature calculationor from the source# This is suitable for SDK developerspipinstall--upgradepip
gitclonegit@github.com:ExpressAI/DataLab.gitcdDatalab
pipinstall-e.[dev]python-mnltk.downloaderomw-1.4# to support more feature calculationBy adding[dev], someextra librarieswill be installed, such aspre-commit.Code Quality Check?If you would like to contribute to DataLab, checking the code style and quality before your pull
request is highly recommended. In this project, three types of checks will be expected: (a) black
(2) flake8 (3) isortyou could achieve this in two ways:Manually (suitable for developers using Github Destop)pre-commitinstall
gitinit.
pre-commitrun--all-filesorwherepre-commit run -all-filesis equivalent topre-commitrunblack# (this is also equivalent to python -m black .)pre-commitrunisort# (this is also equivalent to isort .)pre-commitrunflake8# (this is  also equivalent to flake8)Notably,blackandisortcan help us fix code style automatically, whileflake8only
provide hints with us, which means we need to fix these issues raised byflake8.Automatically (suitable for developers using Git CLI)pre-commitinstall
gitinit.
gitcommit-m"your update message"Thegit commitwill automatically activate the commandpre-commit run -all-filesUsing DataLabBelow we give several simple examples to showcase the usage of DataLab:You can also view documentation:Online TutorialAdding Datasets to the SDKAdding a new Task to the SDK# pip install datalabsfromdatalabsimportload_datasetdataset=load_dataset("ag_news")# Preprocessing operationfrompreprocessimport*res=dataset["test"].apply(lower)print(next(res))# Featurizing operationfromfeaturizeimport*res=dataset["test"].apply(get_text_length)# get lengthprint(next(res))res=dataset["test"].apply(get_entities_spacy)# get entityprint(next(res))# Editing/Transformation operationfromeditimport*res=dataset["test"].apply(change_person_name)#  change person nameprint(next(res))# Prompting operationfrompromptimport*res=dataset["test"].apply(template_tc1)print(next(res))# Aggregating operationfromaggregate.text_classificationimport*res=dataset["test"].apply(get_statistics)AcknowledgmentDataLab originated from a fork of the awesomeHuggingface DatasetsandTensorFlow Datasets. We highly thank the Huggingface/TensorFlow Datasets for building this amazing library. More details on the differences between DataLab and them can be found in the section.
We thank Antonis Anastasopoulos for sharing the mapping data between countries and languages, and thank Alissa Ostapenko, Yulia Tsvetkov, Jie Fu, Ziyun Xu, Hiroaki Hayashi, and Zhengfu He for useful discussion and suggestions for the first version.