Package: ml-datasets
Machine learning dataset loaders for testing and examplesLoaders for various machine learning datasets for testing and example scripts.
Previously inthinc.extra.datasets.Setup and installationThe package can be installed via pip:pipinstallml-datasetsLoadersLoaders can be imported directly or used via their string name (which is useful if they're set via command line arguments). Some loaders may take arguments – see the source for details.# Import directlyfromml_datasetsimportimdbtrain_data,dev_data=imdb()# Load via registryfromml_datasetsimportloadersimdb_loader=loaders.get("imdb")train_data,dev_data=imdb_loader()Available loadersNLP datasetsID / FunctionDescriptionNLP taskFrom URLimdbIMDB sentiment datasetBinary classification: sentiment analysis✓dbpediaDBPedia ontology datasetMulti-class single-label classification✓cmuCMU movie genres datasetMulti-class, multi-label classification✓quora_questionsDuplicate Quora questions datasetDetecting duplicate questions✓reutersReuters dataset (texts not included)Multi-class multi-label classification✓snliStanford Natural Language Inference corpusRecognizing textual entailment✓stack_exchangeStack Exchange datasetQuestion Answeringud_ancora_pos_tagsUniversal Dependencies Spanish AnCora corpusPOS tagging✓ud_ewtb_pos_tagsUniversal Dependencies English EWT corpusPOS tagging✓wikinerWikiNER dataNamed entity recognitionOther ML datasetsID / FunctionDescriptionML taskFrom URLmnistMNIST dataImage recognition✓Dataset detailsIMDBEach instance contains the text of a movie review, and a sentiment expressed as0or1.train_data,dev_data=ml_datasets.imdb()fortext,annotintrain_data[0:5]:print(f"Review:{text}")print(f"Sentiment:{annot}")Download URL:http://ai.stanford.edu/~amaas/data/sentiment/Citation:Andrew L. Maas et al., 2011PropertyTrainingDev# Instances2500025000Label values{0,1}{0,1}Labels per instanceSingleSingleLabel distributionBalanced (50/50)Balanced (50/50)DBPediaEach instance contains an ontological description, and a classification into one of the 14 distinct labels.train_data,dev_data=ml_datasets.dbpedia()fortext,annotintrain_data[0:5]:print(f"Text:{text}")print(f"Category:{annot}")Download URL:Via fast.aiOriginal citation:Xiang Zhang et al., 2015PropertyTrainingDev# Instances56000070000Label values1-141-14Labels per instanceSingleSingleLabel distributionBalancedBalancedCMUEach instance contains a movie description, and a classification into a list of appropriate genres.train_data,dev_data=ml_datasets.cmu()fortext,annotintrain_data[0:5]:print(f"Text:{text}")print(f"Genres:{annot}")Download URL:http://www.cs.cmu.edu/~ark/personas/Original citation:David Bamman et al., 2013PropertyTrainingDev# Instances417930Label values363 different genres-Labels per instanceMultiple-Label distributionImbalanced: 147 labels with less than 20 examples, whileDramaoccurs more than 19000 times-Quoratrain_data,dev_data=ml_datasets.quora_questions()forquestions,annotintrain_data[0:50]:q1,q2=questionsprint(f"Question 1:{q1}")print(f"Question 2:{q2}")print(f"Similarity:{annot}")Each instance contains two quora questions, and a label indicating whether or not they are duplicates (0: no,1: yes).
The ground-truth labels contain some amount of noise: they are not guaranteed to be perfect.Download URL:http://qim.fs.quoracdn.net/quora_duplicate_questions.tsvOriginal citation:Kornél Csernai et al., 2017PropertyTrainingDev# Instances36385940429Label values{0,1}{0,1}Labels per instanceSingleSingleLabel distributionImbalanced: 63% label0Imbalanced: 63% label0Registering loadersLoaders can be registered externally using theloadersregistry as a decorator. For example:@ml_datasets.loaders("my_custom_loader")defmy_custom_loader():returnload_some_data()assert"my_custom_loader"inml_datasets.loaders