Package: datati
Datati: Modern (tabular) datasets require modern solutionsDataset to model, in one go!datatiis a small library to streamline tabular dataset loading and preprocessing.
The goal of this library is to minimize the boring boilerplate code that separates choosing a dataset to work on,
and actually getting it ready to train a classification model .datatiprovides simple interfaces to load, preprocess, and encode a dataset for training your model of choice:fromdatati.datasetimportDatasetfromdatati.models.treesimportContinuousTreeModeler# load datasetdataset=Dataset("mstz/adult",config="income",split="train",load_from="huggingface",target_feature="over_threshold")tree_dataset=ContinuousTreeModeler().process(dataset).to_array()x,y=tree_dataset[:,:-1],tree_dataset[:,-1]This snippet allows us to load a dataset (Dataset("mstz/adult")) in a desired configuration
(config="income") and split (split="train"") of choice.
Then we use aModelerobject to map the initialDatasetinto an encoding suitable for Decision Tree induction
(modeler.process(dataset)).datatibuilds on top of the huggingface hub, providing an interface to integrate it with common preprocessing
pipelines.Quickstartpip installWhat datasets are available?datatiallows you to load huggingface (load_from="huggingface""), or local (load_from="local"") datasets,
whether they arenumpy.arrays,pandas.DataFrames, orpyarrow.ArrowDatasets.What can I do with a dataset?Most operations have no side-effects, that is, they yield anewDatasetobject, rather than modifying the
existing one.
Extendingpandas.DataFrame, all operations supported on apandas.DataFrameare also supported onDatasetinstances.
Methods yielding apandas.DataFramehave been overwritten to yield aDatasetinstead.DundersDatasetimplements most dunder methods.
A datasetdcan be both copied (copy.copy(d)) and deepcopied (copy.deepcopy(d)), it can be checked for equality,
and hashed (hash(d)).Conversion to/from other formatsDatasets can be directly exported to:pandas.DataFrame(dataset.to_pandas())numpy.array(dataset.to_array())list(dataset.to_list())Model-specific encodingNOTEAs of nowdatatiis aimed exclusively at single-output tabular classifiers, hence string/object features
are treated as categorical.TheModelerclass (datati.models.Modeler) implements a minimal interface to map a dataset for processing for the
algorithm of choice.
Currently,datatiimplements:AlgorithmInfo`ContinuousTreeModelerDecision treeCategorical features are encoded through target encoding.OneHotTreeModelerDecision treeCategorical features are encoded through one-hot encoding.SBRLModelerSBRLAll features are binned, then binarized.CorelsModelerCORELSAll features are binned, then binarized.All implementedModelers leave a trace of their own transformations by enriching the transformedDatasetwith
transformation-specific mappings:frompprintimportpprintdataset=Dataset("mstz/adult",config="income",split="train",load_from="huggingface")# preprocess dataset for decision tree classificationdataset.target_feature="over_threshold"modeler=ContinuousTreeModeler()tree_dataset=modeler.process(dataset)pprint(tree_dataset.bins_encoding_dictionaries)# {}pprint(tree_dataset.one_hot_encoding_dictionaries)# {}pprint(tree_dataset.target_encoding_dictionaries)#{'marital_status': {'Divorced': array([0.12109962]),#                    'Married-AF-spouse': array([0.00057328]),#                    'Married-civ-spouse': array([0.25382872]),#                    'Married-spouse-absent': array([0.01165679]),#                    'Never-married': array([0.3155797]),#                    'Separated': array([0.02942863]),#                    'Widowed': array([0.02855505])},# 'native_country': {'?': array([0.01321285]),#                    'Cambodia': array([0.00035489]),#                    'Canada': array([0.00232044]),#                    'China': array([0.00174715]),#                    'Columbia': array([0.00185635]),#                    'Cuba': array([0.00204745]),# ...Similarly, when applying one-hot encoding,dataset.one_hot_encoding_dictionarywill hold the encoding indexes:frompprintimportpprintdataset=Dataset("mstz/adult",config="income",split="train",load_from="huggingface")# preprocess dataset for decision tree classificationdataset.target_feature="over_threshold"modeler=ContinuousTreeModeler()tree_dataset=modeler.process(dataset)pprint(tree_dataset.one_hot_encoding_dictionary)# {'marital_status': {'Divorced': 0,#                     'Married-AF-spouse': 1,#                     'Married-civ-spouse': 2,#                     'Married-spouse-absent': 3,#                     'Never-married': 4,#                     'Separated': 5,#                     'Widowed': 6},#  'native_country': {'?': 0,#                     'Cambodia': 1,#                     'Canada': 2,#                     'China': 3,#                     'Columbia': 4,#                     'Cuba': 5,# ...Encoding LegosSome lower-level modelers can be composed as small building blocks to provide the desired result.
Currently,datatiimplements:InfoTargetModelerCategorical features are encoded through target encoding.OneHotModelerCategorical features are encoded through one-hot encoding.BinModelerNumerical features are discretized into bins.BinaryModelerAllfeatures are first binned, then each bin is transformed into a boolean feature.BoolToIntModelerBooleans are mapped to{0, 1}.IntToBoolModelerIntegers are mapped to booleans.SBRLModelerAll features are binned, then binarized.CorelsModelerAll features are binned, then binarized.Similarly toscikit-learnpipelines, you can implement your ownModelerby combining existing modelers through a
pipeline, as it is done in theContinuousTreeModeler.
Here, we first target-encode categorical variables, then transform booleans into integers:classContinuousTreeModeler(NumericModeler):"""Model data as continuous, mapping boolean features to 0/1, and categorical features to target encoders."""def__init__(self):super().__init__()self.pipeline=Pipeline(TargetModeler(),BoolToIntModeler(guess_booleans=True))defprocess(self,dataset:Dataset,**kwargs)->Dataset:"""Adapt the given `dataset` to be fed to the model of choice.Args:dataset: The dataset to process.**kwargs: Keyword arguments.Returns:The processed dataset."""returnself.pipeline(dataset,**kwargs)Run on your own (local) datasetLocal or remote doesn't matter,datatican be integrated to work on your own local dataset.
All it takes, is to specify that we're working on a local (local=True) dataset:fromdatati.datasetimportDatasetdataset=Dataset("./adult",load_from="local")