Package: data-iq
Data-IQ: Characterizing subgroups with heterogeneous outcomes in tabular dataThis repository contains the implementation of Data-IQ, a "Data-Centric AI" framework to characterize subgroups with heterogeneous outcomes in tabular data.Data-IQ studies training dynamics and specifically studies the inherent data uncertainty (aleatoric uncertainty), to characterize data examples into the following subgroups:EASY,AMBIGUOUSandHARD.Data-IQ can be used to characterize data usinganymachine learning model trained iteratively. We provide an interface for using Data-IQ, which can be used withPytorch models, as well as,SKLearn style API modelsincludingXGBoost, LightGBM and CatBoost.The utility of the subgroups extend to a variety of use-cases.For more details, please read our [NeurIPS 2022 paper](Link coming soon):Data-IQ: Characterizing subgroups with heterogeneous outcomes in tabular data.InstallationClone the repositoryCreate a new virtual environment with Python 3.7. e.g:virtualenvdataiq_envWith the venv activated, run the following command from the repository directory:Minimum requirements to run Data-IQ on your own datapipinstalldata_iqor from source,pipinstall.Full requirements to run Data-IQ tests and tutorialspipinstall.[testing]Link the venv to the kernel:python-mipykernelinstall--user--name=dataiq_envUse Data-IQ in your projects with 2-lines of code to your training loops!Getting Data-IQOption 1: Install as a package (calleddata_iq) from PyPI usingpipinstalldata_iqor from source usingpython-mpipinstall-e.Option 2: Import from thedata_iqfolderTwo examples using a neural network (Pytorch-style) and XGBoost (Sklearn-style) is shown below.The lines of Data-IQ code that need to be added to your training loops are highlighted with stars in the comments. e.g.*** comment ***1. Data-IQ Example with Pytorch interface (NEURAL NETWORK)# Import Data-IQ# Option 1 - if package is installedfromdata_iqimportDataIQ_Torch# Option 2 - import from folderfromdata_iq.dataiq_classimportDataIQ_Torch# Pytorch data loader heretrain_loader=DataLoader(dataset=train_data,batch_size=128,shuffle=True)# Define Pytorch devicedevice=torch.device("cuda:0"iftorch.cuda.is_available()else"cpu")# Define Pytorch modelnet=Example_NN(input_size=X_train.shape[1],nlabels=len(np.unique(y_train)))net.to(device)# *** Initialize Data-IQ [LINE 1] ***dataiq=DataIQ_Torch(X=X_train,y=y_train,sparse_labels=True)# Fit Pytorch model [Training loop]foreinrange(1,EPOCHS+1):net.train()forX_batch,y_batchintrain_loader:### ADD TRAIN LOOP HERE #### *** CALL DATA-IQ on EPOCH END [LINE 2] ***dataiq.on_epoch_end(net,device=device)# *** Access metrics ***aleatoric_uncertainty=dataiq_xgb.aleatoricconfidence=dataiq_xgb.confidence2. Data-IQ Example with Sklearn style interface (XGBOOST)# Import Data-IQ# Option 1 - if package is installedfromdata_iqimportDataIQ_SKLearn# Option 2 - import from folderfromdata_iq.dataiq_classimportDataIQ_SKLearn# Arbitrary data loader - numpy arraysX_train,X_test,y_train,y_test=load_data()# *** Initialize Data-IQ [LINE 1] ***dataiq_xgb=DataIQ_SKLearn(X=X_train,y=y_train)# Fit XGBoostclf=xgb.XGBClassifier(n_estimators=10)clf.fit(X_train,y_train)foriinrange(1,nest):# *** Characterize with Data-IQ [LINE 2] ***dataiq_xgb.on_epoch_end(clf=clf,iteration=i)# *** Access metrics ***aleatoric_uncertainty=dataiq_xgb.aleatoricconfidence=dataiq_xgb.confidenceGetting startedTo get started with Data-IQ on your own data, we provide two tutorial notebooks to illustrate the usage of Data-IQ. Examples are provided both Pytorch style and SKLearn style (XGBoost, Catboost, LightGBM) models.These notebooks can be found in the/tutorialfolder. The Adult open-source dataset is given as an example dataset in the tutorials for ease of accessbility.tutorial_torch_api.ipynbExample integration of Data-IQ with Pytorch modelstutorial_sklearn_api.ipynbExample integration of Data-IQ with SKLearn style models trained iteratively (e.g. XGBoost, LigthGBM, CatBoost)Data-IQ metricsThe Data-IQ package provides a wide variety of metrics to characterize training dynamics.The primary & suggested metrics are:Aleatoric uncertaintyandPredictive confidence. However, other metrics are also included as part of the package.The different metrics available to characterize training dynamics are as follows and can be accessed through thedataiqobject:Aleatoric uncertainty via:dataiq.aleatoricPredictive confidence via:dataiq.confidenceVariability via:dataiq.variabilityEntropy via:dataiq.entropyMutual information via:dataiq.miCorrectness over training via:dataiq.correctnessUse-casesWe highlight different use-cases of Data-IQ from understanding learning dynamics, creating characteristic curves, feature acquisition etc as well as, different data modalities in notebooks which can be found in the/use_casesfolder.CitingIf you use this code, please cite the associated paper:@inproceedings
{seedat2022dataiq,
title={Data-IQ: Characterizing subgroups with heterogeneous outcomes in tabular data},
author={Seedat, Nabeel and Crabbe, Jonathan and Bica, Ioana and van der Schaar, Mihaela},
booktitle={Advances in Neural Information Processing Systems},
year={2022}
}