Package: setvaluedprediction
Uncertainty-aware classification with set-valued predictionsPackage for set-valued prediction in flat and hierarchical classification.DescriptionThis package provides different set-valued predictors for flat and hierarchical classification with support for Scikit-learn and PyTorch.TODO: support for multi-label classification.InstallationClone this repositorytfmortie/setvaluedpredictionand runpip install . -r requirements.txtor install by means ofpip install setvaluedprediction.ExamplesFor multi-class classification, we provide the following set-valued predictors:svp.multiclass.SVPClassifier: follows the Scikit-learn APIsvp.multiclass.SVPNet: follows the PyTorch APISome minimal examples are given below.svp.multiclass.SVPClassifierWe start by importing some packages that we will need throughout the example:fromsklearn.linear_modelimportSGDClassifierfromsklearn.model_selectionimporttrain_test_splitfromsklearn.datasetsimportload_digitsCreating a flat and hierarchical set-valued predictor in Scikit-learn is as simple as:fromsvp.multiclassimportSVPClassifierest=SGDClassifier(loss="log_loss")# classifier used for flat and hierarchical model# create two set-valued predictorsflat=SVPClassifier(est,hierarchy="none")hier_r=SVPClassifier(est,hierarchy="random")With argumenthierarchy="random", we specify that no predefined hierarchical labels are going to be provided. In this case,SVPClassifierautomatically constructs a random hierarchy. The min and max degree of each node in the randomly generated tree can be controlled by means of the argumentk:# predictor with randomly generated binary tree as hierarchyhier_r=SVPClassifier(est,hierarchy="random",k=(2,2),random_state=2022)Next, we load a non-hierarchical dataset provided from Scikit-learn and split in a training and validation set:# our datasetX,y=load_digits(return_X_y=True)X_tr,X_te,y_tr,y_te=train_test_split(X,y,test_size=0.5,random_state=2022,stratify=y)SVPClassifierfollows the Scikit-learn API, with support for standard training and inference procedures:# training the modelsflat.fit(X_tr,y_tr)hier_r.fit(X_tr,y_tr)# obtain predictions and class probabilitiesflat_preds=flat.predict(X_te)hier_r_preds=hier_r.predict(X_te)flat_probs=flat.predict_proba(X_te)hier_r_probs=hier_r.predict_proba(X_te)Hence,SVPClassifierboils down to a standard Scikit-learn estimator, albeit with additional support for set-valued predictions:# initialize the set-valued predictor settingsparams_flat={"c":10,# our representation complexity"svptype":"errorctrl",# minimize set size, while controlling the error rate"error":0.01# upper bound the error rate by 1%}params_hier_r={"c":1,# our representation complexity -> in this case only internal nodes are allowed"svptype":"errorctrl",# minimize set size, while controlling the error rate"error":0.01# upper bound the error rate by 1%}# obtain set-valued predictionssvp_preds_flat=flat.predict_set(X_te,params_flat)svp_preds_hier_r=hier_r.predict_set(X_te,params_hier_r)For more information related to the different set-valued prediction settings, see references below.svp.multiclass.SVPNetCreating a set-valued predictor in PyTorch is very similar toSVPClassifier:importtorchimporttorch.nnasnnfromtorch.utils.dataimportTensorDataset,DataLoaderfromsklearn.model_selectionimporttrain_test_splitfromsklearn.datasetsimportload_digitsfromsvp.multiclassimportSVPNet# first load data and get training and validation setsX,y=load_digits(return_X_y=True)X_tr,X_te,y_tr,y_te=train_test_split(X,y,test_size=0.5,random_state=2021,stratify=y)tensor_x_tr,tensor_y_tr=torch.Tensor(X_tr),torch.Tensor(y_tr)tensor_x_te,tensor_y_te=torch.Tensor(X_te),torch.Tensor(y_te)dataset=TensorDataset(tensor_x_tr,tensor_y_tr)dataloader=DataLoader(dataset)# create your dataloader# create feature extractor for SGDNet and construct the set-valued predictorsphi=nn.Identity()flat=SVPNet(phi=phi,hidden_size=X.shape[1],classes=y,hierarchy="none")hier_r=SVPNet(phi=phi,hidden_size=X.shape[1],classes=y,hierarchy="random")# start fitting modelsiftorch.cuda.is_available():flat=flat.cuda()hier_r=hier_r.cuda()optim_f=torch.optim.SGD(flat.parameters(),lr=0.01)optim_hr=torch.optim.SGD(hier_r.parameters(),lr=0.01)for_inrange(50):for_,datainenumerate(dataloader,1):inputs,labels=dataiftorch.cuda.is_available():inputs=inputs.cuda()optim_f.zero_grad()optim_hr.zero_grad()loss_f,loss_hr=flat(inputs,labels),hier_r(inputs,labels)loss_f.backward()loss_hr.backward()# obtain top-1 predictionsiftorch.cuda.is_available():tensor_x_te=tensor_x_te.cuda()flat.eval()hier_r.eval()preds_f=flat.predict(tensor_x_te)preds_hr=hier_r.predict(tensor_x_te)# obtain set-valued predictions with error rate control and maximal representation complexityparams={"c":10,"svptype":"sizectrl","error":0.01}svp_preds_f=flat.predict_set(tensor_x_te,params)svp_preds_hr=hier_r.predict_set(tensor_x_te,params)Hierarchical models with predefined hierarchiesIn case you want to work with predefined hierarchies, simply set argumenthierarchy="predefined"and make sure that provided labels are encoded in the following way:# example of two hierarchical labels from a predefined hierarchy
y = ["root;Family1;Genus1;Species1", "root;Family1;Genus1;Species2"]Moreover, labels must be encoded as strings and should correspond to paths in the predefined hierarchy with nodes separated by;.Experiments paper(s)Accompanying code for paperSet-valued prediction in hierarchical classification with constrained representation complexitycan be found in the foldersrc/test/svphc.CitingIf you usesetvaluedpredictionin your work, please use the following citation:@InProceedings{Mortier22SVPHCCRC,title={Set-valued prediction in hierarchical classification with constrained representation complexity},author={Mortier, Thomas and H\"ullermeier, Eyke and Dembczy\'nski, Krzysztof and Waegeman, Willem},booktitle={Proceedings of the Thirty-Eight Conference on Uncertainty in Artificial Intelligence},year={2022},series={Proceedings of Machine Learning Research},publisher={PMLR}}If you need more information, feel free to contact me by thomas(dot)mortier92(at)gmail(dot)com.ReferencesEfficient set-valued prediction in multi-class classification, Mortier et al., Data Mining and Knowledge Discovery (2021)Set-valued classification - overview via a unified framework, Chezhen et al., CoRR abs/2102.12318 (2021)Set-valued prediction in hierarchical classification with constrained representation complexity, Mortier et al., Proceedings of Machine Learning Research (2022)