Package: backbone-learn
OverviewBackboneLearnis an open-source software package and framework for scaling mixed integer optimization (MIO) problems with indicator variables to high-dimensional problems. This optimization paradigm can naturally be used to formulate fundamental problems in interpretable supervised learning (e.g., sparse regression and decision trees), in unsupervised learning (e.g., clustering), and beyond;BackboneLearnsolves the aforementioned problems faster than exact methods and with higher accuracy than commonly used heuristics, while also scaling to large problem sizes. The package is built in Python and is user-friendly and easily extendible: users can directly implement a backbone algorithm for their MIO problem at hand.What do we mean by indicators?Indicators are binary variables that are part of the MIO problem we use to train the aforementioned models.Sparse regression: Each regression coefficient (and the corresponding feature) is paired with an indicator, which is 1 if the coefficient is nonzero and 0 otherwise.Decision trees: An indicator corresponds to a feature in a decision tree node, being nonzero if that feature is chosen for branching at that node.Clustering: An indicator represents whether a pair of data points are in the same cluster, being nonzero if they are clustered together.BackboneLearnThe backbone framework, upon whichBackboneLearnis built, operates in two phases: we first extract a “backbone set” of potentially ``relevant indicators'' (i.e., indicators that are nonzero in the optimal solution) by solving a number of specially chosen, tractable subproblems; we then use traditional techniques to solve a reduced problem to optimality or near-optimality, considering only the backbone indicators. A screening step often proceeds the first phase, to discard indicators that are almost surely irrelevant. For more details, check the paper by Bertsimas and Digalakis Jr (2022)https://doi.org/10.1007/s10994-021-06123-2.Getting StartedInstallationInstall BackboneLearn using pip:pipinstallbackbone-learnNote: For ODT implementations, please follow the instructions described in the libary to install compile CBC from source using coinbrew:https://github.com/D3M-Research-Group/odtlearn#cbc-binariesExample UsageHere are some example usages of BackboneLearn for different tasks that you could find in 'notebooks':Sparse Regression with L0BnB Modelfrombackbone_learn.backbone.backbone_sparse_regressionimportBackboneSparseRegression# Initialize BackboneSparseRegressionbackbone=BackboneSparseRegression(alpha=0.5,beta=0.5,num_subproblems=5,num_iterations=1,lambda_2=0.001,max_nonzeros=10)# Fit the modelbackbone.fit(X,y)# Make predictionsy_pred=backbone.predict(X)Decision Trees with BendersOCT Modelfrombackbone_learn.backbone.backbone_decision_treeimportBackboneDecisionTree# Initialize BackboneDecisionTreebackbone=BackboneDecisionTree(alpha=0.5,beta=0.5,num_subproblems=5,num_iterations=1,depth=3,_lambda=0.5)# Fit the modelbackbone.fit(X,y)# Make predictionsy_pred=backbone.predict(X)Clustering with MIO Formulation Modelfrombackbone_learn.backbone.backbone_clusteringimportBackboneClustering# Initialize BackboneClusteringbackbone=BackboneClustering(beta=1.0,num_subproblems=5,num_iterations=1,n_clusters=5)# Fit the modelbackbone.fit(X)# Make predictionsy_pred=backbone.predict(X)Running SimulationsTo run benchmark simulations, please follow these commands and change the hyper-parameters for each script in 'experiments':# Install Poetry dependenciespoetryinstall# Activate Poetry shell (optional but recommended)poetryshell# Run benchmark simulations for decision trees, sparse regression, and clusteringpythonexperiments/benchmark_decision_tree.py
pythonexperiments/benchmark_sparse_regression.py
pythonexperiments/benchmark_clustering.pyCustom Implementation GuideFollow these steps to implement your backbone algorithms inBackboneLearn:Custom Screening MethodExtendScreenSelectorBasein 'backbone_learn/screen_selector'.Implement thecalculate_utilitiesmethod, which computes utilities (or importances) for each feature based on your screening criteria. Features with the lowest scores may be candidates for elimination. The number of features to keep is defined as a fraction of the total features (alpha * n_features).Custom Heuristic MethodExtendHeuristicSolverBasein 'backbone_learn/heuristic_solvers'.Implementfitandget_relevant_features. Thefitmethod trains a model within each subproblem, which needs to be highly efficient. This function fits a sparse model (regression, trees, etc.) to the data inputs of the subproblem. Theget_relevant_featuresmethod identifies and extracts all features deemed relevant by the model. Ensure to add any necessary parameters in theinitorfitmethod and pass them to the backbone. The fraction of features to include in the feature set for each subproblem is defined by 'beta'. The number of subproblems and iterations are defined by 'num_subproblems' and 'n_iterations', accordingly.Custom Exact SolverExtendExactSolverBasein 'backbone_learn/exact_solvers'.Implementfitandpredictmethods. Thefitfunction is applied to the reduced backbone set and does not need to be efficient; instead, it should use a method with optimality guarantees. The model parameters can again be defined ininitand passed in the backbone class. Thepredictfunction ensures that the model can be used for prediction.Custom Backbone AlgorithmExtendBackboneSupervisedfor supervised algorithms orBackboneUnsupervisedfor unsupervised algorithms in 'backbone_learn/backbone'. The supervised algorithms perform feature selection, while the unsupervised algorithms perform data selection.Implementset_solvers. Add your customized screen selector, heuristic solver, and exact solver as class instances. It's optional to define a screen selector and heuristic solver. Parameters for each solver are recommended to be passed manually. See examples inbackbone_learn/backbone/backbone_decision_tree.Example Usage for Customized Backbone AlgorithmHere's an example of how you can crate a custom Backbone algorithm for supervised method:classCustomBackboneAlgorithm(BackboneSupervised):defset_solvers(self,**kwargs):# Init screen selector or set None to skipself.screen_selector=CustomScreenSelector(**kwarg)# Init heuristic solver or set None to skipself.heuristic_solver=CustomHeuristicSolver(**kwarg)# Init exact solverself.exact_solver=CustomExactSolver(**kwarg)Here's how you can use the custom Backbone algorithm:# Initialize with custom parametersbackbone_algorithm=CustomBackboneAlgorithm(alpha=0.5,beta=0.3,num_subproblems=3,**kwarg)# Fit the modelbackbone_algorithm.fit(X,y)# Make predictionspredictions=backbone_algorithm.predict(X_new)Citing BackboneLearnIf you find BackboneLearn useful in your research, please consider citing the following papers.Paper 1 (Toolkit):Coming soonPaper 2 Methodology:@article{bertsimas2022backbone,
title={The backbone method for ultra-high dimensional sparse machine learning},
author={Bertsimas, Dimitris and Digalakis Jr, Vassilis},
journal={Machine Learning},
volume={111},
number={6},
pages={2161--2212},
year={2022},
publisher={Springer}
}LicenseApache License 2.0