Package: gosdt
Fast Sparse Decision Tree Optimization via Reference EnsemblesThis code creates optimized sparse decision trees. It is a direct competitor of CART[3] and C4.5 [6], as well as DL8.5[1], BinOct[7], and OSDT[4]. Its advantage over CART and C4.5 is that the trees are globally optimized, not constructed just from the top down. This makes it slower than CART, but it provides better solutions. On the other hand, it tends to be faster than other optimal decision tree methods because it uses bounds to limit the search space, and uses a black box model (a boosted decision tree) to “guess” information about the optimal tree. It takes only seconds or a few minutes on most datasets.To make it run faster, please use the options to limit the depth of the tree, and increase the regularization parameter above 0.02. If you run the algorithm without a depth constraint or set the regularization too small, it will run more slowly.This work builds on a number of innovations for scalable construction of optimal tree-based classifiers: Scalable Bayesian Rule Lists[8], CORELS[2], OSDT[4], and, most closely, GOSDT[5].Table of ContentInstallationConfigurationExampleLicenseFAQsInstallationYou may use the following commands to install GOSDT along with its dependencies on macOS, Ubuntu and Windows.You needPython 3.7 or laterto use the modulegosdtin your project.pip3installattrspackagingeditablespandassklearnsortedcontainersgmpy2matplotlib
pip3installgosdtConfigurationThe configuration is a JSON object and has the following structure and default values:{"regularization":0.05,"depth_budget":0,"reference_LB":false,"path_to_labels":"","time_limit":0,"uncertainty_tolerance":0.0,"upperbound":0.0,"worker_limit":1,"stack_limit":0,"precision_limit":0,"model_limit":1,"verbose":false,"diagnostics":false,"balance":false,"look_ahead":true,"similar_support":true,"cancellation":true,"continuous_feature_exchange":false,"feature_exchange":false,"feature_transform":true,"rule_list":false,"non_binary":false,"costs":"","model":"","timing":"","trace":"","tree":"","profile":""}Key parametersregularizationValues: Decimal within range [0,1]Description: Used to penalize complexity. A complexity penalty is added to the risk in the following way.ComplexityPenalty = # Leaves x regularizationDefault: 0.05Note: We highly recommend setting the regularization to a value larger than 1/num_samples. A small regularization could lead to a longer training time. If a smaller regularization is preferred, you must set the parameterallow_small_regto true, which by default is false.allow_small_regValues: true or falseDescription: Flag for allowing regularization < 1/n , where n = num_samples (if false, regularizations below 1/n are automatically set to 1/n)Default: falsedepth_budgetValues: Integers >= 1Description: Used to set the maximum tree depth for solutions, counting a tree with just the root node as depth 1. 0 means unlimited.Default: 0reference_LBValues: true or falseDescription: Enables using a vector of misclassifications from another (reference) model to lower bound our own misclassificationsDefault: falseNote: Ifreference_LBis set to true, you must provide a validpath_to_labels.path_to_labelsValues: String representing a path to a file.Description: This file must be a single-column csv representing a class prediction for each training observation (in the same order as for the training data, using the same class labels as for the training data, and predicting each class present in the training set at least once across all training points). Typically this csv is obtained by fitting agradient boosted decision treemodel on the training data, and saving its training set predictions as a csv file.Example for a dataset with classes 1 and 0:predicted_class
0
1
1
1
0Default: Emptry stringtime_limitValues: Decimal greater than or equal to 0Description: A time limit upon which the algorithm will terminate. If the time limit is reached, the algorithm will terminate with an error.Special Cases: When set to 0, no time limit is imposed.Default: 0More parametersFlagbalanceValues: true or falseDescription: Enables overriding the sample importance by equalizing the importance of each present classDefault: falsecancellationValues: true or falseDescription: Enables propagate up the dependency graph of task cancellationsDefault: truelook_aheadValues: true or falseDescription: Enables the one-step look-ahead bound implemented via scopesDefault: truesimilar_supportValues: true or falseDescription: Enables the similar support bound imeplemented via the distance indexDefault: truefeature_exchangeValues: true or falseDescription: Enables pruning of pairs of features using subset comparisonDefault: falsecontinuous_feature_exchangeValues: true or falseDescription: Enables pruning of pairs continuous of feature thresholds using subset comparisonDefault: falsefeature_transformValues: true or falseDescription: Enables the equivalence discovery through simple feature transformationsDefault: truerule_listValues: true or falseDescription: Enables rule-list constraints on modelsDefault: falsenon_binaryValues: true or falseDescription: Enables non-binary encodingDefault: falsediagnosticsValues: true or falseDescription: Enables printing of diagnostic trace when an error is encountered to standard outputDefault: falseverboseValues: true or falseDescription: Enables printing of configuration, progress, and results to standard outputDefault: falseTunersuncertainty_toleranceValues: Decimal within range [0,1]Description: Used to allow early termination of the algorithm. Any models produced as a result are guaranteed to score within the lowerbound and upperbound at the time of termination. However, the algorithm does not guarantee that the optimal model is within the produced model unless the uncertainty value has reached 0.Default: 0.0upperboundValues: Decimal within range [0,1]Description: Used to limit the risk of model search space. This can be used to ensure that no models are produced if even the optimal model exceeds a desired maximum risk. This also accelerates learning if the upperbound is taken from the risk of a nearly optimal model.Special Cases: When set to 0, the bound is not activated.Default: 0.0Limitsmodel_limitValues: Decimal greater than or equal to 0Description: The maximum number of models that will be extracted into the output.Special Cases: When set to 0, no output is produced.Default: 1precision_limitValues: Decimal greater than or equal to 0Description: The maximum number of significant figures considered when converting ordinal features into binary features.Special Cases: When set to 0, no limit is imposed.Default: 0stack_limitValues: Decimal greater than or equal to 0Description: The maximum number of bytes considered for use when allocating local buffers for worker threads.Special Cases: When set to 0, all local buffers will be allocated from the heap.Default: 0worker_limitValues: Decimal greater than or equal to 1Description: The maximum number of threads allocated to executing th algorithm.Special Cases: When set to 0, a single thread is created for each core detected on the machine.Default: 1FilescostsValues: string representing a path to a file.Description: This file must contain a CSV representing the cost matrix for calculating loss.The first row is a header listing every class that is present in the training dataEach subsequent row contains the cost incurred of predicitng classiwhen the true class isj, whereiis the row index andjis the column indexExample where each false negative costs 0.1 and each false positive costs 0.2 (and correct predictions costs 0.0):negative,positive
0.0,0.1
0.2,0.0Example for multi-class objectives:class-A,class-B,class-C
0.0,0.1,0.3
0.2,0.0,0.1
0.8,0.3,0.0Note: costs values are not normalized, so high cost values lower the relative weight of regularizationSpecial Case: When set to empty string, a default cost matrix is used which represents unweighted training misclassification.Default: Emptry stringmodelValues: string representing a path to a file.Description: The output models will be written to this file.Special Case: When set to empty string, no model will be stored.Default: Emptry stringprofileValues: string representing a path to a file.Description: Various analytics will be logged to this file.Special Case: When set to empty string, no analytics will be stored.Default: Emptry stringtimingValues: string representing a path to a file.Description: The training time will be appended to this file.Special Case: When set to empty string, no training time will be stored.Default: Emptry stringtraceValues: string representing a path to a directory.Description: snapshots used for trace visualization will be stored in this directorySpecial Case: When set to empty string, no snapshots are stored.Default: Emptry stringtreeValues: string representing a path to a directory.Description: snapshots used for trace-tree visualization will be stored in this directorySpecial Case: When set to empty string, no snapshots are stored.Default: Emptry stringExampleThe [https://github.com/ubc-systopia/gosdt-guesses/](GOSDT source code repository) contains example code and datasets to
run GOSDT with threshold guessing, lower bound guessing, and depth limits.
The example python file is available inhttps://github.com/ubc-systopia/gosdt-guesses/gosdt/example.py. A tutorial ipython notebook is available inhttps://github.com/ubc-systopia/gosdt-guesses/gosdt/tutorial.ipynb.The script below will run only if you clone the git repo and run there, however,
it should serve as an example for how to use gosdt.import pandas as pd
import numpy as np
import time
import pathlib
from sklearn.ensemble import GradientBoostingClassifier
from model.threshold_guess import compute_thresholds
from model.gosdt import GOSDT

# read the dataset
df = pd.read_csv("experiments/datasets/fico.csv")
X, y = df.iloc[:,:-1].values, df.iloc[:,-1].values
h = df.columns[:-1]

# GBDT parameters for threshold and lower bound guesses
n_est = 40
max_depth = 1

# guess thresholds
X = pd.DataFrame(X, columns=h)
print("X:", X.shape)
print("y:",y.shape)
X_train, thresholds, header, threshold_guess_time = compute_thresholds(X, y, n_est, max_depth)
y_train = pd.DataFrame(y)

# guess lower bound
start_time = time.perf_counter()
clf = GradientBoostingClassifier(n_estimators=n_est, max_depth=max_depth, random_state=42)
clf.fit(X_train, y_train.values.flatten())
warm_labels = clf.predict(X_train)
elapsed_time = time.perf_counter() - start_time
lb_time = elapsed_time

# save the labels from lower bound guesses as a tmp file and return the path to it.
labelsdir = pathlib.Path('/tmp/warm_lb_labels')
labelsdir.mkdir(exist_ok=True, parents=True)
labelpath = labelsdir / 'warm_label.tmp'
labelpath = str(labelpath)
pd.DataFrame(warm_labels, columns=["class_labels"]).to_csv(labelpath, header="class_labels",index=None)


# train GOSDT model
config = {
            "regularization": 0.001,
            "depth_budget": 5,
            "warm_LB": True,
            "path_to_labels": labelpath,
            "time_limit": 60,
            "similar_support": False
        }

model = GOSDT(config)

model.fit(X_train, y_train)

print("evaluate the model, extracting tree and scores", flush=True)

# get the results
train_acc = model.score(X_train, y_train)
n_leaves = model.leaves()
n_nodes = model.nodes()
time = model.utime

print("Model training time: {}".format(time))
print("Training accuracy: {}".format(train_acc))
print("# of leaves: {}".format(n_leaves))
print(model.tree)OutputX: (10459, 23)
y: (10459,)
gosdt reported successful execution
training completed. 1.658/0.098/1.756 (user, system, wall), mem=364 MB
bounds: [0.290914..0.290914] (0.000000) loss=0.282914, iterations=13569
evaluate the model, extracting tree and scores
Model training time: 1.6584229469299316
Training accuracy: 0.7170857634573095
# of leaves: 8
if ExternalRiskEstimate<=67.5 = 1 and MSinceMostRecentInqexcl7days<=-7.5 = 1 then:
    predicted class: 1
    misclassification penalty: 0.027
    complexity penalty: 0.001

else if ExternalRiskEstimate<=67.5 != 1 and MSinceMostRecentInqexcl7days<=-7.5 = 1 then:
    predicted class: 0
    misclassification penalty: 0.006
    complexity penalty: 0.001

else if ExternalRiskEstimate<=74.5 = 1 and MSinceMostRecentInqexcl7days<=-7.5 != 1 and MSinceMostRecentInqexcl7days<=0.5 = 1 and PercentTradesWBalance<=80.5 = 1 then:
    predicted class: 1
    misclassification penalty: 0.071
    complexity penalty: 0.001

else if ExternalRiskEstimate<=74.5 != 1 and MSinceMostRecentInqexcl7days<=-7.5 != 1 and MSinceMostRecentInqexcl7days<=0.5 = 1 and PercentTradesWBalance<=80.5 = 1 then:
    predicted class: 0
    misclassification penalty: 0.061
    complexity penalty: 0.001

else if ExternalRiskEstimate<=78.5 = 1 and MSinceMostRecentInqexcl7days<=-7.5 != 1 and MSinceMostRecentInqexcl7days<=0.5 = 1 and PercentTradesWBalance<=80.5 != 1 then:
    predicted class: 1
    misclassification penalty: 0.033
    complexity penalty: 0.001

else if ExternalRiskEstimate<=78.5 != 1 and MSinceMostRecentInqexcl7days<=-7.5 != 1 and MSinceMostRecentInqexcl7days<=0.5 = 1 and PercentTradesWBalance<=80.5 != 1 then:
    predicted class: 0
    misclassification penalty: 0.005
    complexity penalty: 0.001

else if ExternalRiskEstimate<=67.5 = 1 and MSinceMostRecentInqexcl7days<=-7.5 != 1 and MSinceMostRecentInqexcl7days<=0.5 != 1 then:
    predicted class: 1
    misclassification penalty: 0.026
    complexity penalty: 0.001

else if ExternalRiskEstimate<=67.5 != 1 and MSinceMostRecentInqexcl7days<=-7.5 != 1 and MSinceMostRecentInqexcl7days<=0.5 != 1 then:
    predicted class: 0
    misclassification penalty: 0.054
    complexity penalty: 0.001FAQsIf you run into any issues when running GOSDT, consult theFAQsfirst.LicenseThis software is licensed under a 3-clause BSD license (see the LICENSE file for details).Related Work[1] Aglin, G.; Nijssen, S.; and Schaus, P. 2020. Learning optimal decision trees using caching branch-and-bound search. InAAAI Conference on Artificial Intelligence, volume 34, 3146–3153.[2] Angelino, E.; Larus-Stone, N.; Alabi, D.; Seltzer, M.; and Rudin, C. 2018. Learning Certifiably Optimal Rule Lists for Categorical Data.Journal of Machine Learning Research, 18(234): 1–78.[3] Breiman, L.; Friedman, J.; Stone, C. J.; and Olshen, R. A. 1984.Classification and Regression Trees. CRC press.[4] Hu, X.; Rudin, C.; and Seltzer, M. 2019. Optimal sparse decision trees. InAdvances in Neural Information Processing Systems, 7267–7275.[5] Lin, J.; Zhong, C.; Hu, D.; Rudin, C.; and Seltzer, M. 2020. Generalized and scalable optimal sparse decision trees. InInternational Conference on Machine Learning (ICML), 6150–6160.[6] Quinlan, J. R. 1993. C4.5:Programs for Machine Learning. Morgan Kaufmann[7] Verwer, S.; and Zhang, Y. 2019. Learning optimal classification trees using a binary linear program formulation. InAAAI
Conference on Artificial Intelligence, volume 33, 1625–1632.[8] Yang, H., Rudin, C., & Seltzer, M. (2017, July). Scalable Bayesian rule lists. InInternational Conference on Machine Learning (ICML)(pp. 3921-3930). PMLR.