Package: roct
ROCT: Robust Optimal Classification TreesThis folder contains the scripts and code needed to reproduce the results of 'Robust Optimal Classification Trees Against Adversarial Examples'. Much of our code extends the code retrieved from 'GROOT' athttps://github.com/tudelft-cda-lab/GROOT.Installation instructionsTo only install theroctpackage run:pip install roctThe code needs a new version of python, at least 3.7.To also run the experiments clone the repository and install the requirements fromrequirements.txt. We recommend using virtual environments.Simple exampleBelow is a small example for running ROCT (using MILP or MaxSAT solver) on a toy dataset using the Scikit-learn API.fromroct.milpimportOptimalRobustTree,BinaryOptimalRobustTreefromroct.maxsatimportSATOptimalRobustTreefromgroot.toolboximportModelfromsklearn.datasetsimportmake_moonsfromsklearn.preprocessingimportMinMaxScaler# Load the datasetX,y=make_moons(noise=0.3,random_state=0)X_test,y_test=make_moons(noise=0.3,random_state=1)scaler=MinMaxScaler()X=scaler.fit_transform(X)X_test=scaler.transform(X_test)# Define the attacker's capabilities (L-inf norm radius 0.3)epsilon=0.1attack_model=[epsilon,epsilon]names=("MILP","Binary MILP","MaxSAT")trees=(OptimalRobustTree(max_depth=2,attack_model=attack_model),BinaryOptimalRobustTree(max_depth=2,attack_model=attack_model),SATOptimalRobustTree(max_depth=2,attack_model=attack_model),)forname,treeinzip(names,trees):tree.fit(X,y)# Determine the accuracy and adversarial accuracy against attackersaccuracy=tree.score(X_test,y_test)model=Model.from_groot(tree)adversarial_accuracy=model.adversarial_accuracy(X_test,y_test,attack="tree",epsilon=epsilon)print(name)print("Accuracy:",accuracy)print("Adversarial Accuracy:",adversarial_accuracy)print()Reproducing resultsOur figures and fitted trees can be accessed under theout/directory, but the results can also be generated from scratch. Fitting all trees and running all experiments can take many days depending on how many parallel cores are available.Downloading the datasetsRunning the scriptdownload_datasets.pywill download all required datasets from openML into thedata/directory (which has already been done).Main resultsThe main script for fitting and scoring trees isrun.py, which can be accessed by a command line interface. It uses 3-fold cross validation to set themax_depthhyperparameter, trains a tree with the best setting and tests it performance on the test set. To use the script to train and score all trees one can run the commands inall_jobs.txt, theparallelGNU command is particularly useful here. For example, the following will run therun.pyscript in parallel on 15 cores:parallel -j 15 :::: all_jobs.txtThe resulting trees will generate underout/results/. To generate figures and tables please runpython process_results.pyafter the parallel process has finished.Optimality experimentTo fit trees for our optimality experiment we have a similar procedure but it trains trees for 2 hours instead of 30 minutes (per tree). Please run:parallel -j 15 :::: all_jobs_single.txtThe resulting trees will generate underout/results_single_7200_5/. To generate figures please runpython process_optimality.pyafter the parallel process has finished.Bound (theorem 2) and choosing epsilonThe script for computing the bounds while varying epsilon ischoose_epsilons.py. In the resulting tableout/epsilon_choiceswe only change the last column ofblood-transfusionin the paper to avoid duplicate entries.Solver progress over timeThe script for plotting solver progress over time isperformance_over_time.py. This script runs each early-stoppable solver one after the other with trees of depth 3 on one epsilon setting per dataset. This script does not run in parallel so it can take many hours to run.Short scriptsThe XOR dataset, trees and images can be created usingxor_example.py, a summary of the used datasets usingsummarize_datasets.py.