SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that TMDP is the only method that is evaluated here that considers multi-way splits instead of binary splits.
For these methods, we use an implementation that is available online. C4.5 is implemented in Weka, with CSID3, EG2, and IDX implemented as C4.5 but with the new splitting criterion. Leaf nodes do not select the majority class, but the class with the lowest total misclassification costs.
Normalized costs Table 2 shows the normalized cost results for all methods on a variety of datasets. On 14 out of 15 datasets STreeD has the best performance or is not statistically significantly worse than the method with the best performance. The results show that TMDP and STreeD have similar performance and significantly outperform the other heuristics on all datasets except Flare. Therefore, the rest of the analysis will focus on TMDP and STreeD.
Table 3 shows the cost results for TMDP and STreeD for low, middle, and high misclassification costs. The difference in the results can be explained by several factors: multi-way splits versus binary splits, and the quality of the binarization. TMDP considers multi-way splits and therefore has a larger solution space, but does not necessarily find the best tree in this solution space according to the training data. STreeD, on the other hand, only considers binary decision trees and its solution quality depends on the binarization. However, it always finds the best binary decision tree for the training data.
Runtime Figure 4 shows the difference in runtime for STreeD (hypertuned with  $d = 4$ ) and TMDP. As can be seen, STreeD is often an order of magnitude faster than TMDP, and is significantly faster than TMDP for all datasets ( $p < 5\%$ ), except for Iris, the smallest dataset.
Interpretability Table 4 shows how the methods compare in terms of interpretability metrics. When only looking at the numbers, TMDP has the best score, and STreeD follows shortly after. The heuristics, as expected, score much worse. However, TMDP allows for multi-way split trees and the others do not. According to Piltaver et al. [71], a higher branching factor increases the classifying time of an average user, although this effect can only be seen with branching factors higher than 3.
$^{2}$ https://github.com/shanigu/CostSensitive
Table 3: Test normalized cost (\%) for varying misclassification costs. Significantly  $(p < 5\%)$  best results are marked in bold. Timedouts are marked as '-'.
Dataset	Low costs	Middle costs	High costs
STreeD	TMDP	STreeD	TMDP	STreeD	TMDP
Annealing	1.3	1.6	1.2	1.6	1.2	1.5
Breast	8.3	8.3	15.7	15.6	37.3	39.1
Car	9.8	9.8	14.0	15.8	20.5	22.9
Diabetes	7.3	7.2	11.6	11.8	23.4	23.5
Flare	9.9	9.8	15.2	16.3	24.9	28.0
Glass	10.9	10.8	13.8	14.1	18.5	18.1
Heart	4.0	4.0	7.2	7.5	16.7	17.1
Hepatitis	8.4	9.2	12.9	13.5	25.6	26.0
Iris	7.5	7.5	10.7	10.7	20.2	20.3
Krk	1.8	-	1.8	-	1.7	-
Mushroom	1.7	-	1.8	-	2.1	-
Nursery	0.1	0.1	0.1	0.1	0.1	0.1
Soybean	10.9	18.7	11.6	16.3	11.8	11.3
Tictactoe	6.8	6.8	12.7	12.7	28.8	28.9
Wine	8.0	11.3	11.9	12.5	11.2	12.8
Figure 4: Runtime (s) comparison. The top-down induction heuristics (C4.5, CSID3, EG2, and IDX) have been grouped together as 'TDI Heuristics' because they almost do not differ in runtime. A timeout of 300 seconds is used. Note the logarithmic x-axis.
Table 4: Interpretability metrics. Significantly  $(p < 5\%)$  best results are marked in bold. Datasets for which TMDP exceeded the timeout limit are ignored.
C4.5	CSID3	EG2	IDX	MetaCost	TMDP	STreeD
Tree depth	7.9	7.1	7.1	7.1	7.2	1.9	2.2
Leaf nodes	26.1	27.2	28.6	28.7	24.8	7.4	5.4
Question length	4.4	4.7	5.0	5.0	4.7	1.7	2.8
Therefore, we can conclude that both TMDP and STreeD clearly outperform the heuristics in terms of interpretability.
C.4 Discussion
The results above clearly show the advantage of TMDP and STreeD over the heuristics. For most datasets, the cost performance of TMDP and STreeD was much better than the heuristics. This is best explained by the fact that the splitting criteria of the heuristics do not directly capture the objective, and in these cases even ignore the misclassification costs.
Comparing TMDP and STreeD is more difficult. In terms of costs, both at times perform best, although on average STreeD's out-of-sample costs are (significantly) lower:  $13.3\%$  normalized costs versus TMDP's  $14.3\%$ . Both methods show some overfitting and this could be further addressed. Direct comparison is difficult though, because they differ in many ways: TMDP has categorical data
with multi-way splits and seeks a tree without a depth limit. STreeD has binary data with binary splits and seeks the best tree up to some maximum size.
The runtime of STreeD is at least an order of magnitude lower than TMDP. One of the consequences here is that TMDP was not able to find good trees for several datasets within the given time limit. Moreover, these results are produced with TMDP's limit of considering only subsets for the 10 most costly features. Without this limit, TMDP's runtime would have been worse. STreeD therefore is more scalable than TMDP.
Both STreeD and TMDP have good interpretability and clearly outperform the heuristics. The trees returned by the heuristics are often too large to be considered interpretable.
From these results it can be concluded that STreeD can successfully be applied to cost-sensitive classification, outperforming the state-of-the-art in cost performance and runtime, while providing small interpretable trees.
D Prescriptive Policy Generation
Decision trees can also be used to prescribe policies. Given a set of instances, actions, and outcomes, the decision tree prescribes which action for an unseen instance will give the best predicted outcome. An example use case is prescribing medical treatment on basis of historical treatment response.
However, since historical data does not necessarily have the outcome for every possible action, the prescriptive model needs to reason about the counterfactuals. Therefore, conceptually, this problem can be split into a prediction and a prescription problem. The prediction problem predicts the counterfactuals. The prescription problem decides what action to take.
The following sections present first the related work and then a formal problem definition and other preliminaries. We then describe the experiments and conclude with a discussion.
D.1 Related Work
Athey and Imbens [7] use trees to estimate the causal effect of treatments. They use part of the dataset to estimate heterogeneous partitions and the rest is used to estimate treatment effects in the leaves, including confidence intervals for these values. However, this method does not directly assign policies, and in the case of non-binary treatments, several trees need to be combined to produce a policy tree, reducing its interpretability.
Kallus [45] combines the prediction and prescription into one learning problem and proposes both a top-down induction heuristic and an optimal MIP method that directly optimizes prescription error. Their methods work under the assumption that the training data is either produced in a randomized experiment or that the generated tree partitions the data such that the historical treatment assignment becomes independent of the individual. However, in practice, the optimal MIP method does not scale to sufficient depth to reach such a partition. Their results therefore also show that the optimal MIP method in most situations performs worse than other solution methods.
Bertsimas et al. [12] improve on [45] by observing that the performance of prescriptive trees relies not only on a good heterogeneous partition but also on the quality of the prediction of outcomes. Therefore they propose an optimal MIP formulation that co-optimizes both the prescription and prediction error in one objective. To circumvent the scalability issues for finding the global optimum, they repeatedly use a coordinate descent method [11] to find local optima and select the best tree.
Biggs et al. [13] and Zhou et al. [92] note the disadvantage of trying to incorporate the predictive aspect into the MIP model. Instead they use a separate predictive teacher model and use this model to guide a top-down greedy heuristic. Zhou et al. [92] also provide an optimal recursive search method, but they did not incorporate any of the scalability improvements from similar approaches [3, 20] such as caching and lower-bounding.
Amram et al. [5] take a similar approach, but instead of a greedy heuristic, they propose to adapt the MIP model of [12] by updating the objective to incorporate the counterfactual data obtained from a predictive model, using objectives from causal inference. Their experiments show its advantage over both the greedy heuristics and the method by Bertsimas et al. [12].
Jo et al. [43] analytically show the limitations of both [45] and [12]: their assumption is generally not satisfied in conditionally randomized experiments. Both methods can return trees that actively violate their own assumption. Therefore Jo et al. [43], like Amram et al. [5], propose to drop this assumption and use separate predictive and prescriptive models. Apart from this, they show that under mild conditions their method provides an optimal out-of-sample policy when the number of training samples approaches infinity. Their model also allows for the addition of budget constraints. They propose to use the same objectives as Amram et al. [5], but their MIP formulation is based on the more scalable formulation of Aghaei et al. [2], so unlike Bertsimas et al. [12] and Amram et al. [5] they do not need to fall back on coordinate descent for local optima. However, despite their scalability improvements, almost half of the instances in their experiments are still not solvable within one hour.
Finally, Subramanian et al. [79] find near-optimal solutions by using column generation to improve scalability. Furthermore, their model allows for the addition of a variety of constraints.
D.2 Preliminaries
We consider a dataset  $\mathcal{D}$  containing instances  $(x,k,y)$  with  $x\in \mathcal{X}$  the feature vector,  $k\in \mathcal{K}$  the label or historically assigned treatment, and  $y\in \mathbb{R}$  the observed outcome value. We assume no information on the historical treatment assignment policy. Let  $Y(x,k)$  be the value outcome when assigning treatment  $k$  to an individual described by  $x$ . The goal now is to find a policy  $\pi :\mathcal{X}\to \mathcal{K}$  that maximizes the expected outcome  $Q(\pi) = \mathbb{E}[Y(x,\pi (x))]$ .
Because not every treatment was applied to each person, we do not have full information on the values of  $Y(x,k)$ , and therefore we need a teacher model that provides information on the counterfactuals, based on the data that we do have. Similar to [5, 43], we will here consider three teacher models: the direct method (DM), inverse propensity weighting (IPW), and the doubly robust approach (DR).
Direct method The direct method (also called Regress and Compare) trains a teacher model  $\hat{v}_k$  for the expected outcome  $y$  for each possible treatment  $k$  by splitting the training data based on the historical treatment. The objective value  $Q$  can therefore be formulated as follows:
$$
Q _ {\mathrm {D M}} (\mathcal {D}, \pi) = \frac {1}{| \mathcal {D} |} \sum_ {(x, k, y) \in \mathcal {D}} \hat {v} _ {\pi (x)} (x) \tag {62}
$$
Inverse propensity weighting The propensity scores  $\mu (x,k) = \mathbb{P}(k\mid x)$  give the probability of a treatment  $k$  for a given feature vector  $x$ . IPW implicitly models the counterfactuals by reweighing the actual data based on the inverse of their propensity scores. This requires the training of a propensity score model  $\hat{\mu} (x,k)$ . Any ML model can be used to learn  $\hat{\mu}$ . Given a model for  $\hat{\mu}$ , the objective value  $Q$  of the policy becomes:
$$
Q _ {\mathrm {I P W}} (\mathcal {D}, \pi) = \frac {1}{| \mathcal {D} |} \sum_ {(x, k, y) \in \mathcal {D}} \frac {\mathbb {1} (\pi (x) = k)}{\hat {\mu} (x , k)} \tag {63}
$$
Doubly robust Finally, the doubly robust method [25] combines IPW and DM as follows:
$$
Q _ {\mathrm {D R}} (\mathcal {D}, \pi) = \frac {1}{| \mathcal {D} |} \sum_ {(x, k, y) \in \mathcal {D}} \left(\hat {v} _ {\pi (x)} (x) + (y - \hat {v} _ {k} (x)) \frac {\mathbb {1} (\pi (x) = k)}{\hat {\mu} (x , k)}\right) \tag {64}
$$
D.3 Objective Task Formulation
Once the objective value function  $Q$  is known as stated above, this can be directly applied in our framework since each of the objectives  $Q_{\mathrm{IPW}}$ ,  $Q_{\mathrm{DM}}$  and  $Q_{\mathrm{DR}}$  are separable, once the teacher model values resulting from  $\hat{\mu}$  and  $\hat{v}$  have been precomputed. For each of these functions, the combining operator is addition, the comparator is  $>$  (maximization) and the cost function is determined by the value of the policy  $\pi_k$  that assigns treatment  $k$  to all individuals:
$$
g (\mathcal {D}, k) = Q (\mathcal {D}, \pi_ {k}) \tag {65}
$$
Table 5: Mean effect and treatment effect functions for the synthetic dataset, based on [7].
φ(x)	κ(x)
f=2	1/2x1+x2	1/2x1
f=10	1/2∑j=12xj+∑j=36xj	∑j=12max(0,xj)
f=20	1/2∑j=14xj+∑j=58xj	∑j=14max(0,xj)
D.4 Experiments
The following experiments compare the performance of STreeD with the state-of-the-art optimal methods for prescriptive policy generation. In this analysis, our main consideration is runtime performance, since the other methods optimize precisely the same objective. We examine the methods on synthetic data and on a real-life dataset from the literature.
Jo-PPG-MIP [43] is a MIP model based on a max-flow formulation for optimal decision trees from [2] but with the aforementioned  $Q$ -function as the objective. We call this method Jo-PPG-MIP, after their first author. Their experiments show their advantage over previous MIP methods [12] and [45]. Our implementation of Jo-PPG-MIP is based on the open-source code for [2], for which we updated the objective.<sup>3</sup>
CAIPWL-opt [92] also observe the scalability issues with MIP-based methods and therefore propose a recursive tree-search algorithm that maximizes the doubly robust metric. Their method does not use caching, and therefore this method does not qualify as a dynamic programming approach. They suggest their optimal method is best for trees of at most depth three. For deeper trees, they suggest using approximate methods. In our experiments we use their publicly available R-package. $^4$
D.4.1 Synthetic Data
For the experiment with synthetic data, we follow the setup as done in [7, 43]. We consider two possible treatments  $K = \{0,1\}$ , input vectors  $X$ , which consist of  $f$  independent variables with distribution  $N(0,1)$  and for each  $X_{i}$  corresponding potential outcomes  $Y_{i}(k)$ :
$$
Y _ {i} (k) = \phi \left(X _ {i}\right) + \frac {1}{2} (2 k - 1) \cdot \kappa \left(X _ {i}\right) + \varepsilon_ {i}. \tag {66}
$$
In this equation,  $\phi(x)$  is the mean effect,  $\kappa(x)$  is the treatment effect, and  $\varepsilon_i \sim N(0, 0.1)$  is independent noise. Based on [7] we consider three variations of  $\phi$  and  $\kappa$ , for  $f = 2, 10, 20$ , as shown in Table 5. As can be seen from this table, for  $f = 2$ , the mean effect  $\phi$  depends on both variables, and the treatment effect only on one. For  $f = 10$  and  $f = 20$ , the mean effect depends on a subset of the variables. The treatment effect also depends on a subset of the variables, but only when those values are positive. Moreover, for these two cases, some variables are not related to the potential outcomes at all and function as noise in the dataset. For the training dataset, per instance, we then select the optimal treatment (the treatment with the maximum value for  $Y_j$ ) as the historical treatment with probability  $p \in \{0.1, 0.25, 0.5, 0.75, 0.9\}$  and store the corresponding outcome in the dataset. We repeat this process five times and generate training sets and test sets with sizes 500 and 10000 respectively.
We binarize each of the original features into 10 bins with a similar frequency count. The teacher models are then trained on the original data. For IPW we train a decision tree model (dt, computed with CART), logistic regression (log), and the true propensity scores. For DM we train a linear regression (lr) and a lasso regression with  $\alpha = 0.08$ . For DR we use all six combinations of the IPW and DM methods.
<sup>3</sup>https://github.com/pashew94/StrongTree
<sup>4</sup>https://github.com/grf-labs/policytree
Figure 5: Proportion of instances that can be solved per method for the synthetic datasets within a given runtime (s). Note the logarithmic x-axis for runtime.
Table 6: Results for the Warfarin dataset. The reported runtime is the average of runs that did not result in a timeout (300s).
d	Jo	CAIPWL-opt	STreeD Time (s)	OOS Correct Treatment
Timeout	Time (s)	Timeout	Time (s)
1	-	30	-	< 1	< 1	83.1%
2	62%	215	-	5	< 1	84.6%
3	100%	-	-	96	< 1	86.6%
4	100%	-	100%	-	< 1	88.0%
5	100%	-	100%	-	3	89.7%
Runtime The setup above results in 75 scenarios that are solved with 11 different teacher methods. We ran all three methods with a maximum tree depth of 1, 2, and 3, resulting in 2475 problems for each method to solve.
Figure 5 shows the runtime performance by listing how many instances of the synthetic dataset can be solved within the specified amount of runtime. Note that this is only the time required to build the tree. It does not include the time to train the teacher model. These results show that STreeD is several orders of magnitude faster than Jo-PPG-MIP and also scales much better than CAIPWL-opt while providing exactly the same solutions.
D.4.2 Warfarin Dosing
Similar to previous studies, we test our method on the Warfarin dosing use-case [12, 43, 45]. Warfarin is an anticoagulant, but the appropriate dose may vary up to a factor 10 among patients. Based on experimental tests, a dataset is compiled which prescribes for 5700 patients a suggested dose [39]. For our experiment, unless otherwise specified, we follow the setup by [43].
We calculate the square root of the suggested weekly dose by the formula presented in the supplementary material section S1e of [39] and add noise of  $N(0, 0.02)$ . This number is then squared and divided by seven to give the suggested daily dose. The features used in their formula are also the features used for the dataset, preprocessed as specified by them. The only three non-binary features (age, weight, and height) are binarized by splitting them into five buckets with approximately the same number of instances. Instances with missing values for weight, height, age, and the therapeutic dose of Warfarin are removed. This leaves 4490 instances. The daily suggested dose is discretized into three groups, with  $k^{opt} = 0$  if the dose is less than  $3\mathrm{mg/day}$ ,  $k^{opt} = 1$  if the dose is between 3 and  $7\mathrm{mg/day}$ , and  $k^{opt} = 2$  if the dose is larger than  $7\mathrm{mg/day}$ . The observed outcome is  $Y_{i}(k_{i}) = 1$ , if  $k_{i} = k_{i}^{opt}$ ; and  $Y_{i}(k_{i}) = 0$  otherwise.
For simulating historical data, three different setups are tested. In the first, every individual receives a uniformly random treatment  $k_{i}$  in the historical data with the corresponding outcome  $Y_{i}$ , to simulate a marginally randomized setting. In the second and third setups, historical treatments are obtained by using the formula for the weekly dose from [39] mentioned above with each coefficient in this formula perturbed by a random amount from  $U(-0.06, 0.06)$  and  $U(-0.11, 0.11)$  respectively, to change the probability of the historical data having the correct assignment. Each setup is used five
Table 7: Runtime (s) for minimizing biobjective misclassification score for two classes, with timeouts beyond 600s denoted with '-'. Best results are shown in bold. Average of five runs.
Dataset	d=3	d=4
|D|	|D+|	|D-|	|F|	MurTree	STreeD	MurTree	STreeD
Anneal	812	625	187	93	<1	<1	24	2
Audiology	216	57	159	148	<1	<1	54	2
Australian-Credit	653	357	296	125	2	<1	139	19
Breast-Wisconsin	683	444	239	120	<1	<1	61	7
COMPAS	6907	3196	3711	12	2	<1	7	<1
Diabetes	768	500	268	112	3	<1	117	13
Fico	10459	5000	5459	17	6	<1	26	3
German-Credit	1000	700	300	112	6	<1	164	34
Heart-Cleveland	296	160	136	95	<1	<1	43	6
Hepatitis	137	111	26	68	<1	<1	6	<1
Hypothyroid	3247	2970	277	88	<1	<1	28	3
Ionosphere	351	225	126	445	26	11	-	-
Kr-vs-kp	3196	1669	1527	73	<1	<1	22	3
Letter	20000	813	19187	224	46	7	-	469
Lymph	148	81	67	68	<1	<1	8	1
Mushroom	8124	4208	3916	119	<1	<1	<1	<1
Pendigits	7494	780	6714	216	13	4	-	240
Primary-Tumor	336	82	254	31	<1	<1	<1	<1
Segment	2310	330	1980	235	<1	<1	<1	<1
Soybean	630	92	538	50	<1	<1	3	<1
Splice-1	3190	1655	1535	287	17	9	-	-
Tic-Tac-Toe	958	626	332	27	<1	<1	1	<1
Vehicle	846	218	628	252	5	2	-	130
Vote	435	267	168	48	<1	<1	3	1
Yeast	1484	463	1021	89	5	<1	68	8
times to generate historical treatments. Each of these 15 datasets is then randomly split five times into a training and test set with  $75\%$  and  $25\%$  of the instances respectively. For each of these, a decision tree is trained for IPW scores and a random forest regressor for DM estimates.
Results Table 6 shows the results for applying Jo-PPG-MIP and STreeD to the Warfarin case. The results are very similar for the different teacher methods and the probability of having the correct assignment in the historical data. Therefore we just show the average results for trees of depth 1-5.
These results show that STreeD is several orders of magnitude faster than both Jo-PPG-MIP and CAIPWL-opt. Because of this, STreeD can find optimal trees for larger depth, which for this problem results in a better out-of-sample (OOS) correct treatment rate.
E Nonlinear Classification Metrics
For imbalanced datasets, it is beneficial to use metrics other than accuracy, such as F1-score and Matthews correlation coefficient. However, these metrics are nonlinear and can therefore not (easily) be optimized with state-of-the-art methods for optimal decision trees. To address this, Demirovic and Stuckey [19] developed a DP-based approach for binary classification that can optimize any function that is monotonic with respect to the misclassification score for both classes. This class of functions includes F1-score and Matthews-correlation coefficient.
STreeD can also optimize this whole class of functions by using the fact that multiple separable optimization tasks can be combined to form a new separable optimization task, according to Prop. A.4.
This is different from the results in [43]. We have corresponded with the authors about this but were not able to reproduce their results. We will include the test setup in our repository to enable reproducibility of the results.
The next section provides more related work. Then we show experimental results for applying STreeD to nonlinear objectives.
E.1 Related Work
Lin et al. [52] also propose a DP method for solving a variety of objectives, such as balanced accuracy and F1-score. However, they observe that F1-score is harder to optimize than other metrics that are monotonic in relation to the number of false positives and negatives because the best assignment in one leaf node is dependent on the assignment in another node. Therefore, they simplify the problem by introducing a parameter  $\omega$  that helps determine which label should be assigned in leaf nodes, but as a result, the solution is no longer guaranteed to be optimal.
Xin et al. [91] present a DP method for finding the full Rashomon set of sparse decision trees, which is the set of almost-optimal decision trees (including the optimal). They then show how a Rashomon set for trees optimized for accuracy can be used to find the Rashomon set for F1-score. For this, they provide a bound on the accuracy score such that an Accuracy Rashomon set covers the F1-score Rashomon set.
E.2 Experiments
In our experiments, we evaluate STreeD's performance on finding trees that maximize F1-score. F1-score is the harmonic mean of precision and recall and can be calculated as follows based on the number of true positives  $(tp)$ , false positives  $(fp)$ , and false negatives  $(fn)$ :  $tp / (tp + 0.5(fp + fn))$ .