SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pseudo-code for STreeD and additional algorithmic techniques for speeding up computation, such as a special depth-two solver, caching, and upper and lower bounds, as well as techniques for sparse trees and hypertuning, can be found in Appendix B. The appendix also provides the conditions for the use of these techniques. Furthermore, in Appendix A we prove the following Theorem:
Theorem 4.7.  $STreeD$ , as defined in Eq. (8), finds the Pareto front for any optimization task that is separable according to Def. 4.2.
4.4 Examples of separable optimization tasks
To illustrate the flexibility of STreeD, we present four diverse example applications for which separable optimization tasks can be formulated. The first two are also covered by the additivity condition from [66]. The last two do not fit this condition but are covered by our framework.
Cost-sensitive classification Cost-sensitive classification considers costs related to obtaining (measuring) the value of features in addition to misclassification costs [55]. Typical applications are in the medical domain when considering expensive diagnostic tasks and asymmetrical misclassification costs. See Appendix C for a more comprehensive introduction and related work.
Let  $\mathcal{D}$  be the instances in a leaf node and let  $M_{k,\hat{k}}$  be the misclassification costs when an instance with true label  $k$  is assigned label  $\hat{k}$ , and let  $m(F,f)$  be the costs of obtaining the value of feature  $f$  (this value also depends on the set of previously measured features  $F$ , because we consider that some tests can be performed in a group and share costs). This results in the following cost function:
$$
g \left(\left\langle \mathcal {D}, F \right\rangle , \hat {k}\right) = \sum_ {(x, k) \in \mathcal {D}} M _ {k, \hat {k}} \quad g \left(\left\langle \mathcal {D}, F \right\rangle , f\right) = | \mathcal {D} | \cdot m (F, f) \tag {9}
$$
This optimization task is separable because the function  $g$  is Markovian, the combination operator  $\oplus$  is addition, and the comparison operator is  $<$ .
Prescriptive policy generation Decision trees can also be used to prescribe policies based on historical data, to maximize the expected reward (revenue) of the policy [12, 45]. An example is medical treatment assignment based on historical treatment response. This is done by reasoning over counterfactuals (the expected outcome if an action other than the historical action was taken).
In Appendix D we provide related work and show three different methods for calculating the expected reward from the literature: regress and compare, inverse propensity weighting, and the doubly robust method. The cost function for each of these is Markovian and the combining operator is addition. Therefore, prescriptive policy generation can be formulated as a separable optimization task. It is also possible to add constraints to the policy, for example, when maximizing revenue under a capacity constraint. We show in Appendix A.4 that capacity constraints are also separable.
Nonlinear classification metrics Nonlinear objectives such as F1-score and Matthews correlation coefficient are typically used for unbalanced datasets. Demirovic and Stuckey [19] provide a globally optimal DP formulation for optimizing such nonlinear objectives for binary classification. STreeD can also optimize for these metrics. For F1-score, for example, one can formulate a combined optimization task that measures the false positive rate for each class  $k \in \{0, 1\}$  as a separate optimization task:
$$
g _ {k} (\mathcal {D}, \hat {k}) = \left\{ \begin{array}{l l} | \{(x, k ^ {\prime}) \in \mathcal {D} \mid k ^ {\prime} \neq \hat {k} \} | & \hat {k} = k \\ 0 & \text {o t h e r w i s e} \end{array} \right. \tag {10}
$$
Both of these tasks are separable, and in Appendix A.5 we show that multiple separable tasks can be combined into a new separable optimization task that results in the Pareto front for misclassifications for each class. This Pareto front can then be used to find the decision tree with, e.g., the best F1-score. See Appendix E for more related work and other details.
Group fairness Optimizing for accuracy can result in unfair results for different groups and therefore several approaches have been recommended to prevent discrimination [60], one of which is demographic parity [28], which states that the expected outcome for two classes should be the same. Previous MIP formulations for group fairness have been formulated [1, 44] and recently also a DP formulation, even though at first hand a group fairness constraint does not seem separable [53].
Let  $a$  denote a discrimination-sensitive binary feature and  $y$  a binary outcome, with  $y = 1$  the preferred outcome and  $\hat{y}$  the predicted outcome; then demographic parity can be described as satisfying  $P(\hat{y} = 1 \mid a = 1) = P(\hat{y} = 1 \mid a = 0)$ . This requirement is typically satisfied by limiting the difference to some percentage  $\delta$ . Let  $N(a)$  be the number of people in group  $a$ , and  $N(\bar{a})$  the people not in group  $a$ . The demographic parity requirement can now be formulated as two separable threshold constraints:
$$
g _ {a} (\mathcal {D}, \hat {k}) = \left\{ \begin{array}{l l} \frac {\left| \mathcal {D} _ {a} \right|}{N (a)} & \hat {k} = 1 \\ \frac {\left| \mathcal {D} _ {\bar {a}} \right|}{N (\bar {a})} & \hat {k} = 0 \end{array} \right. \quad g _ {\bar {a}} (\mathcal {D}, \hat {k}) = \left\{ \begin{array}{l l} \frac {\left| \mathcal {D} _ {a} \right|}{N (a)} & \hat {k} = 0 \\ \frac {\left| \mathcal {D} _ {\bar {a}} \right|}{N (\bar {a})} & \hat {k} = 1 \end{array} \right. \tag {11}
$$
These cost functions with the two threshold constraints that limit both to  $1 + \delta$ , and an accuracy objective can be combined into one separable optimization task as shown in Appendix A.4 and A.5, and therefore it can be solved to optimality with STreeD. We show the derivation of these constraints, a similar derivation for equality of opportunity, and more related work in Appendix F. Note that we require demographic parity on the whole tree and not on every subtree. In fact, requiring demographic parity on every subtree is an example of a constraint that does not satisfy anti-monotonicity.
4.5 Comparison to previous theory and methods
Dynamic programming theory In contrast to what is common in general DP theory, our theory does not assume solution values to be real valued [48] or totally ordered [29]. For example, Karp and Held [48] formulate the monotonicity requirement of a DP as follows: for a given state  $s$ , action  $a$ , cost  $c$  and data  $p$ , let  $h(c, s, a, p)$  be the cost of reaching state  $t(s, a)$  through an input sequence that reaches state  $s$  with cost  $c$ . Monotonicity holds iff  $c_1 \leq c_2$  implies  $h(c_1, s, a, p) \leq h(c_2, s, a, p)$ . In words: if a partial input sequence  $A_1$  with cost  $c_1$  dominates another input sequence  $A_2$  with  $c_2$ , then any other input sequence that starts with  $A_1$  dominates any other input sequence that starts with  $A_2$ , thus satisfying the principle of optimality. This notion is similar to our order preservation, but we do not assume that the costs are real valued or that the costs are totally ordered.
Similarly, Li and Haimes [51] show how multiple optimization tasks can be combined into one, as we also show in Appendix A.5. They construct a separable DP formulation for optimization tasks that can be deconstructed into a series of optimization tasks, each of which is separable and monotonic. Their method also returns the set of nondominated solutions. However, their theory assumes that the solution value for each sub-objective is real, completely ordered, and additive. This limits their theory to element-wise additive optimization tasks. Our theory does not share these limitations.
Dynamic programming approaches In comparison to the DP methods DL8 [66] and GOSDT [52], STreeD covers a wider variety of objectives and constraints. Both STreeD and DL8 require constraints to be anti-monotonic. GOSDT does not support constraints. DL8 and GOSDT require additivity, whereas STreeD requires the less restrictive condition order preservation (see Def. 4.4). Moreover, STreeD can cover a range of new objectives by also allowing for objectives and constraints with a partially defined comparison operator, and by allowing to combine several such objectives and constraints. Examples of problems that could not be solved to global optimality by DL8 and GOSDT, but can be solved with our framework, are group fairness constraints, nonlinear metrics, and revenue maximization under a capacity constraint. Moreover, compared to DL8 and GOSDT, STreeD implements several algorithmic techniques for increasing scalability, such as a specialized solver for trees up to depth two (see Appendix B).
Other approaches MIP can be used to model many optimization tasks, including non-separable optimization tasks. For example, optimizing decision tree policies for Markov Decision Processes can be optimized with MIP [89], but it is not clear how to do this with DP. However, MIP cannot deal with separable, but non-linear objectives, such as F1-score. Moreover, as our experiments also show, DP outperforms the scalability of MIP for the considered optimization tasks by several orders of magnitude.
CP so far has only been applied to maximizing accuracy [84] and (Max)SAT only to maximizing accuracy [34, 76] or finding the smallest perfect tree [42, 64].
Dunn [26] proposes a framework that can optimize many objectives for decision trees (see also [11]). Because of MIP's scalability issues, they propose a local search method based on coordinate descent. However, this method does not guarantee a globally optimal solution. Moreover, their method is not able to find a Pareto front that optimizes multiple objectives at the same time.
To show the flexibility of STreeD, we test it on the four example domains introduced in Section 4.4. Additionally, we compare STreeD with two state-of-the-art dynamic programming approaches for maximizing classification accuracy. STreeD is implemented in  $\mathrm{C}++$  and is available as a Python package. $^1$  All experiments are run on a  $2.6\mathrm{GHz}$  Intel i7 CPU with 8GB RAM using only one thread. MIP models are solved using Gurobi 9.0 with default parameters [33].
In our analysis, we focus on scalability performance in comparison to the state-of-the-art optimal methods (if available). The following shows a summary of the results. The setup of all experiments, more extensive results, and related work of the application domains is provided in the appendices C-G.
5.1 Cost-sensitive classification
For cost-sensitive classification, we compare STreeD with TMDP [58] both in terms of average expected costs and scalability. A direct fair comparison with TMDP is difficult, because TMDP provides no guarantee of optimality, allows for multi-way splits, and is run without a depth limit. We test STreeD and TMDP on three setups for 15 datasets from the literature and report normalized costs. We tune STreeD with a depth limit of  $d = 4$ . In Appendix C we show extended results and also compare with several heuristics from the literature. To our knowledge, no MIP methods for cost-sensitive classification including group discounts (i.e., when certain features are tested together, the feature cost decreases) exist, and therefore are not considered here.
For 21 out of 45 scenarios STreeD generates trees with significantly lower average out-of-sample costs than TMDP ( $p$ -value  $< 5\%$ ). TMDP has lower costs in 1 out of 45 scenarios. In all other scenarios,
<sup>1</sup>https://github.com/AlgTUDelft/pystreed
(a) Cost-sensitive classification
(b) Prescriptive policy generation (synthetic & Warfarin)
(c) Nonlinear classification metrics: F1-score
Figure 2: The percentage of instances for which the algorithm returns the (optimal) tree within the given runtime (higher is better). When MIP methods hit the timeout, the plot shows which percentage of problems are solved within the reported MIP gap. Instances for which all methods return a tree within one second are left out. Note the logarithmic x-axis.
(d) Group fairness: Demographic parity
there is no significant difference. Both methods return trees that perform better and are smaller than those constructed by heuristics. Fig. 2a shows that STreeD is significantly faster than TMDP. Thus, it can be concluded that STreeD on average has slightly better out-of-sample performance than TMDP, returns trees of similar size, and is significantly faster than TMDP.
5.2 Prescriptive policy generation
For prescriptive policy generation, we compare our method with the MIP model Jo-PPG-MIP [43] and the recursive tree search algorithm CAIPWL-opt [92]. Jo et al. [43] show both analytically and through experiments the benefit of their method over [12] and [45]. Their method constructs optimal binary trees based on three different teacher models: regress and compare, inverse propensity weighting, and a doubly robust method. CAIPWL-opt constructs optimal binary trees based on the doubly robust method. Our method implements the same teacher models; thus, all methods find the same solution. Therefore, we compare here only the runtime performance.
As described in detail in Appendix D, we test on the Warfarin dataset [39] and generate 225 training scenarios with 3367 instances and 29 binary features and run each algorithm with a given maximum depth of  $d \in [1, 5]$ . We also generate 275 synthetic datasets with 10, 100, and 200 binary features, each with 500 training samples, and run each algorithm on every scenario with a given maximum depth of  $d = 1, 2$  and 3. When Jo-PPG-MIP hits the timeout of 300 seconds, we report the MIP gap.
Fig. 2b shows that STreeD can find optimal solutions orders of magnitude faster than both JOPPG-MIP and CAIPWL-opt while obtaining the same performance in maximizing the expected outcome.
5.3 Nonlinear classification metrics
For assessing STreeD's performance on optimizing nonlinear metrics such as F1-score, we first compare it with the existing nonlinear MurTree method [19]. We tasked both methods to find optimal trees of depth 3-4 according to an F1-score metric for 25 binary classification datasets from the literature. Fig. 2c shows that STreeD, while being more general, is on average seven times faster than MurTree (computed with geometric mean). This performance gain is in part due to a more efficient computation of the lower and upper bounds, as explained in Appendix B.
This shows that STreeD can be applied to nonlinear objectives and even improves on the performance of a state-of-the-art specialized DP method for this task.
5.4 Group fairness
For group fairness, we follow the experiment setup from [53] as explained in Appendix F. The task is to search for optimal trees for 12 datasets and for trees of depth  $d \in [1, 3]$  under a demographic parity constraint with at most  $1\%$  discrimination on the training data. We compare to the DP-based method DPF [53] and the MIP methods Aghaei-Fair-MIP [1] and Jo-Fair-MIP [44]. When the MIP methods hit the timeout of 600 seconds, we report the MIP gap.
Fig. 2d shows the runtime results. As expected, DPF performs best since it is a DP method specifically designed for this task. STreeD is more general and yet remains close to DPF, while being several orders of magnitude faster than both MIP methods. In Appendix F we further show how STreeD is the first optimal DP method for optimizing an equality-of-opportunity constraint.
It can be concluded that STreeD allows for easy modeling of complex constraints such as demographic parity and equality of opportunity while outperforming MIP methods by a large margin.
5.5 Classification accuracy
Finally, we compare STreeD with two state-of-the-art dynamic programming approaches for optimal decision trees that maximize classification accuracy: DL8.5 [3, 4] and MurTree [20]. All three methods find optimal decision trees for maximum depth four and five for 25 binary classification datasets from the literature. The results in Appendix G show that on average STreeD is more than six times faster than DL8.5 and is close to MurTree, with MurTree being 1.25 faster (computed with the geometric mean). Since DL8.5 outperforms DL8 [3] and MurTree outperforms GOSDT by a large margin [20], we can conclude that STreeD not only allows for a broader scope of optimization tasks than DL8 and GOSDT, but is also more scalable.
We present STreeD: a general framework for finding optimal decision trees using dynamic programming (DP). STreeD can optimize a broad range of objectives and constraints, including nonlinear objectives, multiple objectives, and non-additive objectives, provided that these objectives are separable. We apply DP theory to provide necessary and sufficient requirements for such separability in decision tree optimization tasks and introduce the new notion order preservation that guarantees the principle of optimality [9]. We show that these results are more general than the state-of-the-art general DP solvers for decision trees [52, 66].
We illustrate the generalizability of STreeD in five application domains, two of which are not covered by the previous general DP solvers. For each, we present a separable optimization formulation and compare STreeD's scalability to the domain-specific state-of-the-art. The results show that STreeD sustains or improves the scalability performance of DP methods, while being flexible, and outperforms mixed-integer programming methods by a large margin on these example domains.
STreeD allows for a much broader set of applications than presented here. Therefore, future work should further investigate STreeD's performance on, for example, regression tasks and other non-additive objectives. STreeD could also be used to further develop optimization of trees that lack explanation redundancy [40]. Finally, STreeD could further be improved by adding multi-way branching and allowing for continuous features.
[1] Sina Aghaei, Mohammad Javad Azizi, and Phebe Vayanos. Learning Optimal and Fair Decision Trees for Non-Discriminative Decision-Making. In Proceedings of AAAI-19, pages 1418-1426, 2019.
[2] Sina Aghaei, Andres Gomez, and Phebe Vayanos. Strong Optimal Classification Trees. arXiv preprint arXiv:2103.15965, 2021.
[3] Gaël Aglin, Siegfried Nijssen, and Pierre Schaus. Learning Optimal Decision Trees Using Caching Branch-and-Bound Search. In Proceedings of AAAI-20, pages 3146–3153, 2020.
[95] M. Zwitter and M. Soklic. Breast Cancer Dataset. URL https://archive.ics.uci.edu/ml/datasets/ Breast+Cancer.
[96] M. Zwitter and M. Soklic. Lymphography Dataset, 1988. URL https://archive.ics.uci.edu/ml/datasets/Lymphography.
[97] M. Zwitter and M. Soklic. Primary Tumor Dataset, 1988. URL https://archive.ics.uci.edu/ml/datasets/primary+tumor.
The following sections provide proofs for the propositions and theorems from the main text. We first prove Prop. 4.3 that claims that an optimization task is separable if it has a Markovian cost and transition function and if it satisfies the principle of optimality. We then prove Theorem 4.6 about the necessary and sufficient conditions for an optimization task to be separable. Next, we prove Theorem 4.7 about STreeD being able to solve separable optimization tasks to optimality. We extend the theory of the main text by introducing anti-monotonic threshold constraints. Finally, we show how multiple separable optimization tasks can be combined into a new separable optimization task.
A.1 Proof of Prop. 4.3 about Separability
Def. 4.2 defines separability by stating that for a given optimization task the optimal solution to any subtree in a tree search should be independent of any of the decision variables other than the decision variables of the subtree search and the parent node branching decisions. Prop. 4.3 states that this is the same as saying that the optimization task has a Markovian cost and transition function and satisfies the principle of optimality [9], i.e., optimal solutions can only be constructed from optimal solutions to subproblems:
Definition A.1 (Principle of Optimality). Let  $o = \langle g, t, \succ, \oplus, c, s_0 \rangle$  be an optimization task, let  $s$  be a state, let  $f$  be a branching decision, and let  $\Theta_1$  and  $\Theta_2$  be all the solutions to subproblems with state  $t(s, f)$  and  $t(s, \bar{f})$ , and let  $\Theta = \mathrm{merge}(\Theta_1, \Theta_2, s, f)$ . Then optimization task  $o$  satisfies the principle of optimality if and only if for every state  $s$ , branching decision  $f$  and solution sets  $\Theta_1$  and  $\Theta_2$ , the following holds:
$$
\operatorname {o p t} (\Theta , s) = \operatorname {o p t} \left(\operatorname {m e r g e} \left(\operatorname {o p t} \left(\Theta_ {1}, t (s, f)\right), \operatorname {o p t} \left(\Theta_ {2}, t (s, \bar {f})\right), s, f\right)\right) \tag {12}
$$
With this definition in place, we can prove Prop. 4.3.
Proof. Prop. 4.3 requires two conditions: a Markovian cost and transition function, and the principle of optimality. To prove Prop. 4.3, we need to show that both conditions are necessary and sufficient. Therefore, we first show that if both conditions hold, the optimization task must be separable. We then show that if one of the two conditions of Prop. 4.3 fails, the optimization task is not separable.
Sufficient: To show that the two conditions of Prop. 4.3 are sufficient we use total induction over the tree depth  $d$ .
The base step  $(d = 0)$ , considers the search for optimal leaf nodes. If a Markovian cost function  $g(s, \hat{k})$  exists, the optimal label assignment can be determined by opt  $\left(\cup_{\hat{k} \in \mathcal{K}} g(s, \hat{k}), s\right)$ . This does not depend on any decision variable other than the state  $s$  and the leaf node label assignment  $\hat{k}$  and therefore Def. 4.2 is satisfied for all tree searches up to depth zero.
The induction step (for  $d > 0$ ) assumes Def. 4.2 is satisfied for all tree searches up to depth  $d - 1$ . For these tree searches, optimal solutions can now be constructed by iterating over all possible branching decisions for the root node, using Eq. (12) to find optimal solutions when the root node branching decision is fixed, and using opt to select the optimal solution from the union of all these solution sets. Since neither the subtree search, nor Eq. (12), nor the transition function, nor the branching cost function, nor the union operator, nor the opt function depend on any decision variable other than the decision variables for this tree search and the parent node decisions, the optimal solution for this search can also be constructed independently of those variables.
Necessary conditions: If an optimization task is separable, a Markovian cost function and transition function necessarily exist. Since a node's state is the product of applying the transition function on the previous state, this state can be considered as a function of the initial state  $s_0$ , the parent nodes' branching decisions  $F$ , and the immediate action: the branching decision or the label assignment. Now assume that no Markovian cost and transition function exist. This means that the cost of an action cannot be expressed in terms of only the state, the immediate action, and the parent's branching decisions. This means that Def. 4.2 is not satisfied: a contradiction. Therefore a Markovian cost and transition function necessarily exists if an optimization task is separable.
For tree searches with a depth larger than zero, optimal solutions should be able to be constructed from optimal solutions to subtree searches, as defined in Eq. (12). Consider such a subtree search,
which after deciding on the branching decision in its root node, creates two subproblems for the resulting children nodes. The optimal solution should be constructible from the optimal solution to these subproblems according to Eq. 12. If the optimal solution to this tree search cannot be constructed from the optimal solution to its subtree searches, then the optimal solutions to the subtree searches are not optimal: a contradiction. Therefore Eq. (12) is a necessary condition.
A.2 Proof of Theorem 4.6 about Conditions for Separability