SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jacobus G. M. van der Linden Mathijs M. de Weerdt Emir Demirović
Delft University of Technology, Department of Computer Science {J.G.M.vanderLinden, M.M.deWeerdt, E.Demirovic}@tudelft.nl
Global optimization of decision trees has shown to be promising in terms of accuracy, size, and consequently human comprehensibility. However, many of the methods used rely on general-purpose solvers for which scalability remains an issue. Dynamic programming methods have been shown to scale much better because they exploit the tree structure by solving subtrees as independent subproblems. However, this only works when an objective can be optimized separately for subtrees. We explore this relationship in detail and show the necessary and sufficient conditions for such separability and generalize previous dynamic programming approaches into a framework that can optimize any combination of separable objectives and constraints. Experiments on five application domains show the general applicability of this framework, while outperforming the scalability of general-purpose solvers by a large margin.
Many high-stake domains, such as medical diagnosis, parole decisions, housing appointments, and hiring procedures, require human-comprehensible machine learning (ML) models to ensure transparency, safety, and reliability [74]. Decision trees offer such human comprehensibility, provided the trees are small [71]. This motivates the search for optimal decision trees, i.e., trees that globally optimize an objective for a given maximum size.
The aforementioned application domains wildly differ from one another, and therefore the objective and constraints also differ, leading to a variety of tasks: e.g., cost-sensitive classification for medical diagnosis [32, 82]; prescriptive policy generation for revenue maximization [13]; or ensuring fairness constraints are met [1]. For broad-scale application of optimal decision trees, we need methods that can generalize to incorporate objectives and constraints such as the ones from these application domains, while remaining scalable to real-world problem sizes.
However, many state-of-the-art methods for optimal decision trees lack the required scalability, for example, to optimize datasets with more than several thousands of instances or to find optimal trees beyond depth three. This includes approaches such as mixed-integer programming (MIP) [10, 86], constraint programming (CP) [84], boolean satisfiability (SAT) [42] or maximum satisfiability (MaxSAT) [34].
Dynamic programming (DP) approaches show scalability that is orders of magnitude better [3, 20, 52] by directly exploiting the tree structure. Each problem is solved by a recursive step that involves two subproblems, each just half the size of the original problem. Additional techniques such as bounding and caching are used to further enhance performance.
However, unlike general-purpose solvers, such as MIP, DP cannot trivially adapt to the variety of objectives and constraints mentioned before. For DP to work efficiently, it must be possible to solve the optimization task at hand separately for every subtree.
Therefore, the main research question considered here is to what extent can this separability property be generalized? In answering this question we provide a generic framework called  $STreeD$  (Separable Trees with Dynamic programming). We push the limits of DP for optimal decision trees by providing conditions for separability that are both necessary and sufficient. These conditions are less strict and extend to a larger class of optimization tasks than those of state-of-the-art DP frameworks [52, 66]. We also show that any combination of separable optimization tasks is also separable. We thus attain generalizability similar to general-purpose solvers, while preserving the scalability of DP methods.
In our experiments, we demonstrate the flexibility of STreeD on a variety of optimization tasks, including cost-sensitive classification, prescriptive policy generation, nonlinear classification metrics, and group fairness.
As is commonly done, we assume that features are binarized beforehand. Therefore, STreeD returns optimal binary decision trees under the assumption of a given binarization.
In summary, our main contributions are 1) a generalized DP framework (STreeD) for optimal decision trees that can optimize any separable objective or constraint; 2) a proof for necessary and sufficient conditions for separability; and 3) extensive experiments on five application domains that show the flexibility of STreeD, while performing on par or better than the state of the art.
The following sections introduce related work and preliminaries. We then further define separability, show conditions for separability, and provide a framework that can find optimal decision trees for any separable optimization task. Finally, we provide five diverse example applications and test the performance of the new framework on these applications versus the state-of-the-art.
Decision tree learning Because optimal decision tree search is an NP-Hard problem [38], most traditional solution methods for decision tree learning were heuristics. These heuristics, such as CART [14] and C4.5 [73], often employ a top-down compilation, splitting the tree based on a local information gain or entropy metric. However, for each new objective, new splitting criteria need to be devised, and often this splitting criterion only indirectly relates to the real objective.
With increasing computational power, optimal decision tree search for limited depth has become feasible. Bertsimas and Dunn [10] and Verwer and Zhang [85] showed that optimal trees better capture the information of the dataset than heuristics by directly optimizing the objective at hand, resulting on average in a  $1 - 5\%$  gain in out-of-sample accuracy for trees of similar or smaller size.
However, their MIP models –and also the state-of-the-art MIP models [2, 37, 86, 93]– may often take several hours for medium-sized datasets, even when considering small trees up to depth three. Others have proposed to use SAT [8, 42, 64], MaxSAT [34, 76], and CP [84]. These methods also lack scalability [20], or –in the case of SAT– solve a different problem: finding a minimal tree with no misclassifications. Because scalability is such an issue for these methods, many propose to solve the problem with variants of local search, such as coordinate descent [11, 26], tree alternating optimization [15] and a heuristic SAT approach [75].
In contrast, DP approaches find globally optimal trees while achieving orders of magnitude better scalability [3], so the survey by Costa and Pedreira [18] concludes that specialized DP methods are the most promising future direction for optimal decision trees. Nijssen and Fromont [65] proposed DL8, the first dynamic programming approach for optimal decision trees, which they later generalized to a broader set of objectives and constraints [66]. Aglin et al. [3, 4] improved the scalability of DL8 by adding branch and bound and new caching techniques to the updated algorithm DL8.5. Hu et al. [36] and Lin et al. [52] added sparsity-based pruning and new lower bound techniques. Demirovic et al. [20] improved scalability by adding a special solver for trees of depth two, a similarity lower bound, and other techniques. Recent developments include an anytime algorithm [21, 67] and learning under memory constraints [66].
Objectives and constraints Decision tree learning can be extended by considering a variety of constraints. Nanfack et al. [63] categorized these constraints into structure, attribute, and instance constraints. Structure constraints limit the structure of the tree, such as its depth or size. Attribute constraints impose limits based on the attribute values. Examples are monotonicity constraints [35], cost-sensitive classification [55] and group fairness constraints [44]. Instance constraints operate on
pairs of instances, such as robustness constraints [88], individual fairness constraints [1], and must-or cannot-link constraints in clustering [78]. Among these, structure constraints are most widely included in decision tree learning methods.
Many of these constraints or objectives can be added to MIP models. Therefore, the literature shows a multitude of MIP models for a variety of applications, such as fairness [1], algorithm selection [87], and robustness [88]. However, scalability remains an issue.
Nijssen and Fromont [66] show how the DP-based DL8 algorithm [65] can be generalized for a variety of objectives, provided that the objectives are additive, i.e., the value of an objective in a tree node is equal to the sum of the value in its children. Constraints are required to be anti-monotonic, i.e., a constraint should always be violated for a tree whenever it is violated for any of its subtrees. They consider constraints and objectives such as minimum support in a leaf node, C4.5's estimated error rate, Bayesian probability estimation, cost-sensitive classification, and privacy preservation.
Since then, new DP approaches have improved on the runtime performance of DL8. This, however, reduced the generalizability of those methods. DL8.5 [3, 4], for example, can optimize any additive objective but does not support constraints. MurTree [20] provides new algorithmic techniques that result in much better scalability but only considers accuracy. GOSDT [52] can optimize additive objectives that are linear in the number of false positives and negatives but also does not support constraints. When the objective is nonlinear, they no longer use DP. Demirovic and Stuckey [19] show how DP can be used to optimize even nonlinear functions, provided the objective is monotonically increasing in terms of the false positives and negatives, but they also do not consider constraints.
Summary Decision tree learning is often approached through top-down induction heuristics, which depend on developing complex custom splitting criteria for each new learning task and may perform suboptimally in terms of classification accuracy. Optimal methods often outperform heuristics in accuracy but typically lack scalability. Optimal DP approaches have better scalability but lack generalizability to other objectives and constraints. In the next sections, we address this by pushing the limits of what can be solved by a general yet efficient DP framework for optimal decision trees.
This section introduces notation, defines the problem, and explains how dynamic programming solutions for optimal decision trees work.
Notation and problem definition Let  $\mathcal{F}$  be a set of features and let  $\kappa$  be a set of labels that are used to describe an instance. Let  $\mathcal{D}$  be a dataset consisting of instances  $(x,k)$ , with  $x\in \{0,1\}^{|\mathcal{F}|}$  the feature vector and  $k\in \mathcal{K}$  the label of an instance. Because the features are binary, we introduce the notation  $f$  and  $\bar{f}$  for every feature  $f\in \mathcal{F}$  to denote whether an instance satisfies feature  $f$  or not. In the same way, let  $\mathcal{D}_f$  describe the set of instances in  $\mathcal{D}$  that satisfy feature  $f$ , and  $\mathcal{D}_{\bar{f}}$  the set of instances that does not.
Let  $\tau = (B, L, b, l)$  be a binary tree with  $B$  the internal branching nodes,  $L$  the leaf nodes,  $b: B \to \mathcal{F}$  the assignment of features to branching nodes and  $l: L \to \mathcal{K}$  the assignment of labels to leaf nodes. The left and right child nodes of a branching node  $u \in B$  are given by  $u_{L}$  and  $u_{R}$ , respectively. Instances that satisfy a feature branching test are sent to the right subtree, and the rest to the left.
Then, for example, when minimizing misclassification score (the number of misclassified instances), the cost  $C$  of a decision tree can be computed by the recursive formulation:
$$
\mathrm {C} (\mathcal {D}, u) = \left\{ \begin{array}{l l} \sum_ {(x, k) \in \mathcal {D}} \mathbb {1} (k \neq l (u)) & \text {i f} u \in L \\ \mathrm {C} \left(\mathcal {D} _ {\overline {{b (u)}}}, u _ {L}\right) + \mathrm {C} \left(\mathcal {D} _ {b (u)}, u _ {R}\right) & \text {i f} u \in B \end{array} \right. \tag {1}
$$
The task is to find an optimal decision tree  $\tau$  that minimizes the objective value over the training data given a maximum tree depth  $d$ . We now explain how this can be optimized with DP, and in the next section, we generalize this to any separable decision-tree optimization task.
Dynamic programming DP simplifies a complex problem by reducing it to smaller repeated subproblems for which solutions are cached. For example, minimizing misclassification score with a
binary class can be solved with DP as follows (adapted from [20]):
$$
T (\mathcal {D}, d) = \left\{ \begin{array}{l l} \min  \left\{\left| \mathcal {D} ^ {+} \right|, \left| \mathcal {D} ^ {-} \right| \right\} & d = 0 \\ \min  _ {f \in \mathcal {F}} \left\{T \left(\mathcal {D} _ {f}, d - 1\right) + T \left(\mathcal {D} _ {\bar {f}}, d - 1\right) \right\} & d > 0 \end{array} \right. \tag {2}
$$
This equation computes the minimum possible misclassification score for a dataset  $\mathcal{D}$  and a given maximum tree depth  $d$ . Recursively, as long as  $d > 0$ , a feature  $f$  is selected for branching such that the sum of the misclassification score of the left and right subtree is minimal. In the leaf node (when  $d = 0$ ), the label of the majority class is selected: either the positive  $(\mathcal{D}^{+})$  or negative  $(\mathcal{D}^{-})$  label. This approach is further optimized by using caching and bounds.
As is common with DP notation, Eq. (2) returns the solution value (or cost) and not the solution (the tree). In the remainder of the text, we refer to solution values and solutions interchangeably, with one implying the other and vice versa.
While Eq. (2) yields a single optimal solution, Demirovic and Stuckey [19] investigated the use of DP for nonlinear bi-objective optimization by searching for a Pareto front of optimal solutions: i.e., the set of solutions that are not Pareto dominated  $(\succ)$  by any other solution:
$$
\operatorname {n o n d o m} (\Theta) = \left\{v \in \Theta \mid \neg \exists v ^ {\prime} \in \Theta \left(v ^ {\prime} \succ v\right) \right\} \tag {3}
$$
Consequently, every subtree search also no longer returns just one solution but a Pareto front. For this, they introduce a merge function that combines every solution from one subtree with every solution from the other subtree, resulting in a new Pareto front.
4 Framework for separable objectives
In this section, we generalize previous DP methods for optimal decision trees to a new general framework that can solve any separable optimization task. First, we define the problem. Second, we formalize what is meant by separability in the context of learning decision trees and then we prove necessary and sufficient conditions for optimization tasks to be separable. Third, we present the generalized framework. Finally, we show separable optimization task formulations for four example domains.
4.1 Problem definition and notation
We now generalize the previously introduced problem of minimizing misclassification score to any decision tree optimization task by using common DP notation. Given a state space  $S$  and a solution space  $\mathcal{V}$ , we define:
Definition 4.1 (Optimization task). An optimization task is described by six components:
1. a cost function  $g: \mathcal{S} \times (\mathcal{F} \cup \mathcal{K}) \to \mathcal{V}$  that returns the cost of action  $a \in \mathcal{F} \cup \mathcal{K}$  in state  $s \in \mathcal{S}$ , where the action is either assigning label  $\hat{k} \in \mathcal{K}$  or branching on feature  $f \in \mathcal{F}$ ;
2. a transition function  $t: \mathcal{S} \times \mathcal{F} \times \{0,1\} \to \mathcal{S}$  that provides the next state after branching left or right on feature  $f$ , denoted by  $f$  or  $\bar{f} \in \mathcal{F} \times \{0,1\}$ ;
3. a comparison operator  $\succ : \mathcal{V} \times \mathcal{V} \to \{0,1\}$  that determines Pareto dominance;
4. a combining operator  $\oplus :\mathcal{V}\times \mathcal{V}\to \mathcal{V}$  that combines solution values to one value;
5. a constraint  $c:\mathcal{V}\times \mathcal{S}\to \{0,1\}$  that determines feasibility of a given solution  $v$  and state  $s$
6. and an initial state  $s_0\in S$
For all optimization tasks in this paper, the state  $s$  can be described by the dataset  $\mathcal{D}$  and the branching decisions  $F$  in parent nodes. The transition function is given by  $t(\langle \mathcal{D}, F \rangle, f) = \langle \mathcal{D}_f, F \cup \{f\} \rangle$ , and the initial state  $s_0$  is  $\langle \mathcal{D}, \emptyset \rangle$ . But our framework extends beyond this as well.
When minimizing misclassifications, the cost function is  $g(\langle \mathcal{D}, F \rangle, \hat{k}) = |\{(x, k) \in \mathcal{D} | k \neq \hat{k}\}|$ ; the comparison operator  $\succ$  is  $<$ ; the combining operator  $\oplus$  is addition and the constraint  $c$  returns all solutions as feasible. Similarly, more complex objectives, such as F1-score and group fairness, which do not fit the conditions of DL8 [66], can be defined by using these building blocks. We show four examples in Section 4.4.
Figure 1: An example tree of depth two with five decision variables: two branching decisions F1 and F2, and three leaf node label assignments: L1, L2, and L3.
The final cost of a tree  $\tau = (B, L, b, l)$  with root node  $r$  is now given by calling  $\mathrm{C}(s_0, r)$  on the following recursive function, which generalizes Eq. (1):
$$
\mathrm {C} (s, u) = \left\{ \begin{array}{l l} g (s, l (u)) & \text {i f} u \in L \\ \mathrm {C} (t (s, b (\overline {{u}})), u _ {L}) \oplus \mathrm {C} (t (s, b (u)), u _ {R}) \oplus g (s, b (u)) & \text {i f} u \in B \end{array} \right. \tag {4}
$$
In leaf nodes, the cost of a tree is given by cost function  $g$ . In branching nodes, the combining operator  $\oplus$  combines the cost of the left and right subtree and the branching costs.
Given an optimization task  $o = \langle g, t, \succ, \oplus, c, s_0 \rangle$  and a maximum tree depth  $d$ , the aim is to find a tree (or a Pareto front of trees) that satisfies the constraint  $c$  and optimizes (according to the comparison operator  $\succ$ ) the cost  $C$ .
With this definition of an optimization task, we can generalize several of the concepts introduced in the preliminaries. Let  $\Theta$  describe a set of possible solutions and let
$$
\operatorname {f e a s} (\Theta , s) = \{v \in \Theta \mid c (v, s) = 1 \} \tag {5}
$$
be the subset of all feasible solutions in  $\Theta$  for state  $s$ . The optimal set of solutions is given by
$$
\operatorname {o p t} (\Theta , s) = \operatorname {n o n d o m} (\operatorname {f e a s} (\Theta , s)), \tag {6}
$$
the Pareto front of feasible solutions in  $\Theta$ , with nondom depending on the comparison operator  $\succ$ . Furthermore, we generalize the definition for merge from [19]:
$$
\operatorname {m e r g e} \left(\Theta_ {1}, \Theta_ {2}, s, f\right) = \left\{v _ {1} \oplus v _ {2} \oplus g (s, f) \mid v _ {1} \in \Theta_ {1}, v _ {2} \in \Theta_ {2} \right\} \tag {7}
$$
This returns all combinations of solutions from sets  $\Theta_{1}$  and  $\Theta_{2}$  by using the combining operator  $\oplus$ .
4.2 Separability
An optimization task is separable if optimal solutions to subtrees can be computed independently of other subtrees. For example, in the tree in Fig. 1, the optimal label L3 should be independent of the branching decision F2 and labels L1 and L2. Label L2 should be independent of L1 and L3.
Definition 4.2 (Separable). An optimization task is separable if and only if the optimal solution to any subtree can be determined independently of any of the decision variables that are not part of that subtree or the parent nodes' branching decisions.
The general idea to prove that an optimization task satisfies Def. 4.2 is to use complete induction over the maximum tree depth  $d$ . For the base step ( $d = 0$ ) it is sufficient to require the cost and transition functions to be Markovian: their output should depend only on the current state and the chosen decision. For the induction step ( $d > 0$ ), we need to show that the optimization task satisfies the principle of optimality [9]: that optimal solutions for trees of depth  $d$  can be constructed from only the optimal solutions to its subtrees and the new decision (see Def. A.1). In summary:
Proposition 4.3. An optimization task  $o = \langle g, t, \succ, \oplus, c, s_0 \rangle$  is separable if and only if its cost function  $g$  and transition function  $t$  are Markovian and the task  $o$  satisfies the principle of optimality.
A full proof for this and all subsequent propositions and theorems can be found in Appendix A.
To satisfy the principle of optimality, the combining operator  $\oplus$  must preserve order over  $\succ$ , and the constraint  $c$  must be anti-monotonic. These two notions are explained next.
We introduce the new notion order preservation that guarantees that any combination of optimal solutions using the combining operator  $\oplus$  always dominates a combination with at least one suboptimal solution. Many objectives, such as costs, are additive with  $<$  as the  $\succ$  operator. Addition preserves order over  $<$ . However, additivity is not a necessary condition. We present order preservation instead as a necessary condition (see Appendix A.2).
Definition 4.4 (Order preservation). A combining operator  $\oplus$  preserves order over a given comparator  $\succ$ , if for any given state  $s$ , feature  $f$  and solution sets  $\Theta_1$  and  $\Theta_2$ , with  $s_1 = t(s, f)$ ,  $s_2 = t(s, \bar{f})$ ,  $v_1 \in \mathrm{opt}(\Theta_1, s_1)$ ,  $v_1' \in \Theta_1$  and  $v_2 \in \mathrm{opt}(\Theta_2, s_2)$ , with  $v_1 \succ v_1'$ , then  $v_1 \oplus v_2 \succ v_1' \oplus v_2$  (and  $v_2 \oplus v_1 \succ v_2 \oplus v_1'$ , if  $\oplus$  is not commutative).
To guarantee that applying feas as defined in Eq. (5) does not prune any partial solution that is part of the final optimal solution, constraint  $c$  must be anti-monotonic [66]: if a constraint is violated in a tree, it is also violated in any tree of which this tree is a subtree. However, in order to speak of a necessary condition, we redefine anti-monotonicity to require that any solution which is constructed from at least one subsolution that is infeasible, cannot be an optimal solution.
Definition 4.5 (Anti-monotonic). A constraint  $c$  is anti-monotonic if for any state  $s$ , feature  $f$  and solution sets  $\Theta_1$  and  $\Theta_2$ , with  $s_1 = t(s, f)$ ,  $s_2 = t(s, \bar{f})$ , with  $v_1 \in \Theta_1$  and  $v_2 \in \Theta_2$ , if  $\neg c(v_1, s_1)$  or if  $\neg c(v_2, s_2)$ , then  $v_1 \oplus v_2 \notin \mathrm{opt}(\Theta, s)$ .
Minimum support (minimum leaf node size) is an example of an anti-monotonic constraint [66].
With these conditions and definitions in place, we have the necessary and sufficient conditions for separability and can present the main theoretical result of this paper:
Theorem 4.6. An optimization task  $o = \langle g, t, \succ, \oplus, c, s_0 \rangle$  is separable if and only if its cost function  $g$  and transition function  $t$  are Markovian, its combining operator  $\oplus$  preserves order over its comparison operator  $\succ$  and the constraint  $c$  is anti-monotonic.
4.3 Dynamic programming formulation
We now present the general DP framework  $STreeD$  (Separable Trees with Dynamic programming, pronounced as street):
$$
T (s, d) = \left\{ \begin{array}{l l} \operatorname {o p t} \left(\bigcup_ {\hat {k} \in \mathcal {K}} \left\{g (s, \hat {k}) \right\}, s\right) & d = 0 \\ \operatorname {o p t} \left(\bigcup_ {f \in \mathcal {F}} \operatorname {m e r g e} \left(T (t (s, f), d - 1), T (t (s, \bar {f}), d - 1), s, f\right), s\right) & d > 0 \end{array} \right. \tag {8}
$$