SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Guy Blanc*
gblanc@stanford.edu
Jane Lange*
jlange@mit.edu
Chirag Pabbaraju*
cpabbara@stanford.edu
Colin Sullivan*
colins26@stanford.edu
Li-Yang Tan*
lytan@stanford.edu
Mo Tiwari*
We propose a simple generalization of standard and empirically successful decision tree learning algorithms such as ID3, C4.5, and CART. These algorithms, which have been central to machine learning for decades, are greedy in nature: they grow a decision tree by iteratively splitting on the best attribute. Our algorithm, Top- $k$ , considers the  $k$  best attributes as possible splits instead of just the single best attribute. We demonstrate, theoretically and empirically, the power of this simple generalization. We first prove a greediness hierarchy theorem showing that for every  $k \in \mathbb{N}$ ,  $\operatorname{Top}(-(k + 1)$  can be dramatically more powerful than Top- $k$ : there are data distributions for which the former achieves accuracy  $1 - \varepsilon$ , whereas the latter only achieves accuracy  $\frac{1}{2} + \varepsilon$ . We then show, through extensive experiments, that Top- $k$  outperforms the two main approaches to decision tree learning: classic greedy algorithms and more recent "optimal decision tree" algorithms. On one hand, Top- $k$  consistently enjoys significant accuracy gains over greedy algorithms across a wide range of benchmarks. On the other hand, Top- $k$  is markedly more scalable than optimal decision tree algorithms and is able to handle dataset and feature set sizes that remain far beyond the reach of these algorithms. The code to reproduce our results is available at: https://github.com/SullivanC19/pyd18.5-topk.
Decision trees are a fundamental workhorse in machine learning. Their logical and hierarchical structure makes them easy to understand and their predictions easy to explain. Decision trees are therefore the most canonical example of an interpretable model: in his influential survey [Bre01b], Breiman writes "On interpretability, trees rate an  $\mathrm{A} +$ "; much more recently, the survey [RCC+22] lists decision tree optimization as the very first of 10 grand challenges for the field of interpretable machine learning. Decision trees are also central to modern ensemble methods such as random forests [Bre01a] and XGBoost [CG16], which achieve state-of-the-art accuracy for a wide range of tasks.
Greedy algorithms such as ID3 [Gui86], C4.5 [Gui93], and CART [BFSO84] have long been the standard approach to decision tree learning. These algorithms build a decision tree from labeled data in a top-down manner, growing the tree by iteratively splitting on the "best" attribute as measured with respect to a certain heuristic function (e.g., information gain). Owing to their simplicity, these
*Authors ordered alphabetically.
algorithms are highly efficient and scale gracefully to handle massive datasets and feature set sizes, and they continue to be widely employed in practice and enjoy significant empirical success. For the same reasons, these algorithms are also part of the standard curriculum in introductory machine learning and data science courses.
The trees produced by these greedy algorithms are often reasonably accurate, but can nevertheless be suboptimal. There has therefore been a separate line of work, which we review in Section 2, on algorithms that optimize for accuracy and seek to produce optimally accurate decision trees. These algorithms employ a variety of optimization techniques (including dynamic programming, integer programming, and SAT solvers) and are completely different from the simple greedy algorithms discussed above. Since the problem of finding an optimal decision tree has long been known to be NP-hard [HR76], any algorithm must suffer from the inherent combinatorial explosion when the instance size becomes sufficiently large (unless  $\mathrm{P = NP}$ ). Therefore, while this line of work has made great strides in improving the scalability of algorithms for optimal decision trees, dataset and feature set sizes in the high hundreds and thousands remain out of reach.
This state of affairs raises a natural question:
Can we design decision tree learning algorithms that improve significantly on the accuracy of classic greedy algorithms and yet inherit their simplicity and scalability?
In this work, we propose a new approach and make a case that provides a strong affirmative answer to the question above. Our work also opens up several new avenues for exploration in both the theory and practice of decision tree learning.
1.1.1 Top- $k$ : a simple and effective generalization of classic greedy decision tree algorithms
We introduce an easily interpretable greediness parameter to the class of all greedy decision tree algorithms, a broad class that encompasses ID3, C4.5, and CART. This parameter,  $k$ , represents the number of features that the algorithm considers as candidate splits at each step. Setting  $k = 1$  recovers the fully greedy classical approaches, and increasing  $k$  allows the practitioner to produce more accurate trees at the cost of only a mild training slowdown. The focus of our work is on the regime where  $k$  is a small constant—preserving the efficiency and scalability of greedy algorithms is a primary objective of our work—although we mention here that by setting  $k$  to be the dimension  $d$ , our algorithm produces an optimal tree. Our overall framework can thus be viewed as interpolating between greedy algorithms at one extreme and "optimal decision tree" algorithms at the other, precisely the two main and previously disparate approaches to decision tree learning discussed above.
We will now describe our framework. A feature scoring function  $\mathcal{H}$  takes as input a dataset over  $d$  binary features and a specific feature  $i\in [d]$ , and returns a value quantifying the "desirability" of this feature as the root of the tree. The greedy algorithm corresponding to  $\mathcal{H}$  selects as the root of the tree the feature that has the largest score under  $\mathcal{H}$ ; our generalization will instead consider the  $k$  features with the  $k$  highest scores.
Definition 1 (Feature scoring function). A feature scoring function  $\mathcal{H}$  takes as input a labeled dataset  $S$  over a  $d$ -dimensional feature space, a feature  $i \in [d]$ , and returns a score  $\nu_{i} \in [0,1]$ .
See Section 3.1 for a discussion of the feature scoring functions that correspond to standard greedy algorithms ID3, C4.5, and CART. Pseudocode for Top- $k$  is provided in Figure 1. We note that from the perspective of interpretability, the trained model looks the same regardless of what  $k$  is. During training, the algorithm considers more splits, but only one split is eventually used at each node.
1.1.2 Theoretical results on the power of Top-k
The search space of  $\mathrm{Top} - (k + 1)$  is larger than that of  $\mathrm{Top} - k$ , and therefore its training accuracy is certainly at least as high. The first question we consider is: is the test accuracy of  $\mathrm{Top} - (k + 1)$  only marginally better than that of  $\mathrm{Top} - k$ , or are there examples of data distributions for which even a single additional choice provably leads to huge gains in test accuracy? Our first main theoretical result is a sharp greediness hierarchy theorem, showing that this parameter can have dramatic impacts on accuracy, thereby illustrating its power:
Top-  $k(\mathcal{H},S,h)$
Given: A feature scoring function  $\mathcal{H}$ , a labeled sample set  $S$  over  $d$  dimensions, and depth budget  $h$ .
Output: Decision tree of depth  $h$  that approximately fits  $S$ .
1. If  $h = 0$ , or if every point in  $S$  has the same label, return the constant function with the best accuracy w.r.t.  $S$ .
2. Otherwise, let  $\mathcal{I} \subseteq [d]$  be the set of  $k$  coordinates maximizing  $\mathcal{H}(S,i)$ .
3. For each  $i\in \mathcal{I}$  , let  $T_{i}$  be the tree with
Root  $= x_{i}$
Left subtree  $= \operatorname{Top} - k\left( {\mathcal{H},{S}_{{x}_{i} = 0},h - 1}\right)$
Right subtree  $= \operatorname{Top} - k\left( {\mathcal{H},{S}_{{x}_{i} = 1},h - 1}\right)$  ,
where  $S_{x_i = b}$  is the subset of points in  $S$  where  $x_{i} = b$ .
4. Return the  $T_{i}$  with maximal accuracy with respect to  $S$  among all choices of  $i\in \mathcal{I}$ .
Figure 1: The Top- $k$  algorithm. It can be instantiated with any feature scoring function  $\mathcal{H}$ , and when  $k = 1$ , recovers standard greedy algorithms such as ID3, C4.5, and CART.
Theorem 1 (Greediness hierarchy theorem). For every  $\varepsilon > 0$ ,  $k, h \in \mathbb{N}$ , there is a data distribution  $\mathcal{D}$  and sample size  $n$  for which, with high probability over a random sample  $\pmb{S} \sim \mathcal{D}^n$ ,  $\mathrm{Top} - (k + 1)$  achieves at least  $1 - \varepsilon$  accuracy with a depth budget of  $h$ , but  $\mathrm{Top} - k$  achieves at most  $\frac{1}{2} + \varepsilon$  accuracy with a depth budget of  $h$ .
All of our theoretical results, Theorems 1 to 3, hold whenever the scoring function is an impurity-based heuristic. This broad class includes the most popular scoring functions (see Section 3.1 for more details). Theorem 1 is a special case of a more general result that we show: for all  $k < K$ , there are data distributions on which Top-  $K$  achieves maximal accuracy gains over Top-  $k$ , even if Top-  $k$  is allowed a larger depth budget:
Theorem 2 (Generalization of Theorem 1). For every  $\varepsilon > 0$ ,  $k, K, h \in \mathbb{N}$  where  $k < K$ , there is a data distribution  $\mathcal{D}$  and sample size  $n$  for which, with high probability over a random sample  $S \sim \mathcal{D}^n$ , Top- $K$  achieves at least  $1 - \varepsilon$  accuracy with a depth budget of  $h$ , but Top- $k$  achieves at most  $\frac{1}{2} + \varepsilon$  accuracy even with a depth budget of  $h + (K - k - 1)$ .
The proof of Theorem 2 is simple and highlights the theoretical power of choices. One downside, though, is that it is based on data distributions that are admittedly somewhat unnatural: the labeling function has embedded within it a function that is the XOR of certain features, and real-world datasets are unlikely to exhibit such adversarial structure. To address this, we further prove that the power of choices is evident even for monotone data distributions. We defer the definition of monotone data distributions to Section 4.2.
Theorem 3 (Greediness hierarchy theorem for monotone data distributions). For every  $\varepsilon >0$ , depth budget  $h$ ,  $K$  between  $\tilde{\Omega} (h)$  and  $\tilde{O} (h^2)$  and  $k\leq K - h$ , there is a monotone data distribution  $\mathcal{D}$  and sample size  $n$  for which, with high probability over a random sample  $\pmb {S}\sim \mathcal{D}^n$ , Top- $K$  achieves at least  $1 - \varepsilon$  accuracy with a depth budget of  $h$ , but Top- $k$  achieves at most  $\frac{1}{2} +\varepsilon$  accuracy with a depth budget of  $h$ .
Many real-world data distributions are monotone in nature, and relatedly, they are a common assumption and the subject of intensive study in learning theory. Most relevant to this paper, recent theoretical work has identified monotone data distributions as a broad and natural class for which classical greedy decision tree algorithms (i.e., Top-1) provably succeed [BLT20b, BLT20a]. Theorem 3 shows that even within this class, increasing the greediness parameter can lead to dramatic gains in accuracy. Compared to Theorem 2, the proof of Theorem 3 is more technical and involves the use of concepts from the Fourier analysis of boolean functions [O'D14].
We note that a weaker version of Theorem 3 is implicit in prior work: combining [BLT20b, Theorem 7b] and [BLQT21b, Theorem 2] yields the special case of Theorem 3 where  $K = O(h^2)$  and  $k = 1$ . Theorem 3 is a significant strengthening as it allows for  $k > 1$  and much smaller  $K - k$ .
1.1.3 Experimental results on the power of Top-  $k$
We provide extensive empirical validation of the effectiveness of Top- $k$  when trained on on real-world datasets, and provide an in-depth comparison with both standard greedy algorithms as well as optimal decision tree algorithms.
We first compare the performance of Top-  $k$  for  $k = 1,2,3,4,8,12,16$  (Figure 2), and find that increasing  $k$  does indeed provide a significant increase in test accuracy—in some cases, Top-8 already achieves accuracy comparable to the test accuracy attained by DL8.5 [ANS20], an optimal decision tree algorithm. We further show, in Figures 3 and 6, that Top-  $k$  inherits the efficiency of popular greedy algorithms and scales much better than the state-of-the-art optimal decision tree algorithms MurTree and GOSDT  $\mathrm{[LZH^{+}20]}$ .
Taken as a whole, our experiments demonstrate that Top- $k$  provides a useful middle ground between greedy and optimal decision tree algorithms: it is significantly more accurate than greedy algorithms, but still fast enough to be practical on reasonably large datasets. See Section 5 for an in-depth discussion of our experiments. Finally, we emphasize the benefits afforded by the simplicity of Top- $k$ . Standard greedy algorithms (i.e. Top-1) are widely employed and easily accessible. Introducing the parameter  $k$  requires modifying only a tiny amount of source code and gives the practitioner a new lever to control. Our experiments and theoretical results demonstrate the utility of this simple lever.
Provable guarantees and limitations of greedy decision tree algorithms. A long and fruitful line of work seeks to develop a rigorous understanding of the performances of greedy decision tree learning algorithms such as ID3, C4.5, and CART and to place their empirical success on firm theoretical footing [KM96, Kea96, DKM96, BDM19, BDM20, BLT20b, BLT20a, BLQT21a]. These works identify feature and distributional assumptions under which these algorithms provably succeed; they also highlight the limitations of these algorithms by pointing out settings in which they provably fail. Our work complements this line of work by showing, theoretically and empirically, how these algorithms can be further improved with a simple new parameter while preserving their efficiency and scalability.
The work of [BLQT21b]. Recent work of Blanc, Lange, Qiao, and Tan also highlights the power of choices in decision tree learning. However, they operate within a stylized theoretical setting. First, they consider a specific scoring function that is based on a notion of influence of features, and crucially, computing these scores requires query access to the target function (rather than from random labeled samples as is the case in practice). Furthermore, their results only hold with respect to the uniform distribution. These are strong assumptions that limit the practical relevance of their results. In contrast, a primary focus of this work is to be closely aligned with practice, and in particular, our framework captures and generalizes the standard greedy algorithms used in practice.
Optimal decision trees. Motivated in part by the surge of interest in interpretable machine learning and the highly interpretable nature of decision trees, there have been numerous works on learning optimal decision trees [BD17, VZ17, VZ19, AAV19, ZMP $^{+}$ 20, VNP $^{+}$ 20, NIPMS18, Ave20, JM20, NF07, NF10, HRS19, LZH $^{+}$ 20, DLH $^{+}$ 22]. As mentioned in the introduction, this is an NP-complete problem [HR76]—indeed, it is NP-hard to find even an approximately optimal decision tree [Sie08, AH08, ABF $^{+}$ 09]. Due to the fundamental intractability of this problem, even highly optimized versions of algorithms are unlikely to match the scalability of standard greedy algorithms. That said, these works implement a variety of optimizations that allow them to build optimal decision trees for many real world datasets when the dataset and feature sizes are in the hundreds and the desired depth is small ( $\leq 5$ ).
Finally, another related line of work is that of soft decision trees [IYA12,  $\mathrm{TAA}^{+}19$ ]. These works use gradient-based methods to learn soft splits at each internal node. We believe that one key advantage of our work over these soft trees is in interpretability. With Top- $k$ , since the splits are hard (and not
soft), to understand the classification of a test point, it is sufficient to look at only one root-to-leaf path, as opposed to a weighted combination across many.
3 The Top-  $k$  algorithm
3.1 Background and context: Impurity-based algorithms
Greedy decision tree learning algorithms like ID3, C4.5 and CART are all instantiations of Top- $k$  in Figure 1 with  $k = 1$  and an appropriate choice of the feature-scoring function  $\mathcal{H}$ . Those three algorithms all used impurity-based heuristics as their feature-scoring function:
Definition 2 (Impurity-based heuristic). An impurity function  $\mathcal{G}:[0,1]\to [0,1]$  is a function that is concave, symmetric about 0.5, and satisfies  $\mathcal{G}(0) = \mathcal{G}(1) = 0$  and  $\mathcal{G}(0.5) = 1$ . A feature-scoring function  $\mathcal{H}$  is an impurity-based heuristic, if there is some impurity function  $\mathcal{G}$  for which:
$$
\begin{array}{l} \mathcal {H} (S, i) = \mathcal {G} \left(\underset {\boldsymbol {x}, \boldsymbol {y} \sim S} {\mathbb {E}} [ \boldsymbol {y} ]\right) - \Pr_ {\boldsymbol {x}, \boldsymbol {y} \sim S} [ \boldsymbol {x} _ {i} = 0 ] \cdot \mathcal {G} \left(\underset {\boldsymbol {x}, \boldsymbol {y} \sim S} {\mathbb {E}} [ \boldsymbol {y} \mid \boldsymbol {x} _ {i} = 0 ]\right) \\ - \Pr_ {\boldsymbol {x}, \boldsymbol {y} \sim S} [ \boldsymbol {x} _ {i} = 1 ] \cdot \mathcal {G} \left(\underset {\boldsymbol {x}, \boldsymbol {y} \sim S} {\mathbb {E}} [ \boldsymbol {y} \mid \boldsymbol {x} _ {i} = 1 ]\right) \\ \end{array}
$$
where in each of the above,  $(\pmb{x},\pmb{y})$  are a uniformly random point from within  $S$ .
Common examples for the impurity function include the binary entropy function  $\mathcal{G}(p) = -p\log_2(p) - (1 - p)\log_2(1 - p)$  (used by ID3 and C4.5), the Gini index  $\mathcal{G}(p) = 4p(1 - p)$  (used by CART), and the function  $\mathcal{G}(p) = 2\sqrt{p(1 - p)}$  (proposed and analyzed in [KM99]). We refer the reader to [KM99] for a theoretical comparison, and [DKM96] for an experimental comparison, of these impurity-based heuristics.
Our experiments focus on binary entropy being the impurity measure, but our theoretical results apply to Top- $k$  instantiated with any impurity-based heuristic.
3.2 Basic theoretical properties of the Top- $k$  algorithm
Running time. The key behavioral aspect in which Top- $k$  differs from greedy algorithms is that it is less greedy when trying to determine which coordinate to query. This naturally increases the running time of Top- $k$ , but that increase is fairly mild. More concretely, suppose Top- $k$  is run on a dataset  $S$  with  $n$  points. We can then easily derive the following bound on the running time of Top- $k$ , where  $\mathcal{H}(S,i)$  is assumed to take  $O(n)$  time to evaluate (as it does for all impurity-based heuristics).
Claim 3.1. The running time of  $Top - k(\mathcal{H}, S, h)$  is  $O((2k)^h \cdot nd)$ .
Proof. Let  $T_{h}$  be the number of recursive calls made by Top-  $k(\mathcal{H}, S, h)$ . Then, we have the simple recurrence relation  $T_{h} = 2kT_{h - 1}$ , where  $T_{0} = 1$ . Solving this recurrence gives  $T_{h} = (2k)^{h}$ . Each recursive call takes  $O(nd)$  time, where the bottleneck is scoring each of the  $d$  features.
We note that any decision tree algorithm, including fast greedy algorithms such as ID3, C4.5, and CART, has runtime that scales exponentially with the depth  $h$ . The size of a depth- $h$  tree can be  $2^h$ , and this is of course a lower bound on the runtime as the algorithm needs to output such a tree. In contrast with greedy algorithms (for which  $k = 1$ ), Top- $k$  incurs an additional  $k^h$  cost in running time. As mentioned earlier, in practice, we are primarily concerned with fitting small decision trees (e.g.,  $h = 5$ ) to the data, as this allows for explainable predictions. In this setting, the additional  $k^h$  cost (for small constant  $k$ ) is inexpensive, as confirmed by our experiments.
The search space of Top- $k$ : We state and prove a simple claim that Top- $k$  returns the best tree within its search space.
Definition 3 (Search space of Top- $k$ ). Given a sample  $S$  and integers  $h, k$ , we use  $\mathcal{T}_{k,h,S}$  to refer to all trees in the search space of Top- $k$ . Specifically, if  $h = 0$ , this contains all trees with a height of zero (the constant 0 and constant 1 trees). For  $h \geq 1$ , and  $\mathcal{I} \subseteq [d]$  being the  $k$  coordinates with maximal score, this contains all trees with a root of  $x_i$ , left subtree in  $\mathcal{T}_{k,h-1,S_{x_i}=0}$  and right subtree in  $\mathcal{T}_{k,h-1,S_{x_i}=1}$  for some  $i \in \mathcal{I}$ .
Lemma 3.2 (Top- $k$  chooses the most accurate tree in its search space). For any sample  $S$  and integers  $h, k$ , let  $T$  be the output of Top- $k$  with a depth budget of  $h$  on  $S$ . Then
$$
\operatorname * {P r} _ {\boldsymbol {x}, \boldsymbol {y} \sim S} [ T (\boldsymbol {x}) = \boldsymbol {y} ] = \max  _ {T ^ {\prime} \in \mathcal {T} _ {k, h, S}} \left(\operatorname * {P r} _ {\boldsymbol {x}, \boldsymbol {y} \sim S} [ T ^ {\prime} (\boldsymbol {x}) = \boldsymbol {y} ]\right).
$$
We refer the reader to Appendix A for the proof of this lemma.
4 Theoretical bounds on the power of choices
We refer the reader to the Appendix B for most of the setup and notation. For now, we briefly mention a small amount of notation relevant to this section: we use **bold font** (e.g.  $x$ ) to denote random variables. We also use **bold font** to indicate stochastic functions which output a random variable. For example,
$$
\boldsymbol {f} (x) := \left\{ \begin{array}{l l} x & \text {w i t h p r o b a b i l i t y} \frac {1}{2} \\ - x & \text {w i t h p r o b a b i l i t y} \frac {1}{2} \end{array} \right.
$$
is the stochastic function that returns either the identity or its negation with equal probability. To define the data distributions of Theorems 2 and 3, we will give a distribution over the domain,  $\hat{X}$  and the stochastic function that provides the label given an element of the domain.
Intuition for proof of greediness hierarchy theorem To construct a distribution which Top- $k$  fits poorly and Top- $(k + 1)$  fits well, we will partition features into two groups: one group consisting of features with medium correlation to the labels and another group consisting of features with high correlation when taken all together but low correlation otherwise. Since the correlation of features in the former group is larger than that of the latter group unless all features from the latter group are considered, both algorithms will prioritize features from the former group. However, if the groups are sized correctly, then Top- $(k + 1)$  will consider splitting on all features from the latter group, whereas Top- $k$  will not. As a result, Top- $(k + 1)$  will output a decision tree with higher accuracy.
4.1 Proof of Theorem 2
For each depth budget  $h$  and search branching factor  $K$ , we will define a hard distribution  $\mathcal{D}_{h,K}$  that is learnable to high accuracy by Top-  $K$  with a depth of  $h$ , but not by Top-  $k$  with a depth of  $h'$  for any  $h' < h + K - k$ . This distribution will be over  $\{0,1\}^d \times \{0,1\}$ , where  $d = h + K - 1$ . The marginal distribution over  $\{0,1\}^d$  is uniform, and the distribution over  $\{0,1\}$  conditioned on a setting of the  $d$  features is given by the stochastic function  $f_{h,K}(x)$ . All of the results of this section (Theorems 2 and 3) hold when the feature scoring function is any impurity-based heuristic.
Description of  $f_{h,K}(x)$ . Partition  $x$  into two sets of variables,  $x^{(1)}$  of size  $h$  and  $x^{(2)}$  of size  $K - 1$ . Let  $f_{h,K}(x)$  be the randomized function defined as follows: