SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\boldsymbol {f} _ {h, K} (x) = \left\{ \begin{array}{l l} \operatorname {P a r} _ {h} (x ^ {(1)}) & \text {w i t h p r o b a b i l i t y 1 - \varepsilon} \\ x _ {i} ^ {(2)} \sim \operatorname {U n i f} [ x ^ {(2)} ] & \text {w i t h p r o b a b i l i t y \varepsilon ,} \end{array} \right.
$$
where  $\mathrm{Unif}[x^{(2)}]$  denotes the uniform distribution on  $x^{(2)}$ .  $\operatorname{Par}_h(x^{(1)})$  is the parity function, whose formal definition can be found in Appendix B.
The proof of Theorem 2 is divided into two parts. First, we prove that when the data distribution is  $\mathcal{D}_{h,K}$ , Top-  $K$  succeeds in building a high accuracy tree with a depth budget of  $h$ . Then, we show that Top-  $k$  fails and builds a tree with low accuracy, even given a depth budget of  $h + (K - k - 1)$ .
Lemma 4.1 (Top-  $K$  succeeds). The accuracy of Top-  $K$  with a depth of  $h$  on  $\mathcal{D}_{h,K}$  is at least  $1 - \varepsilon$ .
Lemma 4.2 (Top- $k$  fails). The accuracy of Top- $k$  with a depth of  $h'$  on  $\mathcal{D}_{h,K}$  is at most  $(1/2 + \varepsilon)$  for any  $h' < h + K - k$ .
Proofs of both these lemmas are deferred to Appendix B. Theorem 2 then follows directly from these two lemmas.
4.2 Proof of Theorem 3
In this section, we overview the proof Theorem 3. Some of the proofs are deferred to Appendix B.2.
Before proving Theorem 3, we formalize the concept of monotonicity. For simplicity, we assume the domain is the Boolean cube,  $\{0,1\}^d$ , and use the partial ordering  $x \preceq x'$  iff  $x_i \leq x_i'$  for each  $i \in [d]$ ; however, the below definition easily extends to the domain being any partially ordered set.
Definition 4 (Monotone). A stochastic function,  $\pmb{f}:\{0,1\}^{d}\rightarrow \{0,1\}$ , is monotone if, for any  $x,x^{\prime}\in \{0,1\}^{d}$  where  $x\succeq x^{\prime}$ ,  $\mathbb{E}[\pmb {f}(\pmb {x})]\leq \mathbb{E}[\pmb {f}(\pmb{x}^{\prime})]$ . A data distribution,  $\mathcal{D}$  over  $\{0,1\} ^d\times \{0,1\}$  is said to be monotone if the corresponding stochastic function,  $\pmb {f}(\pmb {x})$  returning  $(\pmb {y}\mid \pmb {x} = x)$  where  $(\pmb {x},\pmb {y})\sim \mathcal{D}$ , is monotone.
To construct the data distribution of Theorem 3, we will combine monotone functions, Majority and Tribes, commonly used in the analysis of Boolean functions due to their extremal properties. See Appendix B.2 for their definitions and useful properties. Let  $d = h + K - 1$ , and the distribution over the domain be uniform over  $\{0,1\}^d$ . Given some  $x \in \{0,1\}^d$ , we use  $x^{(1)}$  to refer to the first  $h$  coordinates of  $x$  and  $x^{(2)}$  the other  $K - 1$  coordinates. This data distribution is labeled by the stochastic function  $f$  given below.
$$
\boldsymbol {f} (x) := \left\{ \begin{array}{l l} \operatorname {T r i b e s} _ {h} (x ^ {(1)}) & \text {w i t h p r o b a b i l i t y} 1 - \varepsilon \\ \operatorname {M a j} _ {K - 1} (x ^ {(2)}) & \text {w i t h p r o b a b i l i t y} \varepsilon . \end{array} \right.
$$
Clearly  $\pmb{f}$  is monotone as it is the mixture of two monotone functions. Throughout this subsection, we'll use  $\mathcal{D}_{h,K}$  to refer to the data distribution over  $\{0,1\}^d \times \{0,1\}$  where to sample  $(\pmb{x},\pmb{y}) \sim \mathcal{D}$ , we first draw  $x \sim \{0,1\}^d$  uniformly and then  $y$  from  $f(x)$ . The proof of Theorem 3 is a direct consequence of the following two Lemmas, both of which we prove in Appendix B.2.
Lemma 4.3 (Top- $K$  succeeds). On the data distribution  $\mathcal{D}_{h,K}$ , Top- $K$  with a depth budget of  $h$  achieves at least  $1 - \varepsilon$  accuracy.
Lemma 4.4 (Top- $k$  fails). On the data distribution  $\mathcal{D}_{h,K}$ , Top- $k$  with a depth budget of  $h$  achieves at most  $\frac{1}{2} + \varepsilon$  accuracy.
Setup for experiments. At all places, the Top-1 tree that we compare to is that given by scikit-learn  $\left[\mathrm{PVG}^{+}11\right]$ , which according to their documentation², is an optimized version of CART. We run experiments on a variety of datasets from the UCI Machine Learning Repository [DG17] (numerical as well as categorical features) having a size in the thousands and having  $\approx 50 - 300$  features after binarization. There were  $\approx 100$  datasets meeting these criteria, and we took a random subset of 20 such datasets. We binarize all the datasets – for categorical datasets, we convert every categorical feature that can take on (say)  $\ell$  values into  $\ell$  binary features. For numerical datasets, we sort and compute thresholds for each numerical attribute, so that the total number of binary features is  $\approx 100$ . A detailed description of the datasets is given in Appendix C.
We build decision trees corresponding to binary entropy as the impurity measure  $\mathcal{H}$ . In order to leverage existing engineering optimizations from state-of-the-art optimal decision tree implementations, we implement the Top- $k$  algorithm given in Figure 1 via simple modifications to the PyDL8.5 [ANS20, ANS21] codebase<sup>3</sup>. Details about this are provided in Appendix D. Our implementation of the Top- $k$  algorithm and other technical details for the experiments are available at https://github.com/SullivanC19/pydl8.5-topk.
5.1 Key experimental findings
Small increments of  $k$  yield significant accuracy gains. Since the search space of Top-  $k$  is a superset of that of Top-1 for any  $k > 1$ , the training accuracy of Top-  $k$  is guaranteed to be larger. The primary objective in this experiment is to show that Top-  $k$  can outperform Top-1 in terms of test accuracy as well. Figure 2 shows the results for Top-1 versus Top-  $k$  for  $k = 2, 3, 4, 8, 12, 16, d$ . Each
$^{2}$ https://scikit-learn.org/stable/modules/tree.html#tree-algorithms-id3-c4-5-c5-0-and-cart
3 https://github.com/aia-uclouvain/pydl8.5
Figure 2: Test accuracy comparison between Top-  $k$  for various values of  $k$ . We can see that Top-  $(k + 1)$  generally obtains higher accuracy than Top-  $k$ , and in some cases (e.g., nursery), Top-8/16's accuracy is even comparable to the optimal tree (Top-  $d$ ). Missing points in the plots correspond to settings that did not terminate within a sufficiently large time limit. All plots are averaged over 10 random train-test splits (except avila and ml-prove that have pre-specified splits) with confidence intervals plotted for 2 standard deviations.
plot is a different dataset, where on the x-axis, we plot the depth of the learned decision tree, and on the y-axis, we plot the test accuracy. Note that  $k = d$  corresponds to the DL8.5 optimal decision tree. We can clearly observe that the test accuracy increases as  $k$  increases—in some cases, the gain is  $> 5\%$  (absolute). Furthermore, for (smaller) datasets like nursery, for which we were able to run  $k = d$ , the accuracy of Top-8/16 is already very close to that of the optimal tree.
Lastly, since Top- $k$  invests more computation towards fitting a better tree on the training set, its training time is naturally longer than Top-1. However, Figure 6 in Appendix E, which plots the training time, shows that the slowdown is mild.
Top- $k$  scales much better than optimal decision tree algorithms. Optimal decision tree algorithms suffer from poor runtime scaling. We empirically demonstrate that, in comparison, Top- $k$  has a significantly better scaling in training time. Our experiments are identical to those in Figures 14 and 15 in the GOSDT paper [LZH $^{+}$ 20], where two notions of scalability are considered. In the first experiment, we fix the number of samples and gradually increase the number of features to train the decision tree. In the second experiment, we include all the features, but gradually increase the number of training samples. The dataset we use is the FICO [FGI $^{+}$ 18] dataset, which has a total of 1000 samples with 1407 binary features. We plot the training time (in seconds) versus number of features/samples for optimal decision tree algorithms (MurTree, GOSDT) and Top- $k$  in Figure 3. We do this for depth  $= 4, 5, 6$  (for GOSDT, the regularization coefficient  $\lambda$  is set to  $2^{-\text{depth}}$ ). We observe that the training time for both MurTree and GOSDT increases dramatically compared to Top- $k$ , in both experiments. In particular, for depth  $= 5$ , both MurTree and GOSDT were unable to build a tree on 300 features within the time limit of 10 minutes, while Top-16 completed execution even with all 1407 features. Similarly, in the latter experiment, GOSDT/MurTree were unable to build a depth-5 tree on 150 samples within the time limit, while Top-16 comfortably finished execution even on 1000 samples. These experiments demonstrate the scalability issues with optimal tree algorithms. Coupled with the accuracy gains seen in the previous experiment, Top- $k$  can thus be seen as achieving a more favorable tradeoff between training time and accuracy.
We note, however, that various optimization have been proposed to allow optimal decision tree algorithms to scale to larger datasets. For example, a more recent version of GOSDT has integrated a guessing strategy using reference ensembles which guides the binning of continuous features, tree
(a) Depth  $= 4$
(d) Depth  $= 4$
(b) Depth  $= 5$
(e) Depth  $= 5$
(c) Depth  $= 6$
(f) Depth  $= 6$
Figure 3: Training time comparison between Top-  $k$  and optimal tree algorithms. As the number of features/samples increases, both GOSDT and MurTree scale poorly compared to Top-  $k$ , and beyond a threshold, do not complete execution within the time limit.
Figure 4: Test accuracy plateaus for large  $k$ . All runs averaged over 10 random train-test splits with maximum depth fixed to 3.
size, and search  $\left[\mathrm{MZA}^{+}\right]$ . Many of these optimizations are generally applicable across optimal tree algorithms and could be combined with Top- $k$  for further improvement in performance.
Increasing  $k$  beyond a point does not improve test accuracy. In our experiments above, we ran Top- $k$  only till  $k = 16$ : in Figure 4, we show that increasing  $k$  to very large values, which increases runtime, often does not improve test accuracy, and in some cases, may even hurt due to overfitting. For 3 datasets - car, hayes-roth and tic-tac-toe - we plot train and test error as a function of  $k$ . Naturally, the train accuracy monotonically increases with  $k$  in each plot. However, for both car and hayes-roth, we can observe that the test accuracy first increases and then plateaus. Interestingly, for tic-tac-toe, the test accuracy first increases and then decreases as we increase  $k$ . These experiments demonstrate that selecting too large of a  $k$ , as optimal decision tree algorithms do, is a waste of computational resources and can even hurt test accuracy via overfitting.
We have shown how popular and empirically successful greedy decision tree learning algorithms can be improved with the power of choices: our generalization, Top- $k$ , considers the  $k$  best features as candidate splits instead of just the single best one. As our theoretical and empirical results demonstrate, this simple generalization is powerful and enables significant accuracy gains while preserving the efficiency and scalability of standard greedy algorithms. Indeed, we find it surprising that such a simple generalization has not been considered before.
There is much more to be explored and understood, both theoretically and empirically; we list here a few concrete directions that we find particularly exciting and promising. First, we suspect that power
of choices affords more advantages over greedy algorithms than just accuracy gains. For example, an avenue for future work is to show that the trees grown by Top- $k$  are more robust to noise. Second, are there principled approaches to the automatic selection of the greediness parameter  $k$ ? Can the optimal choice be inferred from a few examples or learned over time? This opens up the possibility of new connections to machine-learned advice and algorithms with predictions [MV20], an area that has seen a surge of interest in recent years. Finally, as mentioned in the introduction, standard greedy decision tree algorithms are at the very heart of modern tree-based ensemble methods such as XGBoost and random forests. A natural next step is to combine these algorithms with Top- $k$  and further extend the power of choices to these settings.
We thank the NeurIPS reviewers and AC for their detailed and helpful feedback.
Guy and Li-Yang are supported by NSF awards 1942123, 2211237, 2224246 and a Google Research Scholar award. Jane is supported by NSF Graduate Research Fellowship under Grant No. 2141064, NSF Awards CCF-2006664, DMS-2022448, and Microsoft. Mo is supported by a Stanford Interdisciplinary Graduate Fellowship and a Stanford Data Science Scholarship. Chirag is supported by Moses Charikar and Greg Valiant's Simons Investigator Awards.
[AAV19] Sina Aghaei, Mohammad Javad Azizi, and Phebe Vayanos. Learning optimal and fair decision trees for non-discriminative decision-making. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1418-1426, 2019.
[ABF+09] Misha Alekhnovich, Mark Braverman, Vitaly Feldman, Adam Klivans, and Toniann Pitassi. The complexity of properly learning simple concept classes. Journal of Computer & System Sciences, 74(1):16-34, 2009.
[AH08] Micah Adler and Brent Heeringa. Approximating optimal binary decision trees. In Approximation, Randomization and Combinatorial Optimization. Algorithms and Techniques, pages 1-9. Springer, 2008.
[ANS20] Gael Aglin, Siegfried Nijssen, and Pierre Schaus. Learning optimal decision trees using caching branch-and-bound search. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 3146-3153, 2020.
[ANS21] Gael Aglin, Siegfried Nijssen, and Pierre Schaus. Pydl8. 5: a library for learning optimal decision trees. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 5222-5224, 2021.
[Ave20] Florent Avellaneda. Efficient inference of optimal decision trees. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3195-3202, 2020.
[BD17] Dimitris Bertsimas and Jack Dunn. Optimal classification trees. Machine Learning, 106(7):1039-1082, 2017.
[BDM19] Alon Brutzkus, Amit Daniely, and Eran Malach. On the Optimality of Trees Generated by ID3. ArXiv, abs/1907.05444, 2019.
[BDM20] Alon Brutzkus, Amit Daniely, and Eran Malach. ID3 learns juntas for smoothed product distributions. In Proceedings of the 33rd Annual Conference on Learning Theory (COLT), pages 902-915, 2020.
[BFSO84] Leo Breiman, Jerome Friedman, Charles Stone, and Richard Olshen. Classification and regression trees. Wadsworth International Group, 1984.
[BLQT21a] Guy Blanc, Jane Lange, Mingda Qiao, and Li-Yang Tan. Decision tree heuristics can fail, even in the smoothed setting. In Mary Wootters and Laura Sanità, editors, Proceedings of the 25th International Conference on Randomization and Computation (RANDOM), volume 207, pages 45:1-45:16, 2021.
[VZ19] Sicco Verwer and Yingqian Zhang. Learning optimal classification trees using a binary linear program formulation. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 1625-1632, 2019.
$\left[\mathrm{ZMP}^{+}20\right]$  Haoran Zhu, Pavankumar Murali, Dzung Phan, Lam Nguyen, and Jayant Kalagnanam. A scalable MIP-based method for learning optimal multivariate decision trees. Advances in Neural Information Processing Systems, 33:1771-1781, 2020.