Title: Harnessing the power of choices in decision tree learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Decision Trees

Decision Tree Learning

Top-$k$

ID3

Greedy Algorithms
Abstract: We propose a simple generalization of standard and empirically successful decision tree learning algorithms such as ID3, C4.5, and CART.   These algorithms, which have been central to machine learning for decades, are greedy in nature: they grow a decision tree by iteratively splitting on the best attribute.  Our algorithm, Top-$k$, considers the $k$ best attributes as possible splits instead of just the single best attribute. We demonstrate, theoretically and empirically, the power of this simple generalization.  We first prove a greediness hierarchy theorem showing that for every $k\in \mathbb{N}$, Top-$(k+1)$  can be dramatically more powerful than Top-$k$: there are data distributions for which the former achieves accuracy $1-\epsilon$, whereas the latter only achieves accuracy $\frac{1}{2}+\epsilon$.  We then show, through extensive experiments, that Top-$k$ outperforms the two main approaches to decision tree learning: classic greedy algorithms and more recent ``optimal decision tree'' algorithms.  On one hand, Top-$k$ consistently enjoys significant accuracy gains over greedy algorithms across a wide range of benchmarks.  On the other hand, Top-$k$ is markedly more scalable than optimal decision tree algorithms and is able to handle dataset and feature set sizes that remain far beyond the reach of these algorithms. The code to reproduce our results is available at https://github.com/SullivanC19/pydl8.5-topk.

Harnessing the Power of Choices in Decision Tree Learning

Abstract

We propose a simple generalization of standard and empirically successful decision tree learning algorithms such as ID3, C4.5, and CART. These algorithms, which have been central to machine learning for decades, are greedy in nature: they grow a decision tree by iteratively splitting on the best attribute. Our algorithm, Topk , considers the k best attributes as possible splits instead of just the single best attribute.We demonstrate, theoretically and empirically, the power of this simple generalization. We first prove a greediness hierarchy theorem showing that for every k ∈ N , Top-( k +1) can be dramatically more powerful than Topk : there are data distributions for which the former achieves accuracy 1 -ε , whereas the latter only achieves accuracy 1 2 + ε . We then show, through extensive experiments, that Topk outperforms the two main approaches to decision tree learning: classic greedy algorithms and more recent 'optimal decision tree' algorithms. On one hand, Topk consistently enjoys significant accuracy gains over greedy algorithms across a wide range of benchmarks. On the other hand, Topk is markedly more scalable than optimal decision tree algorithms and is able to handle dataset and feature set sizes that remain far beyond the reach of these algorithms. The code to reproduce our results is available at: https://github.com/SullivanC19/pydl8.5-topk .

1 Introduction

Decision trees are a fundamental workhorse in machine learning. Their logical and hierarchical structure makes them easy to understand and their predictions easy to explain. Decision trees are therefore the most canonical example of an interpretable model: in his influential survey [Bre01b], Breiman writes 'On interpretability, trees rate an A+'; much more recently, the survey [RCC + 22] lists decision tree optimization as the very first of 10 grand challenges for the field of interpretable machine learning. Decision trees are also central to modern ensemble methods such as random forests [Bre01a] and XGBoost [CG16], which achieve state-of-the-art accuracy for a wide range of tasks.

Greedy algorithms such as ID3 [Qui86], C4.5 [Qui93], and CART [BFSO84] have long been the standard approach to decision tree learning. These algorithms build a decision tree from labeled data in a top-down manner, growing the tree by iteratively splitting on the 'best' attribute as measured with respect to a certain heuristic function (e.g., information gain). Owing to their simplicity, these

algorithms are highly efficient and scale gracefully to handle massive datasets and feature set sizes, and they continue to be widely employed in practice and enjoy significant empirical success. For the same reasons, these algorithms are also part of the standard curriculum in introductory machine learning and data science courses.

The trees produced by these greedy algorithms are often reasonably accurate, but can nevertheless be suboptimal. There has therefore been a separate line of work, which we review in Section 2, on algorithms that optimize for accuracy and seek to produce optimally accurate decision trees. These algorithms employ a variety of optimization techniques (including dynamic programming, integer programming, and SAT solvers) and are completely different from the simple greedy algorithms discussed above. Since the problem of finding an optimal decision tree has long been known to be NP-hard [HR76], any algorithm must suffer from the inherent combinatorial explosion when the instance size becomes sufficiently large (unless P = NP). Therefore, while this line of work has made great strides in improving the scalability of algorithms for optimal decision trees, dataset and feature set sizes in the high hundreds and thousands remain out of reach.

This state of affairs raises a natural question:

Can we design decision tree learning algorithms that improve significantly on the accuracy of classic greedy algorithms and yet inherit their simplicity and scalability?

In this work, we propose a new approach and make a case that provides a strong affirmative answer to the question above. Our work also opens up several new avenues for exploration in both the theory and practice of decision tree learning.

1.1 Our contributions

1.1.1 Topk : a simple and effective generalization of classic greedy decision tree algorithms

We introduce an easily interpretable greediness parameter to the class of all greedy decision tree algorithms, a broad class that encompasses ID3, C4.5, and CART. This parameter, k , represents the number of features that the algorithm considers as candidate splits at each step. Setting k = 1 recovers the fully greedy classical approaches, and increasing k allows the practitioner to produce more accurate trees at the cost of only a mild training slowdown. The focus of our work is on the regime where k is a small constant-preserving the efficiency and scalability of greedy algorithms is a primary objective of our work-although we mention here that by setting k to be the dimension d , our algorithm produces an optimal tree. Our overall framework can thus be viewed as interpolating between greedy algorithms at one extreme and 'optimal decision tree' algorithms at the other, precisely the two main and previously disparate approaches to decision tree learning discussed above.

We will now describe our framework. A feature scoring function H takes as input a dataset over d binary features and a specific feature i ∈ [ d ] , and returns a value quantifying the 'desirability' of this feature as the root of the tree. The greedy algorithm corresponding to H selects as the root of the tree the feature that has the largest score under H ; our generalization will instead consider the k features with the k highest scores.

Definition 1 (Feature scoring function) . A feature scoring function H takes as input a labeled dataset S over a d -dimensional feature space, a feature i ∈ [ d ] , and returns a score ν i ∈ [0 , 1] .

See Section 3.1 for a discussion of the feature scoring functions that correspond to standard greedy algorithms ID3, C4.5, and CART. Pseudocode for Topk is provided in Figure 1. We note that from the perspective of interpretability, the trained model looks the same regardless of what k is. During training, the algorithm considers more splits, but only one split is eventually used at each node.

1.1.2 Theoretical results on the power of Topk

The search space of Top-( k +1) is larger than that of Topk , and therefore its training accuracy is certainly at least as high. The first question we consider is: is the test accuracy of Top-( k +1) only marginally better than that of Topk , or are there examples of data distributions for which even a single additional choice provably leads to huge gains in test accuracy? Our first main theoretical result is a sharp greediness hierarchy theorem , showing that this parameter can have dramatic impacts on accuracy, thereby illustrating its power:

Given: A feature scoring function H , a labeled sample set S over d dimensions, and depth budget h .

Output: Decision tree of depth h that approximately fits S .

If h = 0 , or if every point in S has the same label, return the constant function with the best accuracy w.r.t. S .

Otherwise, let I ⊆ [ d ] be the set of k coordinates maximizing H ( S, i ) .

For each i ∈ I , let T i be the tree with

where S x i = b is the subset of points in S where x i = b .

Return the T i with maximal accuracy with respect to S among all choices of i ∈ I .

Theorem 1 (Greediness hierarchy theorem) . For every ε > 0 , k, h ∈ N , there is a data distribution D and sample size n for which, with high probability over a random sample S ∼ D n , Top -( k +1) achieves at least 1 -ε accuracy with a depth budget of h , but Top -k achieves at most 1 2 + ε accuracy with a depth budget of h .

All of our theoretical results, Theorems 1 to 3, hold whenever the scoring function is an impuritybased heuristic . This broad class includes the most popular scoring functions (see Section 3.1 for more details). Theorem 1 is a special case of a more general result that we show: for all k < K , there are data distributions on which TopK achieves maximal accuracy gains over Topk , even if Topk is allowed a larger depth budget:

Theorem 2 (Generalization of Theorem 1) . For every ε > 0 , k, K, h ∈ N where k < K , there is a data distribution D and sample size n for which, with high probability over a random sample S ∼ D n , Top -K achieves at least 1 -ε accuracy with a depth budget of h , but Top -k achieves at most 1 2 + ε accuracy even with a depth budget of h +( K -k -1) .

The proof of Theorem 2 is simple and highlights the theoretical power of choices. One downside, though, is that it is based on data distributions that are admittedly somewhat unnatural: the labeling function has embedded within it a function that is the XOR of certain features, and real-world datasets are unlikely to exhibit such adversarial structure. To address this, we further prove that the power of choices is evident even for monotone data distributions. We defer the definition of monotone data distributions to Section 4.2.

Theorem 3 (Greediness hierarchy theorem for monotone data distributions) . For every ε > 0 , depth budget h , K between ˜ Ω( h ) and ˜ O ( h 2 ) and k ≤ K -h , there is a monotone data distribution D and sample size n for which, with high probability over a random sample S ∼ D n , TopK achieves at least 1 -ε accuracy with a depth budget of h , but Topk achieves at most 1 2 + ε accuracy with a depth budget of h .

Many real-world data distributions are monotone in nature, and relatedly, they are a common assumption and the subject of intensive study in learning theory. Most relevant to this paper, recent theoretical work has identified monotone data distributions as a broad and natural class for which classical greedy decision tree algorithms (i.e., Top1 ) provably succeed [BLT20b, BLT20a]. Theorem 3 shows that even within this class, increasing the greediness parameter can lead to dramatic gains in accuracy. Compared to Theorem 2, the proof of Theorem 3 is more technical and involves the use of concepts from the Fourier analysis of boolean functions [O'D14].

We note that a weaker version of Theorem 3 is implicit in prior work: combining [BLT20b, Theorem 7b] and [BLQT21b, Theorem 2] yields the special case of Theorem 3 where K = O ( h 2 ) and k = 1 . Theorem 3 is a significant strengthening as it allows for k > 1 and much smaller K -k .

1.1.3 Experimental results on the power of Topk

We provide extensive empirical validation of the effectiveness of Topk when trained on on real-world datasets, and provide an in-depth comparison with both standard greedy algorithms as well as optimal decision tree algorithms.

We first compare the performance of Topk for k = 1 , 2 , 3 , 4 , 8 , 12 , 16 (Figure 2), and find that increasing k does indeed provide a significant increase in test accuracy-in some cases, Top8 already achieves accuracy comparable to the test accuracy attained by DL8.5 [ANS20], an optimal decision tree algorithm. We further show, in Figures 3 and 6, that Topk inherits the efficiency of popular greedy algorithms and scales much better than the state-of-the-art optimal decision tree algorithms MurTree and GOSDT [LZH + 20].

Taken as a whole, our experiments demonstrate that Topk provides a useful middle ground between greedy and optimal decision tree algorithms: it is significantly more accurate than greedy algorithms, but still fast enough to be practical on reasonably large datasets. See Section 5 for an in-depth discussion of our experiments. Finally, we emphasize the benefits afforded by the simplicity of Topk . Standard greedy algorithms (i.e. Top1 ) are widely employed and easily accessible. Introducing the parameter k requires modifying only a tiny amount of source code and gives the practitioner a new lever to control. Our experiments and theoretical results demonstrate the utility of this simple lever.

2 Related work

Provable guarantees and limitations of greedy decision tree algorithms. A long and fruitful line of work seeks to develop a rigorous understanding of the performances of greedy decision tree learning algorithms such as ID3, C4.5, and CART and to place their empirical success on firm theoretical footing [KM96, Kea96, DKM96, BDM19, BDM20, BLT20b, BLT20a, BLQT21a]. These works identify feature and distributional assumptions under which these algorithms provably succeed; they also highlight the limitations of these algorithms by pointing out settings in which they provably fail. Our work complements this line of work by showing, theoretically and empirically, how these algorithms can be further improved with a simple new parameter while preserving their efficiency and scalability.

The work of [BLQT21b]. Recent work of Blanc, Lange, Qiao, and Tan also highlights the power of choices in decision tree learning. However, they operate within a stylized theoretical setting. First, they consider a specific scoring function that is based on a notion of influence of features, and crucially, computing these scores requires query access to the target function (rather than from random labeled samples as is the case in practice). Furthermore, their results only hold with respect to the uniform distribution. These are strong assumptions that limit the practical relevance of their results. In contrast, a primary focus of this work is to be closely aligned with practice, and in particular, our framework captures and generalizes the standard greedy algorithms used in practice.

Optimal decision trees. Motivated in part by the surge of interest in interpretable machine learning and the highly interpretable nature of decision trees, there have been numerous works on learning optimal decision trees [BD17, VZ17, VZ19, AAV19, ZMP + 20, VNP + 20, NIPMS18, Ave20, JM20, NF07, NF10, HRS19, LZH + 20, DLH + 22]. As mentioned in the introduction, this is an NP-complete problem [HR76]-indeed, it is NP-hard to find even an approximately optimal decision tree [Sie08, AH08, ABF + 09]. Due to the fundamental intractability of this problem, even highly optimized versions of algorithms are unlikely to match the scalability of standard greedy algorithms. That said, these works implement a variety of optimizations that allow them to build optimal decision trees for many real world datasets when the dataset and feature sizes are in the hundreds and the desired depth is small ( ≤ 5 ).

Finally, another related line of work is that of soft decision trees [IYA12, TAA + 19]. These works use gradient-based methods to learn soft splits at each internal node. We believe that one key advantage of our work over these soft trees is in interpretability. With Topk , since the splits are hard (and not

soft), to understand the classification of a test point, it is sufficient to look at only one root-to-leaf path, as opposed to a weighted combination across many.

3 The Topk algorithm

3.1 Background and context: Impurity-based algorithms

Greedy decision tree learning algorithms like ID3, C4.5 and CART are all instantiations of Topk in Figure 1 with k = 1 and an appropriate choice of the feature-scoring function H . Those three algorithms all used impurity-based heuristics as their feature-scoring function:

Definition 2 (Impurity-based heuristic) . An impurity function G : [0 , 1] → [0 , 1] is a function that is concave, symmetric about 0 . 5 , and satisfies G (0) = G (1) = 0 and G (0 . 5) = 1 . A feature-scoring function H is an impurity-based heuristic , if there is some impurity function G for which:

where in each of the above, ( x , y ) are a uniformly random point from within S .

Common examples for the impurity function include the binary entropy function G ( p ) = -p log 2 ( p ) -(1 -p ) log 2 (1 -p ) (used by ID3 and C4.5), the Gini index G ( p ) = 4 p (1 -p ) (used by CART), and the function G ( p ) = 2 √ p (1 -p ) (proposed and analyzed in [KM99]). We refer the reader to [KM99] for a theoretical comparison, and [DKM96] for an experimental comparison, of these impurity-based heuristics.

Our experiments focus on binary entropy being the impurity measure, but our theoretical results apply to Topk instantiated with any impurity-based heuristic.

3.2 Basic theoretical properties of the Topk algorithm

Running time. The key behavioral aspect in which Topk differs from greedy algorithms is that it is less greedy when trying to determine which coordinate to query. This naturally increases the running time of Topk , but that increase is fairly mild. More concretely, suppose Topk is run on a dataset S with n points. We can then easily derive the following bound on the running time of Topk , where H ( S, i ) is assumed to take O ( n ) time to evaluate (as it does for all impurity-based heuristics).

Claim 3.1. The running time of Topk ( H , S, h ) is O ((2 k ) h · nd ) .

Proof. Let T h be the number of recursive calls made by Topk ( H , S, h ) . Then, we have the simple recurrence relation T h = 2 kT h -1 , where T 0 = 1 . Solving this recurrence gives T h = (2 k ) h . Each recursive call takes O ( nd ) time, where the bottleneck is scoring each of the d features.

We note that any decision tree algorithm, including fast greedy algorithms such as ID3, C4.5, and CART, has runtime that scales exponentially with the depth h . The size of a depthh tree can be 2 h , and this is of course a lower bound on the runtime as the algorithm needs to output such a tree. In contrast with greedy algorithms (for which k = 1 ), Topk incurs an additional k h cost in running time. As mentioned earlier, in practice, we are primarily concerned with fitting small decision trees (e.g., h = 5 ) to the data, as this allows for explainable predictions. In this setting, the additional k h cost (for small constant k ) is inexpensive, as confirmed by our experiments.

The search space of Topk : We state and prove a simple claim that Topk returns the best tree within its search space.

Definition 3 (Search space of Topk ) . Given a sample S and integers h, k , we use T k,h,S to refer to all trees in the search space of Topk . Specifically, if h = 0 , this contains all trees with a height of zero (the constant 0 and constant 1 trees). For h ≥ 1 , and I ⊆ [ d ] being the k coordinates with maximal score, this contains all trees with a root of x i , left subtree in T k,h -1 ,S x i =0 and right subtree in T k,h -1 ,S x i =1 for some i ∈ I .

Lemma 3.2 (Topk chooses the most accurate tree in its search space) . For any sample S and integers h, k , let T be the output of Topk with a depth budget of h on S . Then

We refer the reader to Appendix A for the proof of this lemma.

4 Theoretical bounds on the power of choices

We refer the reader to the Appendix B for most of the setup and notation. For now, we briefly mention a small amount of notation relevant to this section: we use bold font (e.g. x ) to denote random variables. We also use bold font to indicate stochastic functions which output a random variable. For example,

is the stochastic function that returns either the identity or its negation with equal probability. To define the data distributions of Theorems 2 and 3, we will give a distribution over the domain, X and the stochastic function that provides the label given an element of the domain.

Intuition for proof of greediness hierarchy theorem To construct a distribution which Topk fits poorly and Top-( k +1) fits well, we will partition features into two groups: one group consisting of features with medium correlation to the labels and another group consisting of features with high correlation when taken all together but low correlation otherwise. Since the correlation of features in the former group is larger than that of the latter group unless all features from the latter group are considered, both algorithms will prioritize features from the former group. However, if the groups are sized correctly, then Top-( k +1) will consider splitting on all features from the latter group, whereas Topk will not. As a result, Top-( k +1) will output a decision tree with higher accuracy.

4.1 Proof of Theorem 2

For each depth budget h and search branching factor K , we will define a hard distribution D h,K that is learnable to high accuracy by TopK with a depth of h , but not by Topk with a depth of h ′ for any h ′ < h + K -k . This distribution will be over { 0 , 1 } d ×{ 0 , 1 } , where d = h + K -1 . The marginal distribution over { 0 , 1 } d is uniform, and the distribution over { 0 , 1 } conditioned on a setting of the d features is given by the stochastic function f h,K ( x ) . All of the results of this section (Theorems 2 and 3) hold when the feature scoring function is any impurity-based heuristic.

Description of f h,K ( x ) . Partition x into two sets of variables, x (1) of size h and x (2) of size K -1 . Let f h,K ( x ) be the randomized function defined as follows:

where Unif[ x (2) ] denotes the uniform distribution on x (2) . Par h ( x (1) ) is the parity function, whose formal definition can be found in Appendix B.

The proof of Theorem 2 is divided into two parts. First, we prove that when the data distribution is D h,K , TopK succeeds in building a high accuracy tree with a depth budget of h . Then, we show that Topk fails and builds a tree with low accuracy, even given a depth budget of h +( K -k -1)

. Lemma 4.1 (TopK succeeds) . The accuracy of TopK with a depth of h on D h,K is at least 1 -ε . Lemma 4.2 (Topk fails) . The accuracy of Topk with a depth of h ′ on D h,K is at most (1 / 2 + ε ) for any h ′ < h + K -k .

Proofs of both these lemmas are deferred to Appendix B. Theorem 2 then follows directly from these two lemmas.

4.2 Proof of Theorem 3

In this section, we overview the proof Theorem 3. Some of the proofs are deferred to Appendix B.2.

Before proving Theorem 3, we formalize the concept of monotonicity. For simplicity, we assume the domain is the Boolean cube, { 0 , 1 } d , and use the partial ordering x ⪯ x ′ iff x i ≤ x ′ i for each i ∈ [ d ] ; however, the below definition easily extends to the domain being any partially ordered set.

Definition 4 (Monotone) . A stochastic function, f : { 0 , 1 } d → { 0 , 1 } , is monotone if, for any x, x ′ ∈ { 0 , 1 } d where x ⪯ x ′ , E [ f ( x )] ≤ E [ f ( x ′ )] . A data distribution, D over { 0 , 1 } d ×{ 0 , 1 } is said to be monotone if the corresponding stochastic function, f ( x ) returning ( y | x = x ) where ( x , y ) ∼ D , is monotone.

To construct the data distribution of Theorem 3, we will combine monotone functions, Majority and Tribes, commonly used in the analysis of Boolean functions due to their extremal properties. See Appendix B.2 for their definitions and useful properties. Let d = h + K -1 , and the distribution over the domain be uniform over { 0 , 1 } d . Given some x ∈ { 0 , 1 } d , we use x (1) to refer to the first h coordinates of x and x (2) the other K -1 coordinates. This data distribution is labeled by the stochastic function f given below.

Clearly f is monotone as it is the mixture of two monotone functions. Throughout this subsection, we'll use D h,K to refer to the data distribution over { 0 , 1 } d ×{ 0 , 1 } where to sample ( x , y ) ∼ D , we first draw x ∼ { 0 , 1 } d uniformly and then y from f ( x ) . The proof of Theorem 3 is a direct consequence of the following two Lemmas, both of which we prove in Appendix B.2.

Lemma 4.3 (TopK succeeds) . On the data distribution D h,K , TopK with a depth budget of h achieves at least 1 -ε accuracy.

Lemma 4.4 (Topk fails) . On the data distribution D h,K , Topk with a depth budget of h achieves at most 1 2 + ε accuracy.

5 Experiments

Setup for experiments. At all places, the Top1 tree that we compare to is that given by scikit-learn [PVG + 11], which according to their documentation 2 , is an optimized version of CART. We run experiments on a variety of datasets from the UCI Machine Learning Repository [DG17] (numerical as well as categorical features) having a size in the thousands and having ≈ 50 -300 features after binarization. There were ≈ 100 datasets meeting these criteria, and we took a random subset of 20 such datasets. We binarize all the datasets - for categorical datasets, we convert every categorical feature that can take on (say) ℓ values into ℓ binary features. For numerical datasets, we sort and compute thresholds for each numerical attribute, so that the total number of binary features is ≈ 100 . A detailed description of the datasets is given in Appendix C.

We build decision trees corresponding to binary entropy as the impurity measure H . In order to leverage existing engineering optimizations from state-of-the-art optimal decision tree implementations, we implement the Topk algorithm given in Figure 1 via simple modifications to the PyDL8.5 [ANS20, ANS21] codebase 3 . Details about this are provided in Appendix D. Our implementation of the Topk algorithm and other technical details for the experiments are available at https://github.com/SullivanC19/pydl8.5-topk .

5.1 Key experimental findings

Small increments of k yield significant accuracy gains. Since the search space of Topk is a superset of that of Top1 for any k > 1 , the training accuracy of Topk is guaranteed to be larger. The primary objective in this experiment is to show that Topk can outperform Top1 in terms of test accuracy as well. Figure 2 shows the results for Top1 versus Topk for k = 2 , 3 , 4 , 8 , 12 , 16 , d . Each

3

4

5

6

7

8

Depth

0.825

0.850

0.875

0.900

0.925

0.950

0.975

Test accuracy

nursery

3

4

5

6

7

8

Depth

0.9825

0.9850

0.9875

0.9900

0.9925

0.9950

0.9975

Test accuracy

occupancy-estimation

3

4

5

6

7

8

Depth

0.44

0.46

0.48

0.50

0.52

Test accuracy

ml-prove

3

4

5

6

7

8

Depth

0.3

0.4

0.5

0.6

Test accuracy

artificial-characters

3

4

5

6

7

8

Depth

0.84

0.86

0.88

0.90

Test accuracy

spambase

3

4

5

6

7

8

Depth

0.55

0.60

0.65

0.70

0.75

0.80

Test accuracy

avila

3

4

5

6

7

8

Depth

0.775

0.800

0.825

0.850

0.875

0.900

Test accuracy

dry-bean

3

4

5

6

7

8

Depth

0.78

0.80

0.82

0.84

0.86

Test accuracy

telescope

3

4

5

6

7

8

Depth

0.66

0.68

0.70

0.72

0.74

Test accuracy

connect-4

3

4

5

6

7

8

Depth

0.2

0.3

0.4

0.5

0.6

Test accuracy

letter-recognition

3

4

5

6

7

8

Depth

0.86

0.87

0.88

0.89

0.90

0.91

Test accuracy

miniboone

3

4

5

6

7

8

Depth

0.6

0.7

0.8

0.9

Test accuracy

sensorless-drive-diagnosis

k=2

k=4

k=12

k=16

k=d

plot is a different dataset, where on the x-axis, we plot the depth of the learned decision tree, and on the y-axis, we plot the test accuracy. Note that k = d corresponds to the DL8.5 optimal decision tree. We can clearly observe that the test accuracy increases as k increases-in some cases, the gain is > 5% (absolute). Furthermore, for (smaller) datasets like nursery, for which we were able to run k = d , the accuracy of Top8 / 16 is already very close to that of the optimal tree.

Lastly, since Topk invests more computation towards fitting a better tree on the training set, its training time is naturally longer than Top1 . However, Figure 6 in Appendix E, which plots the training time, shows that the slowdown is mild.

Topk scales much better than optimal decision tree algorithms. Optimal decision tree algorithms suffer from poor runtime scaling. We empirically demonstrate that, in comparison, Topk has a significantly better scaling in training time. Our experiments are identical to those in Figures 14 and 15 in the GOSDT paper [LZH + 20], where two notions of scalability are considered. In the first experiment, we fix the number of samples and gradually increase the number of features to train the decision tree. In the second experiment, we include all the features, but gradually increase the number of training samples. The dataset we use is the FICO [FGI + 18] dataset, which has a total of 1000 samples with 1407 binary features. We plot the training time (in seconds) versus number of features/samples for optimal decision tree algorithms (MurTree, GOSDT) and Topk in Figure 3. We do this for depth = 4 , 5 , 6 (for GOSDT, the regularization coefficient λ is set to 2 -depth ). We observe that the training time for both MurTree and GOSDT increases dramatically compared to Topk , in both experiments. In particular, for depth = 5 , both MurTree and GOSDT were unable to build a tree on 300 features within the time limit of 10 minutes, while Top16 completed execution even with all 1407 features. Similarly, in the latter experiment, GOSDT/MurTree were unable to build a depth-5 tree on 150 samples within the time limit, while Top16 comfortably finished execution even on 1000 samples. These experiments demonstrates the scalability issues with optimal tree algorithms. Coupled with the accuracy gains seen in the previous experiment, Topk can thus be seen as achieving a more favorable tradeoff between training time and accuracy.

We note, however, that various optimization have been proposed to allow optimal decision tree algorithms to scale to larger datasets. For example, a more recent version of GOSDT has integrated a guessing strategy using reference ensembles which guides the binning of continuous features, tree

0

200

400

600

800

1000

1200

1400

Number of Features

0

100

200

300

400

500

600

Time

Top-1

Top-2

Top-4

Top-8

Top-16

GOSDT

MurTree

(a) Depth

= 4

0

200

400

600

800

1000

1200

1400

Number of Features

0

100

200

300

400

500

600

Time

Top-1

Top-2

Top-4

Top-8

Top-16

GOSDT

MurTree

(b) Depth

= 5

0

200

400

600

800

1000

1200

1400

Number of Features

0

100

200

300

400

500

600

Time

Top-1

Top-2

Top-4

Top-8

Top-16

GOSDT

MurTree

(c) Depth

= 6

0

200

400

600

800

1000

Number of Samples

0

100

200

300

400

500

600

Time

Top-1

Top-2

Top-4

Top-8

Top-16

GOSDT

MurTree

(d) Depth

= 4

0

200

400

600

800

1000

Number of Samples

0

100

200

300

400

500

600

Time

Top-1

Top-2

Top-4

Top-8

Top-16

GOSDT

MurTree

(e) Depth

= 5

0

200

400

600

800

1000

Number of Samples

0

100

200

300

400

500

600

Time

Top-1

Top-2

Top-4

Top-8

Top-16

GOSDT

MurTree

(f) Depth

= 6

0

5

10

15

20

k

0.78

0.80

0.82

Test/Train Accuracy

car

2

4

6

8

10

12

14

k

0.5

0.6

0.7

0.8

Test/Train Accuracy

hayes-roth

0

5

10

15

20

25

k

0.70

0.72

0.74

0.76

0.78

Test/Train Accuracy

tic-tac-toe

Test

Train

size, and search [MZA + ]. Many of these optimizations are generally applicable across optimal tree algorithms and could be combined with Topk for further improvement in performance.

Increasing k beyond a point does not improve test accuracy. In our experiments above, we ran Topk only till k = 16 : in Figure 4, we show that increasing k to very large values, which increases runtime, often does not improve test accuracy, and in some cases, may even hurt due to overfitting. For 3 datasets - car, hayes-roth and tic-tac-toe - we plot train and test error as a function of k . Naturally, the train accuracy monotonically increases with k in each plot. However, for both car and hayes-roth, we can observe that the test accuracy first increases and then plateaus. Interestingly, for tic-tac-toe, the test accuracy first increases and then decreases as we increase k . These experiments demonstrate that selecting too large of a k , as optimal decision tree algorithms do, is a waste of computational resources and can even hurt test accuracy via overfitting.

6 Conclusion

We have shown how popular and empirically successful greedy decision tree learning algorithms can be improved with the power of choices : our generalization, Topk , considers the k best features as candidate splits instead of just the single best one. As our theoretical and empirical results demonstrate, this simple generalization is powerful and enables significant accuracy gains while preserving the efficiency and scalability of standard greedy algorithms. Indeed, we find it surprising that such a simple generalization has not been considered before.

There is much more to be explored and understood, both theoretically and empirically; we list here a few concrete directions that we find particularly exciting and promising. First, we suspect that power

of choices affords more advantages over greedy algorithms than just accuracy gains. For example, an avenue for future work is to show that the trees grown by Topk are more robust to noise . Second, are there principled approaches to the automatic selection of the greediness parameter k ? Can the optimal choice be inferred from a few examples or learned over time? This opens up the possibility of new connections to machine-learned advice and algorithms with predictions [MV20], an area that has seen a surge of interest in recent years. Finally, as mentioned in the introduction, standard greedy decision tree algorithms are at the very heart of modern tree-based ensemble methods such as XGBoost and random forests. A natural next step is to combine these algorithms with Topk and further extend the power of choices to these settings.

References

[AAV19] Sina Aghaei, Mohammad Javad Azizi, and Phebe Vayanos. Learning optimal and fair decision trees for non-discriminative decision-making. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 1418-1426, 2019.

[ABF + 09] Misha Alekhnovich, Mark Braverman, Vitaly Feldman, Adam Klivans, and Toniann Pitassi. The complexity of properly learning simple concept classes. Journal of Computer & System Sciences , 74(1):16-34, 2009.

[AH08] Micah Adler and Brent Heeringa. Approximating optimal binary decision trees. In Approximation, Randomization and Combinatorial Optimization. Algorithms and Techniques , pages 1-9. Springer, 2008.

[ANS20] Gaël Aglin, Siegfried Nijssen, and Pierre Schaus. Learning optimal decision trees using caching branch-and-bound search. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 3146-3153, 2020.

[ANS21] Gaël Aglin, Siegfried Nijssen, and Pierre Schaus. Pydl8. 5: a library for learning optimal decision trees. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence , pages 5222-5224, 2021.

[Ave20] Florent Avellaneda. Efficient inference of optimal decision trees. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 3195-3202, 2020.

[BD17] Dimitris Bertsimas and Jack Dunn. Optimal classification trees. Machine Learning , 106(7):1039-1082, 2017.

[BDM19] Alon Brutzkus, Amit Daniely, and Eran Malach. On the Optimality of Trees Generated by ID3. ArXiv , abs/1907.05444, 2019.

[BDM20] Alon Brutzkus, Amit Daniely, and Eran Malach. ID3 learns juntas for smoothed product distributions. In Proceedings of the 33rd Annual Conference on Learning Theory (COLT) , pages 902-915, 2020.

[BFSO84] Leo Breiman, Jerome Friedman, Charles Stone, and Richard Olshen. Classification and regression trees . Wadsworth International Group, 1984.

[BLQT21a] Guy Blanc, Jane Lange, Mingda Qiao, and Li-Yang Tan. Decision tree heuristics can fail, even in the smoothed setting. In Mary Wootters and Laura Sanità, editors, Proceedings of the 25th International Conference on Randomization and Computation (RANDOM) , volume 207, pages 45:1-45:16, 2021.

[BLQT21b] Guy Blanc, Jane Lange, Mingda Qiao, and Li-Yang Tan. Properly learning decision trees in almost polynomial time. In Proceedings of the 62nd IEEE Annual Symposium on Foundations of Computer Science (FOCS) , 2021.

[BLT20a] Guy Blanc, Jane Lange, and Li-Yang Tan. Provable guarantees for decision tree induction: the agnostic setting. In Proceedings of the 37th International Conference on Machine Learning (ICML) , 2020.

[BLT20b] Guy Blanc, Jane Lange, and Li-Yang Tan. Top-down induction of decision trees: rigorous guarantees and inherent limitations. In Proceedings of the 11th Innovations in Theoretical Computer Science Conference (ITCS) , volume 151, pages 1-44, 2020.

[Bre01a] Leo Breiman. Random forests. Machine learning , 45(1):5-32, 2001.

[Bre01b] Leo Breiman. Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author). Statistical Science , 16(3):199 - 231, 2001.

[CG16] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) , pages 785-794, 2016.

[DG17] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.

[DKM96] Tom Dietterich, Michael Kearns, and Yishay Mansour. Applying the weak learning framework to understand and improve C4.5. In Proceedings of the 13th International Conference on Machine Learning (ICML) , pages 96-104, 1996.

[DLH + 22] Emir Demirovi´ c, Anna Lukina, Emmanuel Hebrard, Jeffrey Chan, James Bailey, Christopher Leckie, Kotagiri Ramamohanarao, and Peter J Stuckey. Murtree: Optimal decision trees via dynamic programming and search. Journal of Machine Learning Research , 23(26):1-47, 2022.

[FGI + 18] FICO, Google, Imperial College London, MIT, University of Oxford, UC Irvine, and UC Berkeley. Explainable Machine Learning Challenge. https://community.fico. com/s/explainable-machine-learning-challenge , 2018.

[HR76] Laurent Hyafil and Ronald L. Rivest. Constructing optimal binary decision trees is np-complete. Information Processing Letters , 5(1):15-17, 1976.

[HRS19] Xiyang Hu, Cynthia Rudin, and Margo Seltzer. Optimal sparse decision trees. Advances in Neural Information Processing Systems , 32, 2019.

[IYA12] Ozan Irsoy, Olcay Taner Yıldız, and Ethem Alpaydın. Soft decision trees. In Proceedings of the 21st international conference on pattern recognition (ICPR2012) , pages 18191822. IEEE, 2012.

[JM20] Mikoláš Janota and António Morgado. SAT-based encodings for optimal decision trees with explicit paths. In International Conference on Theory and Applications of Satisfiability Testing , pages 501-518. Springer, 2020.

[Kea96] Michael Kearns. Boosting theory towards practice: recent developments in decision tree induction and the weak learning framework (invited talk). In Proceedings of the 13th National Conference on Artificial intelligence (AAAI) , pages 1337-1339, 1996.

[Kea98] Michael Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM (JACM) , 45(6):983-1006, 1998.

[KKL88] Jeff Kahn, Gil Kalai, and Nathan Linial. The influence of variables on boolean functions. In Proceedings of the 29th Annual Symposium on Foundations of Computer Science (FOCS) , pages 68-80, 1988.

[KM96] Michael Kearns and Yishay Mansour. On the boosting ability of top-down decision tree learning algorithms. In Proceedings of the 28th Annual Symposium on the Theory of Computing (STOC) , pages 459-468, 1996.

[KM99] Michael Kearns and Yishay Mansour. On the boosting ability of top-down decision tree learning algorithms. Journal of Computer and System Sciences , 58(1):109-128, 1999.

[LZH + 20] Jimmy Lin, Chudi Zhong, Diane Hu, Cynthia Rudin, and Margo Seltzer. Generalized and scalable optimal sparse decision trees. In International Conference on Machine Learning , pages 6150-6160. PMLR, 2020.

[MV20] Michael Mitzenmacher and Sergei Vassilvitskii. Algorithms with predictions. arXiv preprint arXiv:2006.09123 , 2020.

[MZA + ] Hayden McTavish, Chudi Zhong, Reto Achermann, Ilias Karimalis, Jacques Chen, Cynthia Rudin, and Margo Seltzer. Fast sparse decision tree optimization via reference ensembles. Proceedings of the AAAI Conference on Artificial Intelligence , 36(9).

[NF07] Siegfried Nijssen and Elisa Fromont. Mining optimal decision trees from itemset lattices. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 530-539, 2007.

[NF10] Siegfried Nijssen and Elisa Fromont. Optimal constraint-based decision tree induction from itemset lattices. Data Mining and Knowledge Discovery , 21(1):9-51, 2010.

[NIPMS18] Nina Narodytska, Alexey Ignatiev, Filipe Pereira, and Joao Marques-Silva. Learning optimal decision trees with sat. In Ijcai , pages 1362-1368, 2018.

[O'D14] Ryan O'Donnell. Analysis of Boolean Functions . Cambridge University Press, 2014.

[PVG + 11] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

[Qui86] Ross Quinlan. Induction of decision trees. Machine learning , 1(1):81-106, 1986.

[Qui93] Ross Quinlan. C4.5: Programs for Machine Learning . Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1993.

[RCC + 22] Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistics Surveys , 16:1-85, 2022.

[Sie08] Detlef Sieling. Minimization of decision trees is hard to approximate. Journal of Computer and System Sciences , 74(3):394-403, 2008. Computational Complexity 2003.

[TAA + 19] Ryutaro Tanno, Kai Arulkumaran, Daniel Alexander, Antonio Criminisi, and Aditya Nori. Adaptive neural trees. In International Conference on Machine Learning , pages 6166-6175. PMLR, 2019.

[VNP + 20] Hélene Verhaeghe, Siegfried Nijssen, Gilles Pesant, Claude-Guy Quimper, and Pierre Schaus. Learning optimal decision trees using constraint programming. Constraints , 25(3):226-250, 2020.

[VZ17] Sicco Verwer and Yingqian Zhang. Learning decision trees with flexible constraints and objectives using integer optimization. In International Conference on AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems , pages 94-103. Springer, 2017.

[VZ19] Sicco Verwer and Yingqian Zhang. Learning optimal classification trees using a binary linear program formulation. In Proceedings of the AAAI conference on artificial intelligence , volume 33, pages 1625-1632, 2019.

[ZMP + 20] Haoran Zhu, Pavankumar Murali, Dzung Phan, Lam Nguyen, and Jayant Kalagnanam. Ascalable MIP-based method for learning optimal multivariate decision trees. Advances in Neural Information Processing Systems , 33:1771-1781, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is about a new method for machine learning using decision trees. Namely, the authors propose an algorithm that can span the spectrum of the methods that are currently used: \n(i) traditional decision tree learning algorithms that choose features and splitting points based on a criterion that is optimized greedily (e.g., gini impurity), and \n(ii) optimal decision tree learning, which performs an exhaustive search for the best possible short decision tree (we should also have fairly few candidate features for splitting). \n\nIn particular, the authors allow a parameter k that controls the amount of splits that will be considered at every node. In other words, the authors consider the top k greedy choices at every node. This allows one to consider exponentially many trees, and thus choose the best tree among those considered. Note that when k=1, the algorithm reduces to the standard greedy algorithms such as C4.5 and CART, whereas when k=d (where d is the dimension of the instances), we obtain an algorithm that constructs an optimal decision tree.\n\nThe authors have some nice results to complement the proposed approach. For example, in Theorem 1 of the paper, it is shown that there are data distributions where a (k+1) branching factor would allow accuracy of 1-ε, while a branching factor of k can achieve accuracy at most 1/2 + ε. The authors also show (in Theorem 2) a generalization of the previously mentioned theorem. Eventually the authors show experimentally that the proposed algorithm behaves well in an experimental setting with the solutions provided between the two extremes of optimal decision tree learning and pure greedy methods that have been used for decades. In other words, through the experiments, the authors highlight the tradeoff one would be willing to explore by spending some extra time on constructing decision trees that can potentially allow the user to achieve better accuracy, while at the same time one would be able to create much deeper trees, much faster, compared to optimal decision trees.\n\nAfter the rebuttal: Upgraded the score from borderline accept to weak accept since I was satisfied with the responses that I received and largely with the responses that the authors gave to the comments and issues that were raised by the other reviewers.\n\nSTRENGTHS:\nI think this is a well-written paper that is also backed up with theoretical analysis and is also validated (to a large extent) in the experiments. It is an interesting algorithm that appears to be improving the accuracy of the learned decision trees in many practical datasets, while at the same time the increase in performance is probably tolerable. The paper also has a fairly nice summary of related work.\n\nWEAKNESSES:\nThe paper is not based on any striking new idea. While the novelty is limited, it is nevertheless interesting.\n\nQUESTIONS:\nQ1. How many runs did you use for the results in Figure 2? Are the conclusions based on statistically significant observations?\n\nQ2. On a related note, in Figure 2, in the case of spambase (and some other cases), MurTree that calculates the optimal decision tree, provides a solution with worse accuracy compared to what the authors achieve with their method. Can the authors shed some light in this? Also, another observation has to do with the fact that on the same graph (spambase), at depth 8, top-2 outperforms top-3. Is there an explanation for this?\n\nLIMITATIONS:\nI think the paper is ok.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the paper authors propose a new algorithm that allows balancing between two approaches to decision tree learning: fully greedy optimization, and exactly optimal decision tree construction.  Authors discuss the long standing line of research that focuses on either ends of this balance, and propose a novel algorithm Top-K that solves the issues of both sides:\n- Greedy algorithms are fast, but there are counterexamples that lead to arbitrarily bad accuracy\n- Optimal algorithms are inherently slow because of the NP-hardness of the problem.\n\nThe Top-K algorithm is backed up by several theorems that show that increasing K allows solving counterexamples that are unsolvable with high enough accuracy using smaller K. While some of the counterexamples involve artificial constructions, authors propose evaluation on real-world datasets that show practical applicability of the proposed method.\n\nSTRENGTHS:\nMost importantly, this is a novel contribution to the field of Decision Trees learning. The proposed algorithm is easy to implement, theoretically sound, and solves major issues of the existing approaches.\n\nOn a side note, the paper is well-written, provides convincing motivational examples, a thorough literature review, strong theoretical foundation and a clear description of the algorithms.\n\nWEAKNESSES:\nNo major weaknesses. There is one concern outlined below, and other questions (which may be addressed here or solved in the future work) are placed in the \"Questions\" section.\n- The paper focuses on a binary classification setting. And features are also binary in the theorems and definitions. This was confirmed in the experimental section, where categorical features have been converted to binary and numerical features had to be thresholded and converted to binary. This raises a question about applicability to non-binary data; and more importantly, to a regression setting (which is equally practically popular, as classification). Clearly stating this limitation or addressing possible ways to extend both theoretical and practical results will help in convincing the reader about usefulness of the proposed algorithm in real world settings and clarify how high is the impact of the paper on the Decision Trees learning field.\n\nQUESTIONS:\n- Since the paper provides one way to balance between fully-greedy and fully-optimal approaches, reader may be curious whether there other possible ways to achieve a balance between those two ends (This may be addressed in the future work, or perhaps authors already have an answer about why Top-K was implemented in a given way, and what could be other possible implementations?)\n- There is a line of research on optimizing the Optimal Decision tree algorithm: for example, using Dynamic Programming. While it does not solve the problem of slow running time of the Optimal Algorithm due to NP-hardness of the problem, it allows obtaining different running time bounds, that may be better than fully brute-force search for specific problem sizes and parameters. Would be nice to know whether such optimizations are applicable to the proposed Top-K algorithm (this may be a topic of future work, but perhaps authors already have answers to this question?)\n- We see from the experimental results that Top-K outperforms Top-1 on the test set; but in the current implementation Top-K uses binarization and is compared to Top-1 that also uses binarization of the features. Does this binarization significantly reduce the accuracy compared to using a non-binarized dataset? Specifically, what would happen if Top-1 would run on a non-binarized version (because standard Top-1 greedy solvers, such as the ones from scikit-learn do not require binarization of the dataset)?\n\nLIMITATIONS:\nMain concern about limitations is the applicability to problem settings other than binary classification with binary features (outlined in the above sections of the review)", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel generalization of classic greedy decision tree algorithms called Top-k, which considers recursively the top k attributes for possible splits rather than the single best attribute. It enhances traditional decision tree learning algorithms such as ID3, C4.5, and CART, and is expected to be a significant improvement on these established methodologies. The authors provide both theoretical and empirical proof supporting the effectiveness of this modification.\n\nTheoretically, the authors establish the greediness hierarchy theorem that demonstrates there exist data distributions for which one can expect significant improvements in accuracy as k increases. Empirically, they test Top-k against standard greedy algorithms and optimal decision tree algorithms using real-world datasets. The experimental results reveal that for the same tree depths, Top-k significantly outperforms the classic greedy methods while also being much more scalable than optimal decision tree algorithms.\n\nSTRENGTHS:\n- The paper makes a novel contribution to the field of decision tree learning by introducing the Top-k generalization of classic greedy algorithms. The concept is simple and intuitive, yet it has a profound effect on the performance of the algorithm.\n\n- The authors have provided a detailed theoretical basis for their work. The introduction of the greediness hierarchy theorem gives clear theoretical evidence of the potential benefits of considering more attributes for splits. This robust theoretical grounding is a key strength of this paper.\n\n- A comprehensive experimental evaluation is carried out to back the theoretical findings. The authors evaluate the Top-k approach using UCI datasets, comparing the results with both classic greedy algorithms and optimal decision tree algorithms. The performance of Top-k is compelling, exhibiting significant improvements over classic methods and demonstrating comparable scalability.\n\nWEAKNESSES:\n- While Top-k demonstrates stronger performance than classic greedy algorithms for the same tree depths, an additional, more fair setup would be comparing the test accuracy achieved in comparable training time.\n- The runtime scales exponentially with the depth h.\n\nQUESTIONS:\n- In gradient-boosting tree ensembles, the depth of each tree is usually small. Can Top-k improve the predictive performance of GBDTs?\n\nLIMITATIONS:\nThe limitations have been discussed in the Conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies an extension of greedy decision tree algorithms. The main idea is, instead of simply greedily selecting the best feature, to consider several features. This can also be seen as a restriction of optimal decision tree algorithms, so in some sense the studied approach lies somewhere in between greedy and exhaustive algorithm. The point is that greedy approach are fast, optimal algorithms are slow, so something in between may be desirable.\n\nThe authors provide theoretical results and some experiments to illustrate a few points. \n\nEven though the idea of selecting a subset of the features is emphasized in the paper, I find this questionable, e.g., I find that a minor and rather natural point. In my view the main contribution are the theoretical results - these are novel to the best of my knowledge and provide a nice way to formalize relatively intuitive ideas. I think the paper would benefit from focusing on this theoretical aspect since this is a unique point.\n\nWhile I am enthusiastic about the theoretical results, I feel that there are many (small) issues with the paper that need to be addressed before publication. In particular, the relationship with related work needs to be expanded and discussed in more depth, the experiments go in the right direction but the claims are not convincingly backed up by data, and there are several (minor) technical issues.\n\nSTRENGTHS:\nThe theoretical results sound relevant. Recent recent tree works concerning \"practical\" algorithms do not provide virtually any theoretical results, and in this work this is studied formally. The authors also point of the artificialness of the theoretical results, and then propose a more \"realistic\" mathematical setting. All of these points are welcome contributions, and highlight (potential) for good technical depth.\n\nWEAKNESSES:\nConceptual \n\nThe related work is treated very briefly: the related work section is half a page. The text gives a bird's eye overview of the field, which is good, but even this text does not relate to the work done in the paper. I would like to see some more discussion on how related algorithms work and have the authors draw the connection to this approach. This places the work in the right context which is important when publishing a paper. This absence of direct relation to related work is also reflected in the paper later on.\n\nSince the paper covers theory and the authors consider a parameter k, it would be natural to relate this work to the paramaterized complexity of decision trees paper [Parameterized Complexity of Small Decision Tree Learning, AAAI 2021]. This would provide more rigorous analysis.\n\nThe text presents the results as surprising, as going from k to k+1 may make a large difference even though it is only an increase of one. This is a nice observation, and should be mentioned, but the text needs to be toned down. Since the number of decision trees grows exponentially with the number of features, it is clear that going from k to k+1 leads to an exponential increase in number of trees. So it makes sense that this has an impact on accuracy. Note that this does not diminish the value of the contribution, it merely puts it in the right perspective.\n\nThe authors state that considering \"top-k\" compared to \"top-1\" introduces a \"mild\" increase in complexity. However the algorithm increases _exponentially_ with the k parameter, which is also shown in this paper (Claim 3.1). It could be that in practice, if the k is small, the impact is not large in terms of absolute nubmer of seconds, but in theory this is still a large complexity increase. Setting k = number of features leads to the exhaustive algorithm, which is exponential.\n\nI find the idea of top-k interesting and relevant but the idea is rather small. The value of the paper in my view is in the theory part and (possibly improved) experiments. This could be reflected more in the text. Note that the top-k approach has already been implemented in MurTree but I am not sure if there have been thorough evaluations.\n\nDefinition 3 and Lemma 3.2: these follow directly from the algorithm (Figure 1), so it is odd to pack the results as a lemma that has proof in Appendix, makes it sounds more complicated that it is. A simpler proof, if needed to be included, is to show by induction on the recursively formulation in Figure 1.\n\nParallelization is mentioned as an \"easy\" task. First, this is the case with virtually every decision tree algorithm. Second, the situation is only easy on the surface level: it is not trivial to parallelize techniques used by optimal decision tree algorithms (bounding, caching) that give significant runtime improvements, so a trivial parallelization can leads to degradation of performance.\n\nExperiments\n\nThe experiments mention that 100 datasets are available, but the authors select a random subset of 20. But why only 20, why not all? Why not take available datasets that have been directly used in previous works? This seems odd given the ability of the datasets, and the runtimes are still managable.\n\nThe conclusions are made on cherry picked datasets, i.e., only a few datasets even though many datasets are available. The scalability experiments shows that top-k scales much better: this is obvious based on the complexity, but it is still interesting to see. However the dataset seems to be extreme: while other considered datasets have 50-300 features, this dataset has 1400 features. While it is a good idea to use an extreme example to emphasize the scalability problem, this needs to be done all datasets and not just on one.\n\nThe comparison with MurTree and GOSDT is also not entirely proper, since the algorithms solve slightly different problem. MurTree is the most similar, GOSDT is slightly different because of the alpha parameter and no explicit depth. This would need to be discussed, and/or setting the same alpha parameter for MurTree so that the comparisons are more aligned (although in this case this paper does not support the alpha parameter, the authors could also include this parameter).\n\nThe conclusion that after some k value there are no gains in accuracy: this is intuitive and nevertheless interesting, but the conclusion is made on three datasets. It is needed to do more thorough experiments on all datasets to make this claim.\n\nThe authors discuss that top-k can sometimes even lead to better test accuracy. This is problematic for two reasons. First, the method of comparison (dl8.5 in this case) does not optimise in any way for test accuracy, so the comparison is odd. Second, the authors compare against full decision trees, which are prone to overfitting. Since the algorithm is incentivised to use all nodes, it will use all nodes and capture noise in the training data, so it is expected that it will not generalize well. The fix here is to do tuning with not only the depth, but the number of nodes the algorithm is allowed to use, or alternatively use the alpha parameter.\n\nMinor\n\nFigure 5 seems important but it is placed in the Appendix.\n\nPlease provide the intuition behind the proofs. This would add more depth to the paper.\n\nSince monotone is an important concept, please introduce monotonicity for completeness.\n\nQUESTIONS:\nI believe the paper does have potential but there are many issues as outlined in the Weakness section. Since there are many issues, I do not have specific questions at this point. I hope my review will help the authors in revising the work, I would be happy to see it refined and published later on. In case I misunderstood something the authors are welcome to point out flaws in my reasoning.\n\nLIMITATIONS:\nThis is somewhat discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}