Stochastic Induction of Decision Trees with Application to Learning Haar Tree

Abstract: Decision trees are a convenient and established approach for any supervised learning task. Decision trees are used in a broad range of applications from medical imaging to computer vision. Decision trees are trained by greedily splitting the leaf nodes into a split and two leaf nodes until a certain stopping criterion is reached. The procedure of splitting a node consists of finding the best feature and threshold that minimizes a criterion. The criterion minimization problem is solved through an exhaustive search algorithm. However, this exhaustive search algorithm is very expensive, especially, if the number of samples and features are high. In this paper, we propose a novel stochastic approach for the criterion minimization. Asymptotically, the proposed algorithm is faster than conventional exhaustive search by several orders of magnitude. It is further shown that the proposed approach minimizes an upper bound for the criterion. Experimentally, the algorithm is compared with several other related state-of-the-art decision tree learning methods, including the baseline non-stochastic approach. The proposed algorithm outperforms every other decision tree learning (including online and fast) approaches and performs as well as the baseline algorithm in terms of accuracy and computational cost, despite being non-deterministic. For empirical evaluation, we apply the proposed algorithm to learn a Haar tree over MNIST dataset that consists of over $200,000$ features and $60,000$ samples. This tree achieved a test accuracy of $94\%$ over MNIST which is $4\%$ higher than any other known axis-aligned tree. This result is comparable to the performance of oblique trees, while providing a significant speed-up at both inference and training times.

Reviews and author responses:

 Sorry, I had missed Figures 8 and 9. However, they don't seem to be discussed anywhere, and it seems to remain unclear how to choose values for the hyperparameters.

 We highly appreciate your comments and that you have read the paper throughly. We will consider comparing the algorithm with the mentioned models (row and column sub-sampling used in XGBoost and lightgbm). Since most of your comments are aligned with the second reviewer, we encourage you to read our responses to second reviewer. We respond to the rest of your comments here:

Q1: I also quickly looked through the supplement and did not find anything but it is possible I might have missed it. Are we looking at actual wall clock time (which is very implementation dependent) or just at the theoretical runtime complexity (which are in  form relying on unknown constants).

Ans: we are reporting the number of operations applied by each algorithm. It is not clock time.

Q2: Given the random nature of the proposed scheme and other sampling based baselines, it is not clear if the results were obtained with multiple trials, and it is not clear how robust the demonstrated performances are with respect to randomness. Confidence intervals around the performance/computation tradeoffs would demonstrate the robustness of the results.

Ans: We ran the SDT several times. However, all the results are consistently similar (in the same range with small variation). We will report confidence interval in our next version of the paper but they will be very tight based on our previous observation.

Q3: It is not clear why a decision tree on Haar features is named "Haar Trees". Since all decision trees are built on the same Haar features, won't all the decision trees also be "Haar Trees"? Or is it the case that the proposed SDT is built on Haar features while all other decision trees are built on the original features (pixels) of the MNIST/FashionMNIST datasets.

Ans: Only SDT uses the Haar-Features but the other models (OC1,K-means and RBF-SVM in figure 3) have the ability to use combination of features rather than just using only one feature. The Haar-Features provide the ability to extract simple features from images thus improving performance of models such as trees. CART was the only model in figure 3 that uses raw pixels. Please note that the number of features for Haar-Tree is very high (300000 for MNIST) and it just simply does not fit in the memory (for the whole trainset a memory of 1072GB is needed assuming 64byte for each value). Therefore, applying other algorithms over this dataset is just extremely time consuming it is while SDT can be trained over this dataset in matter of minutes without an extreme necessity of using a huge RAM.

Q4: It is not clear how thoroughly the hyperparameters of the baselines were optimized compared to the proposed SDTs. If it is discussed in the supplement, can you please point me to the right section.

Ans: Please look at Q4 and Q8 of the third reviewer.

 Thank you for your comments. We will apply the suggestions and advices regarding the abstract, introduction, mentioning oblique trees and the fact that this paper only concerns axis-aligned trees over numerical values, and etc.

Q1: The stated time complexity is not correct when features are pre-sorted, which should be discussed.

Ans: we will discuss that. Pre-sorting improves the speed of the tree only by a factor of the depth of the tree. However, it is still memory inefficient specially if D is high.

Q2: The description of embedded approaches does not seem quite right.

Ans: we highly appreciate it if the reviewer ellaborates more on that.

Q3: What metric is used for finding splits with the proposed method in Section 5?

Ans: Gini-Index was used.

Q4: The proposed method has three hyperparameters (\alpha, C, n_0). How sensitive is performance wrt these hyperparameters?

Ans: We explored n_0 and C in the appendix. Please see figures 8 and 9. These parameters basically control the computational complexity of constructing the decision tree. They are not sensitive in the sense to affect the final accuracy of the model.

Q5: The RBF-SVM seems highly non-standard. Can this be called an SVM? It seems strange that k-means performs better than this SVM on the MNIST data.

Ans: It is not highly non-standard. The fact is that K-means+RBF was used like a kind of filter approach for feature extraction and then SVM was used on top of this model. We understand that conventionally the RBF SVM is considered when an RBF is trained over every single sample (as a kernel) but that is very expensive in terms of inference and number of parameters. That is only true for when K=1000 but otherwise RBF-SVM performs better than K-means. The parameters of RBF was selected using a validationset. However, our observation was that beyond K=1000, K-means and RBF-SVM perform similarly. Our understanding is that both of these methods act somewhat like a Nearest Neighbor model in the input space (specially when K goes toward infinity). As a result, by increasing K, both models perform almost similar.

Q6: Kudos for including OC1 in the experiments!

Ans: Thanks, we think that OC1 is an important Oblique tree that has attracted less attention than it should have.

Q7: What kind of accuracy do you get using a random forest with axis-parallel splits?

Ans: it depends on the dataset. However, on MNIST with 1000 trees of depth 12, using 0.8 of trainset and using randomly 0.7 of the features, forest will achieve 97% test accuracy.

If you mean with SDT, we have not tested it but we expect it to perform as well as when using CART as base trees.

Q8: Runtime of greedy decision tree learning is not normally considered a problem in practice, particularly once the feature set has been reduced using pre-filtering. However, runtime is the focus of the paper. The impact of what is presented is thus likely to be very small, even ignoring the problem of hyperparameter tuning.

Ans: In this paper, we considered pre-filtering but we noticed that method will underperform as reported in the experiments. Runtime becomes specifically a problem when D is high. Which is true for the case of Haar-Trees. Finally, the hyperparameter tuning in this work is not a problem at all as figures 8 and 9 show the effect of C and N_0. We also have learned as long as we select \alpha as a value between 7-10, the algorithm performs as good as CART algorithm which was the goal of the paper (to decrease complexity while maintaining the accuracy). Please also look at Q1 of the second reviewer regarding when the runtime becomes problematic.

 Q10: The use of several fixed depths for the trees makes it more difficult to analyse the results and compare the methods. Why not computing the accuracy of trees pruned on a separate validation set, to reduce each method to only one score? This would be closer to how these methods are used in practive

Ans: The reason we selected fix depths and did not apply pruning was because we wanted to keep everything constant and purely observe the performance of all algorithms under the same setup and for diverse setups. It is true that pruning is applied in practice but pruning will not change much when one algorithm performs purely better than the others.

Q11: Results with Haar trees are interesting as representative of problems with many features but I don't see the practical interest of using single decision trees in computer vision problems. One knows that using ensemble of trees or convolutional neural networks will work much better on such problems (getting below 90% on MNIST is not impressive).

Ans: We understand the point of your comment. However, one direct example of such problems (although old) is the Viola Jones Face-Detector.

Q12: From the code provided, it seems that the algorithm is implemented in pure matlab. I have not analysed the code thoroughly but matlab does not seem like the best language to implement decision trees (that are not naturally implemented through matrix/vectorial operations). I'm wondering how much this affects the conclusion of the study. How does the computing times for CART in matlab compares with CART as implemented in Scikit-learn (that implements trees in C (cython)) or other standard efficient implementations mentioned earlier?

Ans: We will consider that for a future project to perfectly implement the algorithm in C.

 Q4:I think the computational complexity analysis is not strictly correct. If N0 is actually fixed and not C (as done in the practical implementation of the algorithm) and if alpha is set as a function of the number of features (to reach 0,5% of the features), then the coefficient N0 \log(alpha) can not be neglected with respect to N0 log(N0) as the second term is a constant and alpha is a function of D. And also, the complexity of step 5 should be taken into account. Strictly, if you stop the iterations when 0.5% of the features are remaining, then the complexity of step 5 is O(0.005DNj* log(Nj)) which is the same as O(DNjlog(Nj)) in big O notation. And thus the computational complexity is actually unchanged with respect to standard trees. Part of the problem comes from the fact that the authors use the big-O notation, which is not appropriate here as all algorithms are equal in terms of big-O notations.

Ans: Your comment is to the point. However, the reason that big O notation appears in all notations is because of the quick sort algorithm that happens over the N_j samples. It is while multiplication of D by the term in O(DNlog(N)) is because of the fact that samples must be sorted along all D features. However, D can not be neglected since D itself is a term which can be high and thus, determinitive in actual number of operations. In practice, specially when D is high, the real time needed for inducing the tree is controlled by D. As a result, the computational complexity added to the method by the last step is actually something in the same order as reported for the computational complexity reported in theorem 1 (still faster than the original algorithm in order of magnitudes).

Q5: Theorem 2 is also very imprecisely stated. The authors refer to the objective function of Equation (2) but I guess they mean Equation (1). But then, their analysis does not take into account the tree complexity term in (1). If alpha is >0, then the objective function in (1) will actually start increasing. The theorem formulation should also make the bound explicit. Strictly speaking, L_max + D-k is also an upper bound on the objective function (with alpha=0) that the algorithm monotonically decreases but this is a useless result

Ans: No, we are analyzing Eq. 2. The upper bound is with regard to upper bound of the split objective function in Eq 2.

Q6: Theorem 3 is also very imprecisely stated. What is the probability of overlap? In practice, we will often have an overlap area that covers the whole space. In the proof, why is the probability P(FI_p^k <FI_p) an upper bound to the probability of discarding a feature? This is very unclear.

Ans: Please read the answer to Q3 of reviewer 1 to clarify the meaning of overlapping area. Calculating the exact probability of discarding a feature is very hard since it depends on various characteristics of the dataset and the features. Thank you for the comment. This theorem is true with the assumption that the p^{th} feature must not be discarded (it has a feature importance higher than half of the features) but for a momment assume that due to approximateness of the algorithm discards the feature. This can only happen when the rank of feature importance of the p^{th} is lower than what it is (or the rank of the next feature to be higher than this feature). The least effort needed for the feature to be discarded is to achieve a feature importance ranking lower than its true ranking is to at least pick one extra sample from the overlapping area. As a result, the reported probability is an upper bound to probability of discarding a feature that should not be discarded.

Q7: Theorem 4 is not very useful as it is not specific to SDT. If you want to include it, then the proof of this theorem in Appendix A.1 is too short. It should be explained why Theorem 20.1 and Lemma 20.1 from (Devroye et al., 2013) apply.

Ans: We will add it to the next version of the paper.

Q8: Theorem 5 is again very imprecise: What does it mean that "the objective function of equation (2) decreases as new nodes are added"? I guess the authors again mean equation (1) with alpha=0. Again if alpha>0, the theorem is false.

Ans: This is a general fact regarding the trees.

Q9: Discussion of the results is very minimal. I would have like to see experiments to justify the different steps of the methods (in the absence of a convincing theoretical analysis). What about performing the feature selection in one step instead of alpha steps (using N0 or alpha*N0 examples to select directly 0.5% of the features)?

Ans: we tried this and with this setup method performs poorly. Experimentally speaking, the algorithm slowly converges to selecting some good features. A sudden big step (like the one mentioned in this question) results in poor results and also on the other hand, a conservative feature discarding will result in higher accuracy but might increase the computational complexity.





 Q1:Their actual computing times might be high when the dataset is large but that's not really a problem related to computational complexity. Unlike what is said in the paper, there exist also many efficient, exact or approximate, implementations of decision trees (SPRINT, SLIQ, Yggdrasil, etc.) but most of them reach scalability by distributing the construction of the tree. Personally, I don't think that there is a strong need for an approximate algorithm for growing decision trees. If you use a single decision tree, then you probably use it mainly for its interpretability and I would, in all cases, prefer to wait more to find the best possible split than to risk missing it because of stochasticity.

Ans: Thank you so much for this comment. However, to better clarify the necessity of this algorithm, I give you the example of creating a Haar-Tree on face-detection dataset of Real-time Face-Detection by Viola and Jones. The dataset consisted of around 5000 images with size of 24x24. To find the best feature and threshold, the authors needed a computer with 50GM RAM. Let alone the fact that their dataset was in order of milions. Even with pre-sorting of the samples still the problem is intractable computationally due to high usage of Memory. Our algorithm focuses on this big problem directly and proposes a practical solution. We specifcally trained Haar-Trees on MNIST to demonstrate this advantage of the algorithm. Please also take a look at Q5 of the first Reviewer.

Additionally, you simply can not wait for the other algorithms to finish calculating the best feature and threshold because those algorithms will run out of memory.

Q2: I don't understand some design choice in the method however. The size of the feature set is decreased multiplicatively while the set of samples is increased additively. Why not increasing the set of samples multiplicatively as well (doubling it at each iteration as in [*])? With a simple additive approach and in particular using C=10 as said in the paper, you end up taking decisions based on very few samples even at the end of the iterations (only 1/100 of the training examples even after 10 steps) and even if you have significantly decreased the number of features. With a multiplicative increase, you would also not need step 5 of the algorithm since the final choices will be made using all examples.

Ans: We considered this option. However, we were interested in pushing the algorithm in to its limit while achieving similar accuracy to original CART algorithm. 

Q3: There is also a lot of confusion in the paper about the way the hyper-parameters are defined and set. In Section 4, it is said that C is set to 10 and alpha is chosen between 7 and 10. I think this does not make sense in practice because when the size of the node to split goes below 2^10 then the initial sample size will be too small. For alpha, it's not possible to make 10 iterations when the number of features is below 2^10. But actually, from the beginning of Section 5, I understand that the authors have instead set N0 to a user-defined constant and have also adapted alpha to stop before there remain too few features (0.5%). Why not describe this version in Section 3 then?

Ans: You are correct. We will discuss this part in section three. However, this is not a major issue since the main problem with computational complexity happens at specially the higher levels of the tree (in practice up to depth of 5-6).


 Q1: Theorem 2 seems off to me. Consider a binary classification problem (M=2). At the root node, I randomly chose a small sample and there exists a feature k whose stump perfectly separates the two classes. However, I just got really lucky and any other point from the dataset will no longer result in perfect separation along that feature. In this case, it seems like the loss in iteration (k+1) will be larger than in iteration (k). Am I missing something here?

Ans: Thank you so much for this question. No, You are correct. The theorem guarantees that there is an upper bound that we are certain to decrease. Theoretically speaking, this means there is the possibility that value of equation 2 to increase. However, in practice and in large datasets we noticed that actually the objective function (equation 2) monotonically decreases.

Q2: How is the computational complexity calculated in Figure's 1 and 2?

Ans: We counted the exact number of operations needed for each algorithm during the induction of the trees (implemented inside the code).

Q3: Theorem 3 is a rather weak result. It effectively says that the probability of sampling being effective has to do with how well the classes are separated along a given feature. In the worst case, if a feature had duplicate minimum and maximum values, with one of the extreme points belonging to class 0 and the other belonging to class 1, the probability of overlap is 1 and the probability of discarding the feature is 1, even if it is otherwise informative.

Ans: Thank you so much for this comment. However, this comment is contradictory. Probability of overlapping means how much both classes (all samples and not the samples that are randomly selected) are overlapping. This means if probability of overlapping is 1 then both classes along this feature are completely mixed and therefore this feature is completely uninformative. However, if you mean after sampling, we picked all the samples from the overlapping section, then the feature will get discarded but the point of the theorem is that probability such worst case scenario leans toward 0 even with samples in order of 100. It is because probability of selecting all samples from the overlapping area is (P_0^p)^(number of samples selected).

Intuitively, the whole point of this theorem is that the selected samples are a good representative of the features. As a result, we don't really need to explore the whole dataset to find the best feature and we can rather follow the presented algorithm and approximately find the best feature.

Q4: Why did you not compare to any of the streaming tree growing literature, e.g. "A Streaming Parallel Decision Tree Algorithm" by Ben-Haim and Tom-Tov? That algorithm could be run in single pass along a random ordering of the datasets.

Ans: thank you so much for this suggestion. We will compare with this algorithm for the future version of our paper.

Q: Detailed comments: -broken links (??) in paragraph 7 on page 2 and throughout the manuscript. -What is SDT in paragraph 7 on page 2. -Eqn 2. is a bit unclear. Is the optimization over all D features for the N_j samples at node j? -Figures 1 and 2 have many very similar colors, making it tough to decipher the different methods.

Ans: the broken links were the following papers: 
  Yates, et al. SPAARC: a fast decision tree algorithm.
  	Loh, et al. Faster Hoeffding racing: Bernstein races via jackknife estimates.
  	ZHENG, et al. Integrating boosting and stochastic attribute selection committees for further improving the performance of decision tree learning.

SDT is the proposed algorithm (Stochastic induction of Decision Trees). We will clarify this in next version.

The original problem in Eq. 2 is over all features and N_j in the j^{th} node. The SDT approximates the solution to this problem.

Q5: Overall, I was a bit mixed on this paper. I think the empirical results are quite compelling and show the benefit of the proposed method. However, the idea is very simple, of limited novelty and I found the presentation to be lacking.

Ans: Thank you so much for agreeing with compelling results. Regarding the simplicity of the idea, although that we agree with simplicity of the idea but this is lacking from the literature of decision tree induction while this method should be the bread and butter of decision tree induction simply because if the number of features grow too high (like the example of Haar-Trees) then the existing implemented SOTA can not handle the problem. The reason is simply because the RAM usage will increase and the program should break (e.g., for case of Haar-Trees you need a RAM with 50GB memory please see the paper of Real-time Robust Face-Detection by Viola and Jones). Theoretical comparison of the proposed method and the SOTA method is similar to comparison of Batch Gradient Descent and Gradient Descent. Although the algorithm is simple but it is effective in what it does.

 Dear Authors and Reviewers,

Let me first thank you for supporting ICLR 2022: Authors for submitting their contributions, and Reviewers for going through them and sending their comments and remarks!

As the discussion phase has just begun, let me ask Authors to answer all questions appearing in reviews and to defend your paper, and reviewers to check all other reviews to see whether you coincide and to be ready to respond to authors rebuttals.

We are also looking forward for public comments. I hope for a vivid discussion for this paper.

Best regards, AC for Paper 3349

The paper introduces an approach to decision tree induction based on sub-sampling and pruning heuristics.  Typically, inducing a tree involves O(D*N_j*log(N_j)) work at each node j, where D is the number of features and N_j is the number of observations at node j.  This is because we can find the optimal split at node j for each feature in O(N_j) time after sorting each of the N_j observations. The basic idea introduced in this paper is to reduce the computational complexity incurred at each node by starting with a subsample of N_j, pruning features with sub-par performance, increasing the subsample size, pruning sub-par performing features, .... until the best feature has been chosen on the full sample of N_j observations.

The idea is very simple and the authors attempt to bolster this simple idea with some theoretical results and a wide array of experiments. The method does appear to be effective for datasets with very large D, which for example occurs when constructing Haar features on image data.   The idea in this paper is very straightforward and the literature on decision/regression trees is vast, so I would be a bit surprised if this was truly a novel finding.  However, assuming that this is the first time this idea was formalized, I'm left questioning whether the contribution is indeed significant enough to merit publication.

I do find the experimental results to be quite promising.  The method seems to find trees of comparable accuracy with far less computation, especially as the dimensionality of the dataset increases.

Theorem 2 seems off to me.  Consider a binary classification problem (M=2).  At the root node, I randomly chose a small sample and there exists a feature k whose stump perfectly separates the two classes.  However, I just got really lucky and any other point from the dataset will no longer result in perfect separation along that feature.  In this case, it seems like the loss in iteration (k+1) will be larger than in iteration (k).  Am I missing something here? 

How is the computational complexity calculated in Figure's 1 and 2?

Theorem 3 is a rather weak result.  It effectively says that the probability of sampling being effective has to do with how well the classes are separated along a given feature.  In the worst case, if a feature had duplicate minimum and maximum values, with one of the extreme points belonging to class 0 and the other belonging to class 1, the probability of overlap is 1 and the probability of discarding the feature is 1, even if it is otherwise informative.   

Why did you not compare to any of the streaming tree growing literature, e.g. "A Streaming Parallel Decision Tree Algorithm" by Ben-Haim and Tom-Tov?  That algorithm could be run in single pass along a random ordering of the datasets.

I also think the presentation could use some work (see e.g. detailed comments below).  The experiments section needs additional details, there were broken links throughout and the theoretical results needed a bit more polish.

Detailed comments:
-broken links (??) in paragraph 7 on page 2 and throughout the manuscript.
-What is SDT in paragraph 7 on page 2.
-Eqn 2. is a bit unclear. Is the optimization over all D features for the N_j samples at node j? 
-Figures 1 and 2 have many very similar colors, making it tough to decipher the different methods. Overall, I was a bit mixed on this paper.  I think the empirical results are quite compelling and show the benefit of the proposed method. However, the idea is very simple, of limited novelty and I found the presentation to be lacking. 

The paper proposes a stochastic (i.e. approximate) algorithm to fasten node splitting during tree induction. The idea of the method is to filter out the most irrelevant features using a (growing) subsample of training examples and then to optimize the split using all samples only for the remaining features. The approach is shown to improve computing times for tree construction by several orders of magnitude without impacting accuracy.
 The idea explored in the paper is potentially interesting, although I find it not very well formulated and motivated. The authors claim that decision tree construction has a high computational complexity, which is simply not true. Their computational complexity in big-O notation is close to the best you can get (close to linear w.r.t. both the number of samples and the number of examples). Their actual computing times might be high when the dataset is large but that's not really a problem related to computational complexity. Unlike what is said in the paper, there exist also many efficient, exact or approximate, implementations of decision trees (SPRINT, SLIQ, Yggdrasil, etc.) but most of them reach scalability by distributing the construction of the tree. Personally, I don't think that there is a strong need for an approximate algorithm for growing decision trees. If you use a single decision tree, then you probably use it mainly for its interpretability and I would, in all cases, prefer to wait more to find the best possible split than to risk missing it because of stochasticity.  If one is not interested in interpretability, it's better to use random forests or boosting and there are many fast stochastic tree induction algorithms available in this context.

The proposed approach is nevertheless interesting. I like in particular the idea of progressively decreasing the number of features, while increasing the number of training examples. A similar idea is explored in this paper (section 5.4, about laminating):

[*] Adaptive Sampling for large scale boosting. Dubout and Fleuret. Journal of Machine Learning Research 15 (2014) 1431-1453.

in a close but different context however (i.e., to select best weak models in a boosting approach). 

I don't understand some design choice in the method however.  The size of the feature set is decreased multiplicatively while the set of samples is increased additively. Why not increasing the set of samples multiplicatively as well (doubling it at each iteration as in [*])? With a simple additive approach and in particular using C=10 as said in the paper, you end up taking decisions based on very few samples even at the end of the iterations (only 1/100 of the training examples even after 10 steps) and even if you have significantly decreased the number of features. With a multiplicative increase, you would also not need step 5 of the algorithm since the final choices will be made using all examples.

There is also a lot of confusion in the paper about the way the hyper-parameters are defined and set. In Section 4, it is said that C is set to 10 and alpha is chosen between 7 and 10. I think this does not make sense in practice because when the size of the node to split goes below 2^10 then the initial sample size will be too small. For alpha, it's not possible to make 10 iterations when the number of features is below 2^10. But actually, from the beginning of Section 5, I understand that  the authors have instead set N0 to a user-defined constant and have also adapted alpha to stop before there remain too few features (0.5%). Why not describe this version in Section 3 then?

The theoretical analysis is not well written and was very difficult to follow (in particular Section 4.2). The main problems are as follows:
- I think the computational complexity analysis is not strictly correct. If N0 is actually fixed and not C (as done in the practical implementation of the algorithm) and if alpha is set as a function of the number of features (to reach 0,5% of the features), then the coefficient N0 \log(alpha) can not be neglected with respect to N0 log(N0) as the second term is a constant and alpha is a function of D. And also, the complexity of step 5 should be taken into account. Strictly, if you stop the iterations when 0.5% of the features are remaining, then the complexity of step 5 is O(0.005*D*Nj* log(Nj)) which is the same as O(D*Nj*log(Nj)) in big O notation. And thus the computational complexity is actually unchanged with respect to standard trees. Part of the problem comes from the fact that the authors use the big-O notation, which is not appropriate here as all algorithms are equal in terms of big-O notations.
- Theorem 2 is also very imprecisely stated. The authors refer to the objective function of Equation (2) but I guess they mean Equation (1). But then, their analysis does not take into account the tree complexity term in (1). If alpha is >0, then the objective function in (1) will actually start increasing. The theorem formulation should also make the bound explicit. Strictly speaking, L_max + D-k is also an upper bound on the objective function (with alpha=0) that the algorithm monotonically decreases but this is a useless result
- Theorem 3 is also very imprecisely stated. What is the probability of overlap? In practice, we will often have an overlap area that covers the whole space. In the proof, why is the probability P(FI_p^k <FI_p) an upper bound to the probability of discarding a feature? This is very unclear.
- Theorem 4 is not very useful as it is not specific to SDT. If you want to include it, then the proof of this theorem in Appendix A.1 is too short. It should be explained why Theorem 20.1 and Lemma 20.1 from (Devroye et al., 2013) apply.
- Theorem 5 is again very imprecise: What does it mean that "the objective function of equation (2) decreases as new nodes are added"? I guess the authors again mean equation (1) with alpha=0. Again if alpha>0, the theorem is false.

I think Section 4 needs a complete rewriting. The authors should take a look at [*], where a bound is provided on the probability of picking a bad weak learners.

Experimental results are very good, with the method obtaining a significant improvement of computing times on most problems without impacting too much accuracy. Limitations of the experiments are as follows:
- The number of problems considered is rather small and lack diversity with respect to standard in the domain (i.e. supervised classification).
- Discussion of the results is very minimal. I would have like to see experiments to justify the different steps of the methods (in the absence of a convincing theoretical analysis). What about performing the feature selection in one step instead of alpha steps (using N0 or alpha*N0 examples to select directly 0.5% of the features)?
- The use of several fixed depths for the trees makes it more difficult to analyse the results and compare the methods. Why not computing the accuracy of trees pruned on a separate validation set, to reduce each method to only one score? This would be closer to how these methods are used in practive.
- Not sure why C4.5 is working so poorly with respect to CART trees? This result looks strange to me.
- Results with Haar trees are interesting as representative of problems with many features but I don't see the practical interest of using single decision trees in computer vision problems. One knows that using ensemble of trees or convolutional neural networks will work much better on such problems (getting below 90% on MNIST is not impressive).
- From the code provided, it seems that the algorithm is implemented in pure matlab. I have not analysed the code thoroughly but matlab does not seem like the best language to implement decision trees (that are not naturally implemented through matrix/vectorial operations). I'm wondering how much this affects the conclusion of the study. How does the computing times for CART in matlab compares with CART as implemented in Scikit-learn (that implements trees in C (cython)) or other standard efficient implementations mentioned earlier?

Minor comments:
- The introduction is not well written. To me, the long discussion and the many references about feature selection methods are irrelevant. Feature selection can only reduce computing times if the time needed for the selection is low compared to the training time of the model and this is not the case for all feature selection methods.
- Several references are broken in the paper (bottom of page 2, Page 3 after Equation (1)).
- Alpha is used both in Equation (1) and to denote the number of iterations in the algorithm.
- In Figure 1 and 2, the x axis is the computing time, not the computational complexity.
 The idea is potentially interesting and the empirical results are good but there are too many problems in the paper at this stage to recommend acceptance (mainly the imprecise theoretical analysis and limited experiments).


The paper presents a simple iterative sub-sampling procedure that can substantially reduce the runtime required for finding a split on a numeric feature in axis-parallel decision trees. In each iteration of the algorithm, the lowest-ranked half of features is discarded from further consideration, and the amount of data used for computing the value of a feature is increased for the next iteration, starting with a very small initial set of data. Once a user-specified maximum number of iterations has been reached, standard split selection is performed based on the remaining features and the full dataset available at the node. The paper derives a bound on the runtime, proves that the algorithm will monotonically decrease an upper bound on the split metric, and shows that the probability of discarding a feature depends on the overlap of classes wrt the feature (in a binary classification scenario). It also shows that the algorithm produces a consistent estimator and that the objective function decreases monotonically as new nodes are added. Experimental results on six high-dimensional datasets show that the proposed method produces decision trees that are competitive with standard trees that are obtained after pre-filtering features using chi-square or mRmR, but at a substantially reduced training time. The paper also has an additional experiment on MNIST and F-MNIST looking at trees generated using Haar features. The paper presents an interesting collection of experimental results. A drawback of the proposed method is that it has three hyperparameters (\alpha, C, n_0). The paper does not study how sensitive the result is to the choice of these hyperparameters, and it seems non-obvious how to choose them in practice.

The following statement seems problematic: "The Haar trees are the first axis-aligned trees to achieve higher accuracy than any other axis-aligned trees as per our study." It is not clear to me that the Haar trees are axis-parallel in the *original* feature space. Of course, transforming the feature space and building an axis-parallel tree in the new space can bring significant improvements on datasets such as MNIST. 


Other comments
~~~~~~~~~~~~~

Abstract (and possibly in the introduction):

Not all decision trees are binary trees.

Not all decision tree learners are greedy algorithms.

Threshold-based splitting applies to numeric features only.

Not all algorithms use exhaustive search to find the best split.

The exhaustive search is linear time assuming pre-sorted features for standard splitting criteria, which is fast.

Mentioning MNIST and 94% accuracy as the only result is not a good motivation for the reader.

How is inference time improved?

Introduction:

Make it clear that the discussion considers "univariate" trees only and that there are oblique trees.

The stated time complexity is not correct when features are pre-sorted, which should be discussed.

The references are formatted incorrectly.

The description of embedded approaches does not seem quite right.

There are broken references. Unfortunately, these appear to be to the most pertinent work.

Section 2.1:

"Every decision tree.." - no

Broken reference.

"... is an entropy function" - not necessarily

Time complexity is incorrect if features are pre-sorted.

Section 5:

What metric is used for finding splits with the proposed method in Section 5?

Please include legend in Figure 2 as well.

The proposed method has three hyperparameters (\alpha, C, n_0). How sensitive is performance wrt these hyperparameters?

The RBF-SVM seems highly non-standard. Can this be called an SVM? It seems strange that k-means performs better than this SVM on the MNIST data.

Kudos for including OC1 in the experiments!

What kind of accuracy do you get using a random forest with axis-parallel splits? In the absence of competitive classification performance wrt to neural networks, the primary benefit of using trees is interpretability. The paper does not study this aspect. Runtime of greedy decision tree learning is not normally considered a problem in practice, particularly once the feature set has been reduced using pre-filtering. However, runtime is the focus of the paper. The impact of what is presented is thus likely to be very small, even ignoring the problem of hyperparameter tuning.

This paper focuses on improving the computational efficiency of the construction of a decision tree. The paper proposes a scheme that combines row sub-sampling and an adaptive column subsampling to reduce the computational cost of finding the best split at each internal node during the tree construction. The authors present some theoretical guarantees for the computational gain and the reduction in the loss function. The empirical evaluation compares the train/test error vs computational complexity tradeoffs of the proposed scheme against various baselines and demonstrate significantly better tradeoffs, at times demonstrating over 2-3 orders of magnitude speedups. The empirical evaluation also highlights an application where the number of columns are very high -- a decision tree for Haar features of images, again showing significantly better tradeoffs than considered baselines.
 I believe one of main strengths of this paper is that it is focusing on a very important problem of improving the empirical efficiency of decision tree construction given decision trees are widely used for tabular data both individually and as members of tree ensembles. Another strength of this paper is the strong predictive performance vs computational cost tradeoff of the proposed Stochastic Decision Trees (SDTs) relative to the considered baselines. The SDTs are shown to not only improve the computational cost, at times demonstrating over 2-3 orders of magnitude speedup over baselines, but also showing better test performance than existing exact axis-aligned decision trees.


I believe that the main weakness of the paper is that there is a lot of existing work on improving the computational efficiency of decision tree construction that were not considered in this paper as baselines. One of the most common ways of improving the cost of finding a split in an internal node of a decision tree is row and column subsampling (in each node) [1] and is part of various decision tree libraries such as XGBoost [2] and LightGBM [3] (these focus on decision tree ensembles that require construction of multiple decision trees and hence have a even higher computational overhead). The proposed SDT can be viewed as a more adaptive form of row/column sampling for finding the split, but it is important to see how much empirical improvement this adaptive scheme in SDT brings over just random row/column subsampling (present in existing implementations). Note that this row/column subsampling is different from the adaptive resampling and other feature selection techniques considered as baselines since they either require reconstruction of trees from scratch (increasing computational cost) or removing columns completely from tree construction (potentially hurting predictive performance).


Another common way to speedup the decision tree construction is the use of discretization and histograms [4], also heavily used in popular libraries such as XGBoost [2] and LightGBM [3]. This has shown to significantly speedup tree construction over the traditional $ O(D N \log N) $ scheme and used more commonly now. Most of these techniques are (or can be) used in conjunction with row/column subsampling. It is possible that the proposed technique in SDT can also be used in conjuction with existing techniques. But given the extremely efficient implementations (in some cases, with GPUs), it is essential to understand what kind of empirical performance (or predictive performance vs computational cost tradeoff) SDT provides over these existing techniques.


Beyond the consideration of existing schemes to improve the computational cost of decision tree construction, another limitation of the paper is that some of the theoretical results are very imprecise and appear to be somewhat trivial. For example, the claims in Theorem 2 & 3 appear quite vague and imprecise without the usual "assumptions + conditions lead to some guarantees". With such a setup, we have unknowns in the proof such as "iteration $k$" in the proof of Theorem 2 without any context of which loop is being discussed. Morever, in Theorem 2, the bound and the results are somewhat trivial -- if we make the upperbound really high, minimizing (or decreasing) the upper bound of an objective provides no guarantees on the true objective. In Theorem 2, it appears to be such as case. One might be able to get tighter (albeit probabilistic) bounds using Hoeffding's inequality or something similar. Furthermore, Theorem 3 appears to be stated in a way that appears to be imprecise -- it is not clear what is meant by "overlap"; it is not clear how the $Pr( FI^k_p < FI_p) $ relates to the probability that $p$ is in the top-half of the feature importances in the $k$-th iteration. I think we need some precise details (in the theorem statement and proofs) to understand the values of these theoretical statements.


Some other concerns/issues/questions:

- Given that the "computational cost" is key to the empirical results, it is not clear from the main paper how this "computational cost" is precisely computed for all the methods considered. I also quickly looked through the supplement and did not find anything but it is possible I might have missed it. Are we looking at actual wall clock time (which is very implementation dependent) or just at the theoretical runtime complexity (which are in $O( \cdot )$ form relying on unknown constants).
- Given the random nature of the proposed scheme and other sampling based baselines, it is not clear if the results were obtained with multiple trials, and it is not clear how robust the demonstrated performances are with respect to randomness. Confidence intervals around the performance/computation tradeoffs would demonstrate the robustness of the results.
- It is not clear why a decision tree on Haar features is named "Haar Trees". Since all decision trees are built on the same Haar features, won't all the decision trees also be "Haar Trees"? Or is it the case that the proposed SDT is built on Haar features while all other decision trees are built on the original features (pixels) of the MNIST/FashionMNIST datasets.
- In Figures 1 and 2, the same (or very similar) colour+marker combination is used for multiple methods in some cases (such as C4.5 and mRmR1%). This makes it hard to follow the results.
- It is not clear how thoroughly the hyperparameters of the baselines were optimized compared to the proposed SDTs. If it is discussed in the supplement, can you please point me to the right section.


[1] Friedman, Jerome H. "Stochastic gradient boosting." Computational statistics & data analysis 38.4 (2002): 367-378.

[2] Chen, Tianqi, and Carlos Guestrin. "Xgboost: A scalable tree boosting system." Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 2016.

[3] Ke, Guolin, et al. "Lightgbm: A highly efficient gradient boosting decision tree." Advances in neural information processing systems 30 (2017): 3146-3154.

[4] Tyree, Stephen, et al. "Parallel boosted regression trees for web search ranking." Proceedings of the 20th international conference on World wide web. 2011.
 I recommend a strong reject based on the weaknesses detailed above. I believe that there are various existing schemes that should have been compared with and/or positioned against. Moreover, the theoretical results (which is claimed to be a significant contribution) are stated in quite imprecise manner, making the results appear not as significant. Moreover, the presented theoretical results do not guarantee anything about optimality either (which is claimed as shortcomings of some existing methods).


Meta-review: All reviewers are very consistent with their evaluation of the paper. The discussion phase did not change their initial evaluation. Therefore, I also recommend to reject the paper.

Decision: Reject