Coresets for Decision Trees of Signals

Abstract: Ibrahim Jubran, Ernesto Evgeniy Sanches Shayda, Ilan Newman, Dan Feldman

Reviews and author responses:

This work creates coresets for axis aligned decision trees. A decision tree is defined as a partition of the data (set of vectors + label) into axis-parallel rectangles such that all vectors in the same leaf get the same predicted label, and the cost of the tree is defined in terms of some loss (MSE in current paper) between the predicted and given labels. For a given dataset, this work constructs a “strong” coreset for all decision trees with k leaves. The coreset is linear in the number of decision tree leaves and logarithmic in the datapoints and classes.   Decision trees are clearly an important set of models, and while the combinatorial structure that they impose on the data space is similar to the structure posed by range queries, there has not been enough work on the coreset construction question. The fact that there exists datasets (even in dimension=1) or which there are no sublinear size coresets follows from existing work. This work makes a specific assumption on the dataset -- that “every coordinate has a label”. This seems too strong an assumption. Frankly, it would be nice if the authors could explain why this is a useful assumption to make in getting around the lower bounds, as well as why this reflects real data properties (while this assumption is not directly satisfied, what is the closest thing to it that exists in real data).

The coreset construction is deterministic and utilizes a nice connection between partition tree data structure and decision trees. To me, this also seems to be a novel contribution. The coreset is constructed based on a union of points chosen from each partition of the resulting partition tree. 

Overall, the paper is well written, and the illustrations are helpful. The experimental results do show a significant improvement over uniform sampling, but only over sample size that is < 5-8% of the data. Beyond these, the advantage is not very significant. This is not very uncommon for the performance of coreset algorithms in real datasets.  The only real weakness I see is the strong assumption made for theoretical proofs. 

-
Post response: Thanks, I am satisfied with the response. 
 None. 

 We first thank the reviewer for the careful reading.
 
*Reviewer*: The paper did not adequately compare the results with previous works, and i’m not sure which of the techniques are entirely new.
- Reply #1: We added more clarification and citations in the "Novelty section".
As stated in this section, the main novelty is the (very non trivial) generalization of partition trees to decision trees.
The second novelty is the result itself: not surprisingly, since decision trees are arguably the most common ML (excluding DL) classifier, many researchers, including the authors and their colleagues tried to compute coresets for decision trees for years with no success. This is also due to the third and fourth novelties: the relaxation to signals that saves the theory and works very well in practice, and our novel bicriteria algorithm.
 
*Reviewer*: Where does the log(N) factor in the coreset size stem from?.
- Reply #2: The log(N) factor stems from both the \alpha as well as the \beta terms, as both of them depend on log(N) (Lemma 5).
 
*Reviewer*: Why can you replace the coordinates without violating the approximation guarantees?
- Reply #3: The main idea in our paper is to partition the input into a small number of subsets, such that: (i) Not many subsets can be intersected by a single decision tree, and (ii) the variance of each subset is small. Then, in Algorithm 3, we compress each subset individually and return the union (C,u) of those compressions. We prove that the above two properties suffice in order for (C,u) to be a coreset.
As for the reviewer’s concert: replacing the coordinates of entries inside each subset, while keeping the same labels, does not affect the variance of this subset (i.e., property (ii) is still maintained). Property (i) is trivially still maintained since we swap the coordinates of points inside such a subset, with other coordinates from the same subset. Hence, this operation is valid.
Following the reviewer’s concern, an explanation will be added to the main text.
 
*Reviewer*: How did the authors handle the real-world datasets with dimension higher than 2?
- Reply #4: Those datasets are essentially a 2D matrix with missing entries. We will elaborate more on this in the paper.
 
*Reviewer*: What is a “smooth coreset”? It seems that this notion only appears in the analysis of Algorithm 3.
- Reply #5: The reviewer is indeed correct. This notion is used mostly for the analysis of algorithm 3. A more formal definition was given in the supplementary material. A brief explanation:
Iin Algorithm 3 we computed some small compression C_B, along with a weights function u_B, for every subset B in the partition of the input. The size |C_B| of this compression is a small constant, independent of the (potentially large) size of B. The pair (C_B,u_B) satisfy a set of properties, which we tried to visually demonstrate via the “smooth coreset” notion (which is illustrated on the right of Fig. 3). Informally, the “smoothed version” of (C_B,u_B) is another pair (C_B’,u_B’), such that C_B’ contains a duplication of the elements of C_B. The number of duplications of every element c from C_B is according to its weight u_B(c). 
Following this concern, we will elaborate more on this notion, and point the reader to the formal definition in the supplementary material.


 We first thank the reviewer for his careful reading and for pointing out the strengths in our work.
 
*Reviewer*: There is very little intuition
- Reply #1: As pointed out by the reviewer, the space limitations are usually a serious obstacle. Following the reviewer’s concern, we will add more intuitions and general overviews to the main text, or at least to the supplementary material.
 
*Reviewer*: Notation suggestion.
- Reply #2: An alternative notation will be considered. Suggestions or references to papers that might inspire a less confusing notation are very welcome.
 
*Reviewer*: Typos
- Reply #3: Thank you for pointing those out. Fixed.
 
*Reviewer*: why is there a running time difference between decision trees trained on random sample vs decision trees trained on the coreset?
- Reply #4: The coreset-related computation time presented in the graphs also includes the time to compute the coreset. Computing our coreset is more costly compared to randomly sampling a subset of points.


 *Reviewer*: “The coreset construction is deterministic and utilizes a nice connection between partition tree data structure and decision trees. To me, this also seems to be a novel contribution” + “Overall, the paper is well written, and the illustrations are helpful.”
- Reply #1: We thank the reviewer for appreciating our work.
 
*Reviewer*: Why is this a useful assumption to make in getting around the lower bounds? 
- Reply #2: It is useful assumption in the sense that:
(i) it seems to be the weakest assumption that can enable us to have a provably small coreset for ANY input.
(ii) It is natural for e.g. images, matrices or any input data from sensors (such as GPS) that has a value in every cell or continuous in some other sense.
(iii) It works great in all the experiments we have done so far.
 
*Reviewer*: Why does this reflect real data properties?
- Reply #3: Suppose that the input in practice contains the numbers 1,2,4,5..,n (where the value 3 is missing).
Our assumption basically "smooths" this input to "1,2,3,4,5..n" (including 3).
This seems a natural and harmless interpolation, at least intuitively.
Another observation is that this assumption is not needed for the majority of the cells in our partition. Only for the very few one that are intersected by the optimal decision tree. The other non-intersected cells can be approximated without the need for this assumption.


This paper constructs coresets for $k$-decision trees for 2D-signals. Think of 2-D signals as a 2-dimensional $m x n$ matrix and $k$-decision trees as a partition of this matrix into k disjoint axis-parallel rectangles. Cost of a partition is defined as follows: For each part $P$, define a number $s_P$ that minimizes the squared loss for the entries in the partition. The number $s_P$ is actually just the mean of the entries in that partition. The authors propose coresets to approximate cost of any $k$-decision tree upto a factor of 1+eps while storing only $k\log(N)/\epsilon$ entries of the matrix along with the weights that they compute obtaining a significant reduction in time required to compute the cost of any rectangle and therefore the time required to compute the cost of a $k$-decision tree. The authors do some experiments that show that decision trees can be trained on the coreset much faster than they can be trained on the entire dataset and that the decision trees they obtain have an error on test set not much larger than the decision tree trained on the entire signal.

To construct the coreset, the authors first give a $(\alpha, \beta)$ bicriteria approximation algorithm to obtain a lowerbound on the optimal cost of a k-tree on the dataset. Then they use the bicriteria approximation to compute a balanced partition of the signal such that 1. The number of parts is not too large, 2. Optimum cost for each partition is small, 3. Every k-segmentation intersects only a few sub-signals. They prove that there is a simple greedy algorithm that gives a balanced partition with some parameters that depend on $\alpha, \beta$ and the cost of the bicriteria solution.

Then they show that for each part in the balanced partition, they only have to store 4 values and corresponding weights and this is enough to compute the $1+\epsilon$ approximaton to the cost of $k$-segmentation for the signal.   Strengths:
- Algorithms are simple and easy to understand.
- The proofs seem to be quite natural stemming from the fact that the algorithms are simple.
- First coreset construction for decision trees.
- Paper is well written.
- Experiments show that test error is close to the optimal error and the coreset does better than a uniform sample of same number of points.

Weaknesses:
- In the current form of the paper, there is very little intution on why they are doing what their doing. It is just a series of theorems without much intution as to why they need those parameters in the theorem and how proving one theorem helps in achieving the parameters of the next theorem. Need to add more details connecting the theorems. I understand it may not be possible due to space constraints but more context would greatly help a reader understand the paper much better.


Suggestion:
- Using $(x,y) \in B$ for a signal where $x$ is denotes a tuple and $y$ denotes a value is confusing. Maybe use a better notation?

Typos:
- Line 53 : foreset -> forest
- Line 184 : simplicical -> simplicial
- Definition 6: Should be $|\mathcal{B}| \le c_1$ instead of $c_1mn$

Questions:
- Why is there a running time difference between decision trees trained on random sample vs decision trees trained on your coreset? Is the slowdown mainly due to your algorithm to compute cost of a $k$-segmentation?
 NA

This paper studies the problem of constructing coresets for decision trees of 2D-signals. The coreset aims to preserve the min squared error of every axis-parallel rectangular k-partitions of the data set (where k is a parameter). However, it has been observed that no small coreset exists for this task, and hence the authors add an assumption that the data points are from discrete [m] x [n]. 

The authors propose an efficient algorithm for coreset construction. Empirical results show that their coresets can accelerate the computation time of random forests. The main approach is to first construct a balanced partition of signals via a bi-criteria approximate solution, and then construct an exact coreset for each partition using Caratheodory’s theorem.  Originality: This paper provides the first coreset result for decision trees. The approach of the construction of balanced partition may have independent interest. However, the paper did not adequately compare the result with previous works, and I’m not sure which of the techniques are entirely new, especially the balanced partition. Moreover, I didn’t find a related work section.

Quality: The technical claims are sound, and well supported by details. The pictorial illustrations and examples help the understand a lot, and excellent explanations are provided for key claims and algorithms. The experiments are well designed, and the choice of baseline and datasets in the experiments are appropriate.

Clarity: I find several parts that require clarification. 
-	The coreset size in Theorem 8 contains a log N factor, which seems from \alpha in Lemma 5 (bi-criteria approximate factor). If \alpha=O(1), does the coreset size still contain log N? If not, this point should be clarified.
-	Line 6 of Algorithm 3 is strange to me. Why replacing the coordinates will not violate the coreset approximate guarantee? It is better to give some explanations.
-	Right of Figure 3. I don’t understand the meaning of “smooth” coreset. It seems that this notion only appears in the analysis of Algorithm 3, am I right? In general, I think the authors should give some explanations for this “smoothed” coreset.
-	Line 305. The authors said that they implemented Algorithm 3. However, Algorithm 3 is designed for 2D-signals, which seems different from the setting of real-world datasets in Lines 313-314, whose dimension > 10. I wonder how the authors construct coresets for their real-world datasets.

Significance: The results are interesting and the work initiates the study of the coreset for decision tree problem which is important in coreset areas. 
 Yes.

Meta-review: The reviewers quite liked the first coreset for this fundamental machine learning model - decision trees - and the technical ingredients and ideas involved in the proof. The reviewers also felt the experimental results were solid. The only suggestions for improvement were to clarify more as to why this is the minimal assumption needed to skirt the lower bounds, and to add more intuition to the paper. 

Decision: accept