Nearly-Tight and Oblivious Algorithms for Explainable Clustering

Abstract: Buddhima Gamlath, Xinrui Jia, Adam Polak, Ola Svensson

Reviews and author responses:

 Thank you for your time in reviewing our paper. We agree with your comments on the independent work.

Indeed, we were unaware of these independent works (including the ICML paper by Makarychev and Shan) until these papers became available on ArXiv (about five weeks after the NeurIPS deadline). We became aware of a few more papers in the same situation but we do not know which of these were submitted to NeurIPS.

 Thank you for your time in reviewing our paper.

Regarding the cost function: The cost function for k-medians is the same as in the original paper by Moshkovitz et al.. Even though studying the l_p norm (to the power of p) is one natural choice, we agree that it is also natural to study the sum of l_2 Euclidean distances. This makes the explainable problem less “clean” but nevertheless independent works have studied this setting and achieved similar guarantees for that version.

Regarding expectation: When desired, there is a standard method to convert the expectation bound to a with-high-probability bound -- by Markov inequality the cost is below twice the expectation with probability at least 0.5; repeating the algorithm O(log n) times and taking the cheapest clustering yields the desired bound w.h.p. -- that however requires looking at the n data points to evaluate the clustering cost, while obtaining a clustering with good expected cost can be done with our algorithm by looking at only k centers. Actually, one can see that it is impossible to get very strong concentration bounds without looking at the points: take two centers located at (0,0,...,0) and (1,1,1,...,1) and a single point with coordinates (0,...,0,1,0,...,0), i.e., all 0’s except a single 1 in one out of the d dimensions. Now if we select this dimension at random, then any algorithm that doesn’t look at the points will output a solution of value d with probability at least 1/d whereas the cost of the initial clustering was 1.

Regarding experiments: We agree that an empirical evaluation of our algorithm would be interesting but we also think that a thorough empirical evaluation is beyond the scope of this paper. Indeed, our motivation in this work is to improve our theoretical understanding of the problem, just as the original paper by Moshkovitz et al. does, which does not contain an empirical evaluation.  

Regarding a conclusion: We will add a conclusion/future work section.

 Thank you for your time in reviewing our paper. Addressing your comments and questions:

1. We will only raise this conjecture in the limitations paragraph.
2. We agree that an empirical evaluation of our algorithm would be interesting but we also think that a thorough empirical evaluation is beyond the scope of this paper. Indeed, our motivation in this work is to improve our theoretical understanding of the problem, just as the original paper by Moshkovitz et al. does, which does not contain an empirical evaluation.  
3. On a formal level, the gap is established by the lower bound for k-means being higher than the upper bound for k-medians; in principle, the gap is inevitable. On a more philosophical level, k-means is often harder to prove good guarantees for than k-medians because there is no triangle inequality for k-means. In the case of explainable clustering, the difference also arises because the increase in cost by not  assigning a point to its closest center has a more dramatic impact for a quadratic objective function (k-means) compared to a linear (k-median).


 Thank you for your time in reviewing our paper. We agree that an empirical evaluation of our algorithm would be interesting but we also think that a thorough empirical evaluation is beyond the scope of this paper. Indeed, our motivation in this work is to improve our theoretical understanding of the problem, just as the original paper by Moshkovitz et al. does, which does not contain an empirical evaluation.  


This paper considers the explainable k-clustering problem with respect. Here we are given points in Euclidean spaces and seek to impose a decomposition of the space by orthogonal queries such that there exists a bijection between clusters and leaves of the decomposition. For this problem, the authors give a $\log^2 k$ approximation for k-median and a $k\cdot \log^2 k$ approximation for $k$-means. Furthermore, the authors provide nearly matching lower bounds.  The algorithm proposed here is very simple  which is a good thing. Given a $k$-clustering that is not necessarily explainable, the authors give a black box transformation that makes the clustering explainable. What is nice about this is that, given the centers of the clustering, it is not necessary to scan the input again, i.e. the computational overhead is basically non-existent.  

For $k$-median, one merely samples cuts from the box bounding all the centers uniformly at random. The algorithm recurses on the set of non-separated clusters. For higher powers such as $k$-means, the authors prove a polynomial lower bound, separating k-median from k-means, along with a nearly matching upper bound. The algorithm for this case is similar; the main difference is that the cuts are only sampled among the cuts with enough separation.

I like the algorithms, which are very simple, as well as the analysis which is elegant. The nearly matching lower bound constructions are also nice. The negatives, if any, are not really the author's fault: I think that explainable clustering is an artificial problem that has little to no bearing on the use cases of $k$-clustering. My opinion was only reinforced by the large separation between $k$-median and $k$-means. One could also take this complaint as an affirmation of these results, with the authors demonstrating that perhaps "explainability" as it has been defined is not a meaningful notion for higher powers (which also puts it into question for $k$-median). In any case, I think it is good that this problem has now been resolved.

Unfortunately, the authors have been scooped by two recent ICML papers from Makarychev and Shan and Laber and Murtinho. The second paper by Laber and Murtinho has generally weaker (though not completely comparable) results and a more expensive algorithm with not the clean analysis presented here. The first paper by Makarychev and Shan has very similar results, both in terms of bounds as well as algorithms compared to this submission. I believe that the authors were not aware of these prior results, but the question is what to do about this paper (as well as at least 2 other papers in a similar situation -- if these papers are not known to the PC then I can provide arxiv links). Once the ICML proceedings are out, barring an improved analysis with not "essentially matching" bounds, but actually tight bounds, I believe that the 3 papers on explainable clustering will become unpublishable, or only publishable at smaller, less prestigious venues. What improvement there is in terms of bounds is only in terms of $\text{polylog } k$ factors for both the submission under review here and the other two papers that I believe have also been sent to NeurIPS, along with minor improvements in the low-dimensional regime.

Judging this paper in isolation, as I believe one should, it would have been a clear accept. The unfortunate circumstances that this papers finds itself in does not change my impression of the paper, which is positive, so my rating reflects that. Ultimately the question of acceptance is not on a scientific basis but on a policy basis.

However the PC decides on this paper, fairness dictates that all 3 explainable papers submitted to this NeurIPS should either be accepted or rejected (I am not affiliated with the authors of any of these submissions). In terms of merit, there is no reason to favor one over the other, in my opinion, and while all bounds and algorithms are similar, I think that every analysis is unique enough (in my current understanding) to merit acceptance. I therefore encourage the chairs that are responsible for the other papers to reach a consensus for all of the papers, given the rather unique situation. While I believe that it would be deserving if this paper and its competitors gets accepted, if everyone gets rejected then it will be disappointing but nevertheless fair and understandable for all parties.

Finally, as a side-issue: I believe that this should trigger an open discussion on what NeurIPS policy is on multiple submissions with similar results (including submissions to other conferences overlapping with the NeurIPS submission cycle). To the best of my knowledge there is no policy here, but I think having a unified guideline would make both reviewing and coordination simpler and more predictable.  Not necessary for this paper

Moshkovitz et al. recently propose a formal method to explain clustering algorithms: Build a decision tree where each internal node separates the points into two parts based on one dimension and a specific threshold.

The obvious question is price of explainability in this regime (with respect to the objective of k-means, k-median or some other \ell_p norm). Prior to this work, the best upper bounds were k (for k-median) and k^2 (For k-means), and the lower bound was log k. For small dimension d, one factor k in the upper bounds could be swapped with d log k.

This work makes a serious attempt at closing the gap.
1) They give k^{p-1} log^2 k upper bound (log^2 k for k-median and k log^2 k for k-means).
2) They give a lower bound of k^{p-1} (which means k for k-means).

The new results have two main ingredients:
1) Previous work did not use randomization. Given a non-explainable clustering, they would successively find a threshold cut (separating at least two clusters) that had the smallest "error" (in terms of assigning points to clusters). The new result uses randomization here.
2) The straightforward analysis brings in the depth of the tree into the approximation factor, which can be as bad as k (think of basis vectors). The authors show how to better charge the final error to opt and bring down the approximation factor.  Important topic, significant contribution. As the bounds are still bad for k-means, we probably want to look at other ways to explain clusters.

The writeup is not difficult to read. Experimental study would be nice to have to establish the relevance of improved bounds. Yes

This work provides novel algorithms for explainable clustering, which achieve smaller approximation constants compared to the previous state of the art results. The approximation constants are measured relative to the unconstrained given clustering instance. The paper handles a wide set of cost functions, e.g., k-means, k-medians, and in general l_p norm objective functions.
The authors also improve the existing lower bound on the explainable clustering approximation factors. 
The above results imply that the algorithms presented in the paper achieve nearly tight approximation constants.  Pros:
- The paper is very well written. It smoothly guides the reader through the results and claims.
- I liked the given algorithms and their theoretical bounds, specially when accompanied by the new lower bound.
- The overview of Algorithm 1 and the proof outline are very helpful.
- The presented algorithm and its required modifications are generally very clear and easy to implement.

Cons:
- Regarding the cost function definitions: The k-medians cost function handled in this paper is the sum of l_1 distances between every input point to its closest center. However, in many different papers that handle clustering, the k-medians cost function is defined as the sum of l_2 (Euclidean) distances between every point and its closest center (or generally, in the l_p clustering task, the cost is the sum of l_2 Euclidean distances to the power of p). See e.g.,
(i) Arora, Sanjeev, Prabhakar Raghavan, and Satish Rao. "Approximation schemes for Euclidean k-medians and related problems." Proceedings of the thirtieth annual ACM symposium on Theory of computing. 1998. or
(ii) Feldman, Dan, and Michael Langberg. "A unified framework for approximating and clustering data." Proceedings of the forty-third annual ACM symposium on Theory of computing. 2011.
Is there a reason for this difference?
Is it possible to generalize the given results to handle the k-medians cost function defined in the above papers? 
- All the results seem to hold only in expectation. Is there a hope to develop a bound on the worst case probability of success?
- I am missing some simple experiments that test the empirical approximation constants, and compare the empirical results with previous algorithms, e.g., that of [9].
Furthermore, as common in theory papers, the empirical approximation factor are usually smaller than predicted by the worst case theory. Has this been tested? Is this also the case here? 
As the algorithms presented in the paper are very simple, I do not see why not to conduct such experiments.
- The paper lacks a discussion / conclusion / future work section.


Minor comments:
- At Line 325, the authors claim that "sampling from such a distribution seems very complicated if at all possible". Can the authors elaborate more on why this might not be possible?
- Line 350: "We prove, Appendix D .." -> "We prove, at Appendix D...". Yes.

A clustering is explainable if its cluster is partitioned by a decision tree whose internal nodes are axis-parallel lines. Given a reference clustering, usually a set of k centers, the goal is to transform it into an explainable clustering with a small factor increasing in the objective. 

This paper provides new algorithms and improved analysis for explainable k-median and k-means clustering. For k-median, their analysis shows the algorithm achieves an O(log^2 k) *OPT objective when transforming an optimal clustering with objective value OPT. For k-means clustering, they achieve an O(k log^2 k) upper bound. A new lower bound for explainable k-means clustering is also provided in this paper.

Their results improve the original paper of Moshkovitz, Frost, Dasgupta, and Rashichian, which gives the upper bound of O(k) and O(k^2) for k-median and k-means.  I agree with the motivation of explainable clustering. To decide the cluster of a single point with a decision with the form I[x_i<=c] is a user-friendly structure. 
This paper is an addition to this topic. For k-median clustering, their analysis shows we only need to afford a poly(log k) factor to transform a non-explainable solution to an explainable one. 
Their algorithm is very simple. The analysis uses new observations and gets rid of the dependence on the height of the decision tree, which is Omega(k) in the worst case.

Some comments/questions:
1.  The authors conjecture their algorithms are tight but their analysis is not. I suggest such a statement should only appear in the Limitation paragraph. Currently, it is mentioned in multiple places in the main text, e.g. Line 246.
2. As the algorithm is simple, and the k-median clustering only has a poly(log k) loss, it will be convinced to implement the algorithm and run experiments on real data. 
3. Is there any possible reason why there is a polynomial gap between the upper bounds of loss for explainable k-median and k-means? Why is explainable k-means inherently harder than k-median?

Evaluation: I do think this paper has some interesting results. While these results look incremental upon Moshkovitz, Frost, Dasgupta, and Rashichian. The algorithm seems to be simpler than the previous work. Implementation of the algorithm will certainly earn more credits on novelty.  yes

Meta-review: Considering that the ICML papers are concurrent works, the results in the paper are substantial improvement over previous works in several interesting problems. In the camera-ready version, please include a comparison with the concurrent works and highlight the unique insights and techniques developed in this paper.

Decision: accept