Title: Coresets for Clustering with Missing Values
Authors: Vladimir Braverman
Shaofeng H.-C. Jiang
Robert Krauthgamer
Xuan Wu
Venue: NeurIPS 2021
Keywords: coreset
clustering
k-means
missing values

Abstract: We provide the first coreset for clustering points in $\mathbb{R}^d$ that have multiple missing values (coordinates). Previous coreset constructions only allow one missing coordinate. The challenge in this setting is that objective functions, like \kMeans, are evaluated only on the set of available (non-missing) coordinates, which varies across points. Recall that an $\epsilon$-coreset of a large dataset is a small proxy, usually a reweighted subset of points, that $(1+\epsilon)$-approximates the clustering objective for every possible center set.

Our coresets for $k$-Means and $k$-Median clustering have size $(jk)^{O(\min(j,k))} (\epsilon^{-1} d \log n)^2$, where $n$ is the number of data points, $d$ is the dimension and $j$ is the maximum number of missing coordinates for each data point. We further design an algorithm to construct these coresets in near-linear time, and consequently improve a recent quadratic-time PTAS for $k$-Means with missing values [Eiben et al., SODA 2021] to near-linear time.

We validate our coreset construction, which is based on importance sampling and is easy to implement, on various real data sets. Our coreset exhibits a flexible tradeoff between coreset size and accuracy, and generally outperforms the uniform-sampling baseline. Furthermore, it significantly speeds up a Lloyd's-style heuristic for $k$-Means with missing values.

TL;DR: We provide the first coreset and near-linear time PTAS for clustering with (multiple) missing values

Review discussion:
Coresets for k-means clustering (and related problems) is a heavily studied area, and some improvements in paper seems incremental and not so consequential.  I feel this paper is *not* of that sort.  Missing values in data is a real and serious issue, and this paper basically provides the first scalable solution.  The main innovation is a provably accurate way to construct the coreset that then integrates (and relies on) natural and standard algorithms (Gonzalez & Lloyds, etc).
Moreover, I found the writing of the paper good.  It does not develop any especially deep or important techniques (although the dynamic k-center result is something), but it expertly borrows and builds on several choice advances from the literature.  Sometimes non-standard approaches were needed, and the paper does a good job of pointing out what was standard and what needed extra care and development.  After highlighting the main theoretical developments, it leaves space to carefully describe the experiments which do a good job of demonstrating the algorithms empirical advantages.
I do have one main complaint/suggestion.  The overall algorithm is not totally clear.  Section 1.2 does an ok job of overviewing it, but that section also provides intuition for various choices and where simple techniques fail.  I suggest the paper adds a short section just outlining the main algorithm -- there should be space.
No
* We will add an explicit and self-contained description of the algorithm in the revision.
Seems we are in agreement this paper should be accepted.  I see no major concerns, and agreement the paper is interesting.
Pros:
- In overall I really like this paper. The reduction to the dynamic k-center problem as well as the solution for this problem via the restriction of the input set into multiple sets with not missing values is very interesting.
- The paper is mostly well written.
- The paper presents significant improvement of the state of the art for this problem.
- The provided coreset is original, and seems very useful for real-world problems, as shown in the experimental results section.
Detailed comments and suggestions:
- I did not find the exact definition of \tilde{O}, which is very important.
- There is a typo in the legend of Fig. 4(b).
- The comparison to the previous work of MF19 is not very accurate. MF19 handles any input set of lines, which are not necessarily axis-parallel. The proposed work handles j-dimensional subspaces (for j >=1), but which are axis parallel.
- It was difficult to sum-up the final coreset construction algorithm, as it is spread across the whole paper. A stand-alone pseudo-code would be very helpful.
- In some claims the dependency on some constants, e.g., the probability of failure \delta, is explicitly stated, while in others it is not. I would suggest explicitly adding the dependency on all parameters / constants in all the claims.
- The standard deviation in the experiments is reported only for the Russian housing dataset. Is there a reason why this is not reported for the other datasets?
- Missing a conclusions and future work section.
No
* For the deviation in the experiments, we provided variance only for the RussianHouse experiment in the submission due to page limit.
In fact, our coreset achieves a smaller variance than uniform sampling in every experiment. We will add the other plots in the revision.
* We will try to discuss the future work in the revision.
For instance, this could include our insights about the lower bounds and other possibilities of tradeoffs of parameters.
For the minor comments:
* We will add an exact definition for $\tilde{O}$ in the revision.
* We will fix the typo in the legend of Fig 4(b) in the revision.
* We will add a more detailed and accurate discussion about MF19 in the revision.
* We will add an explicit and self-contained description of the algorithm in the revision.
* We will make the dependency on all parameters clear in the statements of claims in the revision.
The paper gives the first coreset handling clustering points in $\mathbb{R}^d$ that have multiple missing values.
The coreset construction is based on the importance sampling framework introduced by Feldman and Langberg. The main challenge in this coreset with missing values is that distances do not satisfy the triangle inequality, hence importance scores cannot be easily computed. To overcome this hurdle, the authors used a method introduced by Varadarajan and Xiao for projective clustering.
I enjoyed reading the paper, the paper is well written, the results are interesting (also very strong), and the authors evaluate the performance of their results on real and synthetic datasets.
comments:
1) Very important: please provide a detailed paragraph about the novelty of your work.
It seems that the paper relies on other papers for suggesting the coreset, while I believe that there is something very strong hidden in the approach --> please elaborate more on this point, since it is very important.
2) Remove the additional n in "Running" in Figure 4 (b).
No
* We summarize our novelty in the following paragraph, which we will add to our paper. These points are correctly mentioned in the other reviews.
Our coreset construction builds upon the sensitivity-sampling method, e.g., [Feldman, Langberg 2011].
However, a central technical challenge is that the standard method to compute the sensitivity scores breaks, because distances between points with missing values do not satisfy the triangle inequality. We overcome this using another known method, of [Varadarajan, Xiao 2012], that requires a coreset for k-center. Our main innovation is a near-linear time algorithm that computes an O(1)-approximate k-center coreset for points with missing values. To this end, we need the following key steps, which constitute our main technical contribution.
We reduce the k-center coreset construction with missing values, to the construction of traditional k-center coresets (i.e., without missing values) on a series of instances. These instances are built by restricting data points with missing values to a carefully-chosen collection of subspaces. The guarantee needed from this collection is a certain combinatorial structure, and we indeed prove it exists.
The method of Varadarajan and Xiao executes the k-center coreset algorithm many times, and overall takes quadratic time. To improve the running time, we design an efficient dynamic algorithm for the well-known Gonzales’ algorithm (which computes an O(1)-approximate k-center coreset). The main idea in this dynamic algorithm is to project the data points onto (data-oblivious) random 1D lines, and build on each line a dynamic data structure that supports furthest-neighbor queries (in 1D).
Finally, we implemented our algorithm and the experiments indicate that our algorithm is efficient and accurate enough to be potentially applicable in practice.
* For the lower bound, we recently obtained one that justifies our exponential dependence on $min(j,k)$, and we will add this to the paper. The proof is short but a bit tricky, and shows inputs with $d,j,k = \Theta(log n)$, for which every 0.5-coreset must have size $\Omega(n) \geq \Omega(exp( min(j,k) ))$.
* We provided variance only for the RussianHouse experiment in the submission due to the page limit.
In fact, our coreset achieves a smaller variance than uniform sampling in every experiment. We will add the other plots in the revision.
* We will try to discuss the future work in the revision.
For instance, this could include our insights about the lower bounds and other possibilities of tradeoffs of parameters.
* We will address other minor comments in the revision.
I read the author's response and the other reviews. Thanks for the detailed answers to my concerns.
The lower bound and the detailed novelty of course just strengthing my support to increase my score.
This is the first work to give coresets for multiple missing coordinates, and it improves the coreset size upon existing results even for the one missing coordinate case.
In most papers regarding coresets, there are two meta approaches -- either via aggregation using a fine mesh, or via using importance sampling, especially using the framework developed by Langberg et al. This work uses an existing framework to reduce the importance-score computation to the computation of k-center coresets,and being able to compute these k-center coresets for missing coordinates is the main technical contribution. At its heart is a nice implementation of an existing algorithm that makes it more efficient. I also found the restriction into the collection of subsets to be a very interesting construction.
Overall the work seems quite novel and is nicely written. My only slight crib is about the experiments.
While uniform sampling is the most obvious baseline to compare with in this setting, perhaps it would be useful to see some other baselines that might not have a theoretical guarantee (at least in the worst-case setting).
For instance, filling in the missing values using a suitable heuristic, and then using the standard k-means coreset -- how will that perform in practice.
No
* Thanks for the suggestion about adding a new baseline, and we will indeed add to the paper a comparison to this suggested baseline.
In particular, we will fill missing values using a standard imputation method and apply the well-known importance-sampling coreset (e.g., [Feldman and Langberg ‘11]). Our preliminary experiments indicate that the performance of this new baseline is better than uniform sampling but fall behind our coreset.
* We also have some recent progress regarding the lower bound.
In particular, we managed to prove a lower bound that justifies our exponential dependence on $min(j,k)$, and we will add this to the paper. The proof is short but a bit tricky, and shows inputs with $d,j,k = \Theta(log n)$, for which every 0.5-coreset must have size $\Omega(n)\geq \Omega(exp( min(j,k) ))$.

Meta-review:
role: PC
data: comment: The paper provides core set construction for clustering such as k means with multiple missing values. All the reviews are highly positive and support acceptance. The authors should carefully address all the comments that the reviewers have brought up including adding new baselines to their experimental evaluation.

Decision: Accept (Spotlight)