A Constant Approximation Algorithm for Sequential Random-Order No-Substitution k-Median Clustering

Abstract: We study k-median clustering under the sequential no-substitution setting. In this setting, a data stream is sequentially observed, and some of the points are selected by the algorithm as cluster centers. However, a point can be selected as a center only immediately after it is observed, before observing the next point. In addition, a selected center cannot be substituted later. We give the first algorithm for this setting that obtains a constant approximation factor on the optimal cost under a random arrival order, an exponential improvement over previous work. This is also the first constant approximation guarantee that holds without any structural assumptions on the input data. Moreover, the number of selected centers is only quasi-linear in k.  Our algorithm and analysis are based on a careful cost estimation that avoids outliers, a new concept of a linear bin division, and a multi-scale approach to center selection. 


Reviews and author responses:

The authors design a sequential random-order no-substitution algorithm for k-median clustering. Their algorithm outputs O(k\log^2 k) centers and gives constant factor approximation with high probability. The algorithm does not depend on any assumption on the dataset and gives substantially better guarantees over earlier works.  The problem setup is as follows. The points to be clustered arrive one by one in a random order stream. A point arriving in a stream may or may not be chosen as a center, but if it is chosen as a center, it cannot be substituted later. The authors design a subroutine SelectProc that divides the stream into three phases, the length of each phase is a constant factor of the length of the stream. At the end of the first phase, using an offline algorithm for k-median on the points in the first phase, a set of centers is computed. This set of centers is used in the second phase to estimate the cost of clustering of the points in the second phase. In the third phase, the actual centers are selected by the subroutine. The algorithm runs O(\log k/\delta) instances of the subroutine SelectProc in parallel where each instance of the subroutine works on a different prefix of the stream, and the algorithm outputs all centers chosen by these subroutines. This set of centers give constant factor approximation with high probability for the k-median problem.

Overall, I believe this to be a nice result for a fairly important problem. I recommend acceptance of this paper.

My only comments are regarding the writing and the presentation of the paper.
  - Since the description of the algorithm uses a number of parameters, it would be helpful to define the notations appropriately. For example, k_+ is used in the description of the algorithm but it is defined only in Section 4.2.
  - The authors should highlight that the performance of an algorithm in this setting is evaluated using the notion of competitive analysis of online algorithms.


- This additional comment is to acknowledge reading author response. There is no other changes. This work is theoretical in nature.

 Thank you for your response. Indeed, our algorithm provides a significant improvement in the approximation factor if k increases sub-linearly with n. For instance, if $k = \log(n)$, then we obtain a constant approximation factor, compared to an approximation factor of $O(n)$ in the ALT 2021 paper (Mo21). An improvement in the number of centers as a function of $n$ compared to Mo21 is obtained when $k$ is logarithmic or sub-logarithmic in $n$. 

 I see I mis-read the Table 1.  The ALT 21 paper has O(k^5) or O(k log n)  ... not O(k log k).  This makes the improvement better when k is small.  However, then the approximation ratio of exponential in k from that paper is not as bad either.  So really this is a useful improvement in some range when k is not too small (super constant) and not too large (not polynomial in n).  Of course in the k is large case, the approximation factor is also improved.  

Overall, as stated, I am ok accepting the paper.  Just not as excited as other reviewers.  

 Thank you for your review and your comments on the presentation. We will address these comments in the final version of the paper. 


 Thank you for your review. First, we would like to point out that the ALT 2021 paper selects $O(k^5)$  centers and not $O(k \log (k))$ (see Table 1 on page 3), while our algorithm selects only $O(k\log^2(k))$ centers. We further believe that improving the approximation factor from exponential in k to a universal constant is a substantial improvement.  

Regarding the practicality of the algorithm, indeed this paper focuses on the theoretical aspects. Nonetheless, we believe that this algorithm is quite practical. First, it is easy to implement our algorithm by selecting any existing practical offline clustering algorithm as the black box algorithm A. All the other steps in our algorithm are simple and computationally light. In addition, our algorithm can be implemented to have a small memory footprint. We provide a detailed explanation of how this can be easily achieved in our response to Reviewer JtCc.


 Thank you for your review and for your comments on the presentation. We will address these comments in the final version of the paper. 


 Thank you for your review. Regarding your question about memory requirements, the proposed algorithm can be easily implemented with a small memory footprint, by using a 1-pass streaming clustering algorithm as the black-box algorithm A. For instance, the algorithm of [1] requires $O(k \cdot \mathrm{polylog} (n))$ memory.
Our proposed algorithm can be implemented to require additional memory of only $O(k \log(k/\delta))$ on top of the requirements of the black-box algorithm. This question is shortly addressed in the discussion (Section 6). We give below a full explanation of how this can be done.

First, consider the memory requirements of SelectProc:

In phase 1, the points can be fed to A one by one and do not need to be saved in memory. Only $T_\alpha$, the $k_+=O(k+\log(1/\delta))$ output points of A, need to be saved in memory.

In phase 2, calculating the truncated risk of $T_\alpha$ on $P_2$ can be done by saving at most $2\alpha\cdot(k+1)\phi_\alpha = O(k \log(k/\delta))$ at any time, so as to identify the farthest points from $T_\alpha$, which are the ones to disregard.

In phase 3, the points are inspected one by one and do not need to be saved in memory (if one wishes to save the selected centers in memory, although this is not in principle required in our setting, then an additional $\log(1/\delta)$ factor is added).

In total, SelectProc can be implemented using a memory size of $O(k\log(k/\delta))+\text{the memory requirements of A}$. 


The main algorithm (MUNSC) runs $O(\log(k/\delta))$ parallel runs of SelectProc. Thus, it requires at most a memory size of $O(\log(k/\delta)(k\log(k/\delta)+\text{the memory requirements of A}))$.

In fact, this can be made even more memory-efficient, by using an algorithm A which is a streaming anytime algorithm. In this case, it is possible to use a single copy of A for all the parallel runs of SelectProc, since $P_1$ in the i'th copy of SelectProc is the prefix of $P_1$ in the $i+1$'th copy  (see Figure 1 on page 6). That is, A is run on the entire stream and whenever a phase 1 of any copy ends, its current set of centers is retrieved for this copy. In this case, the memory size required by MUNSC is  $O(k\log^2(k/\delta))+\text{the memory requirements of A}$.

References:

[1] Charikar, Moses, Liadan O’callaghan, and Rina Panigrahy. “Better streaming algorithms for clustering problems.” Proceedings of the thirty-fifth annual ACM symposium on Theory of computing. 2003


The paper discusses the k-median algorithm under the online, no-substitution setting. The paper considers the random order arrival of the points and obtains a constant factor approximation without requiring any structural assumptions. They provide an algorithm that gives a constant factor approximation with a $k\log^2k$ centers assuming the $n$ is known (in the discussion they state about the use of doubling to get similar results with $\log n$ overhead).  The paper is well structured and well written. The claims made in the paper seem sound. The problem itself is a relatively well studied theoretical problem and they manage to give a constant factor approximation algorithm that only has a $\log^2 k$ multiplicative overhead under the random order setting, which is a relatively significant improvement on the existing research. They introduce a new concept of linear bin division to complete the analysis of the algorithm. Overall the paper looks good.

Q: How does the Phase 1 and Phase 2 selection of $P_1, P_2$ (of size $\alpha n$ each) affect the memory requirements (could it lead to memory requirement of order $n$)?  The algorithm assumes knowledge of $n$, but the paper states how to remove this with an additional $\log n$ overhead.  They also assume random order. They leave the study of different input orders (other than random and adversarial) as a future research direction. 

As the problem itself is theoretical I don’t think it creates any major negative societal impacts.

The paper studies $k$-median clustering in the random-order streaming model. In this particular setting, the input is randomly permuted and fed to the algorithm element by element. The input itself is a unknown finite set of points that together with a distance function forms a discrete metric space. The goal of the algorithm is to construct a clustering, by selecting a only small number of the input points as cluster centers, such that the cost of the clustering approximates the cost of an optimal $k$-median clustering of the whole space. Imposed restrictions are that the algorithm can only pick an input element as cluster center before it sees the next element and centers that have been chosen can not be substituted.

The paper contributes an algorithm that returns with high probability of success a clustering with $O(k \log^2(k/\delta))$ centers that approximates the cost of an optimal $k$-median clustering by a constant factor.  The problem of the paper is interesting and has a strong practical relevance. It has already been studied in the literature and the results of the paper yield a substantial progress. Furthermore, the idea of the algorithm is novel to me and the analysis seems sound and correct (I checked the proof sketches and large parts of the proofs in the appendix).

Some minor comments:

- please provide the constant of the approximation in the theorem statement (something like $728$) instead of a "universal constant $C$"
- large parts of the clustering community call the objective value the "clustering cost", I have seen that some of the related work also speak of "clustering risk", but you may consider switching to "cost" (I don't see any reason to call it risk here)
- the symbol $\equiv$ is used very rarely in mathematics (mainly in the context of logic) and can be substituted by $=$ in any case in your paper (standard notation improves comprehendability)
- line 131: write $\lvert T_{out} \rvert = O(k \log^2(k/\delta))$ The limitations are adequately addressed, as the authors do, I see no direct path to negative societal impact.

This is a theoretical paper, studying sequentially assigning points as k-median cluster centers.  
It assumes a random arrival order, but has no other structural assumptions.  It offers a constant factor approximation using O(k log^2 k) centers.  This improves on a recent line of work in this model removing structural assumptions and the approximation factor.  

The main idea is using geometrically increasing prefixes of the stream to estimate the k-median solution and risk, and for each using this estimate to choose centers.  A key insight is that by increasing the size of centers maintained a bit, the risk can be approximated after excluding outliers -- which turns out to be the key step in the analysis.  
  From a theoretical perspective, the algorithm is intuitive, and is well explained by the paper, with a lot of insight.  But the bulk of the analysis is left for the supplement.  
The algorithm however, appears far from practical.  
And as the most comparable work is an ALT 2021 paper that uses O(k log k) centers, but has approximation factor exponential in k (under all of the same model/data assumptions), I do not see the improvement as that substantial.  

Ultimately, I'd be ok if this paper is accepted for NeurIPS.  The work is high quality.  However, the improvement, I believe is small and theoretical, and may not have a high impact.   The random order assumption is discussed clearly.  This was useful for the paper to do.  
This work is unlikely to have societal impact, and it was not discussed.  

Meta-review: The authors present the first constant approximation factor on the optimal risk for k-median with no -substitution under a random arrival order model. The problem studied in the paper is an interesting problem and the authors present a simple and elegant algorithm that outperforms previous work. The only concern with the paper is that the problem may not be of interest for a wide audience. Nevertheless, after the discussion phase, all the reviewers agree that it should be accepted.

Decision: accept