SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Leyla Biabani
1. biabani@tue.nl
Morteza Monemizadeh
m.monemizadeh@tue.nl
Annika Hennes
Heinrich Heine University Dusseldorf
Dusseldorf, Germany
annika.hennes@hhu.de
Melanie Schmidt
mschmidt@hhu.de
Given a point set  $P \subseteq M$  from a metric space  $(M, d)$  and numbers  $k, z \in \mathbb{N}$ , the metric  $k$ -center problem with  $z$  outliers is to find a set  $C^* \subseteq P$  of  $k$  points such that the maximum distance of all but at most  $z$  outlier points of  $P$  to their nearest center in  $C^*$  is minimized. We consider this problem in the fully dynamic model, i.e., under insertions and deletions of points, for the case that the metric space has a bounded doubling dimension  $\dim$ . We utilize a hierarchical data structure to maintain the points and their neighborhoods, which enables us to efficiently find the clusters. In particular, our data structure can be queried at any time to generate a  $(3 + \varepsilon)$ -approximate solution for input values of  $k$  and  $z$  in worst-case query time  $\varepsilon^{-O(\dim)} k \log n \log \log \Delta$ , where  $\Delta$  is the ratio between the maximum and minimum distance between two points in  $P$ . Moreover, it allows insertion/deletion of a point in worst-case update time  $\varepsilon^{-O(\dim)} \log n \log \Delta$ . Our result achieves a significantly faster query time with respect to  $k$  and  $z$  than the current state-of-the-art by Pellizzoni, Pietracaprina, and Pucci [18], which uses  $\varepsilon^{-O(\dim)} (k + z)^2 \log \Delta$  query time to obtain a  $(3 + \varepsilon)$ -approximate solution.
Clustering is a fundamental problem in machine learning and it has applications in many areas ranging from natural to social sciences. As a basic unsupervised learning method, it allows us to find structure in data. Classical center-based clustering methods for objectives like  $k$ -means,  $k$ -median and  $k$ -center have been around for many decades [9, 13, 16]. Since the beginning of clustering, the data to be analyzed has changed dramatically, raising new challenges for clustering methods. Data nowadays comes in large to huge batches and is often prone to inherent change: Content on social media platforms is constantly added, re-organized or deleted, streaming services handle an always ongoing flow of people starting and stopping to watch, tweets are the fastest method to distribute news – and delete them later if desired. Data analysis methods thus face the challenge of analyzing huge amounts of data, ideally permanently updating their solution.
In this paper, we consider the problem of dynamic  $k$ -center clustering with outliers. The (metric)  $k$ -center problem is to find a set  $C^* \subseteq P$  of  $k$  points such that the maximum distance of all points of  $P$  to their nearest center in  $C^*$  is minimized. This objective is well understood, 2-approximations for the problem are known [9, 13] and better approximations are not possible unless  $P$  equals  $NP$  [14].
Table 1: Fully dynamic  $k$  -center algorithms without outliers.
Year	Ref.	Metric	Update time	Approx
2018	[3]	general	amort. O(k2logδ/ε)	2 + ε
2019	[19]	Eucl. Rd	amort. O(εd log Δ log n)	16
2021	[10]	doubl.	ε-O(dim) log Δ log log Δ	2 + ε
2023	[1]	general	amort. O(k polylog(n, Δ))	2 + ε
Considering outliers for  $k$ -center is very natural. Since  $k$ -center minimizes the maximum radius, a single measuring error can destroy the structure completely. Thus, the  $(k,z)$ -center problem allows us to ignore  $z$  points. These belong to no clusters but are deemed to be outliers, and so they are ignored when determining the maximum radius of a cluster. The concept was introduced in 2001 by Charikar, Khuller, Mount, and Narasimhan [6] who as one result provide a 3-approximation for  $k$ -center with outliers. This variant has since been the subject of many papers, and the best-known approximation algorithm for it, which gives a 2-approximation, was derived as recently as 2016 [2].
In parallel, fully dynamic  $k$ -center algorithms (without outliers) have been developed [1, 3, 8, 10] which work with a changing point set given as a stream of insertions and deletions. All these maintain a data structure that allows for continual updates to keep adapting to the changing data. An overview is given in Table 1. All results in this table support some form of membership queries: Given a point, return the cluster / center of the cluster of that point in the approximate solution. This type of query can typically be answered in time  $O(1)$ . Reporting the centers takes more time or is not necessarily explained how to do.
Now the challenge is to combine the two parallel developments into a fully dynamic algorithm for  $(k,z)$ -center clustering. We discuss the setting more precisely and then our results and related work.
Our setting We say that a metric space has **doubling dimension** dim if dim is the smallest positive integer such that any ball of radius  $r$  can be covered by at most  $2^{\dim}$  balls of radius  $r/2$ . For an overview of this concept, we also refer the reader to [12, 11]. Let  $(M, d)$  denote such a metric space and let  $P \subseteq M$  be a set of points. By  $\mathrm{d}_{\min} := \inf \{d(x, y) \mid x, y \in P, x \neq y\}$  and  $\mathrm{d}_{\max} := \sup \{d(x, y) \mid x, y \in P\}$  we denote the minimum and maximum inter-point distance within  $P$ , respectively. Without loss of generality, we assume that  $\mathrm{d}_{\min} = 2$ . Let  $\Delta := \mathrm{d}_{\max} / \mathrm{d}_{\min}$  be the aspect ratio of  $P$ .
We are observing a dynamic stream, i.e., we start with a point set  $P = \emptyset$  and then process a sequence of operations whose length is unknown a priori. Each operation either adds a point from  $M$  to  $P$  or it deletes a point which is currently in  $P$  from  $P$ . We assume only deletions of points that are currently present are allowed, and we also assume that we can always compute  $d$  in time  $O(1)$  (otherwise, multiply all times mentioned in this paper by the time necessary to make one distance computation). We refer to the point set after time  $t$  (i.e., after  $t$  operations happened) as  $P_t$ . If it is clear from the context, we will sometimes drop the subscript. We assume that  $d_{\max}$  and  $d_{\min}$  are fixed throughout the whole sequence of updates, i.e.  $d_{\max} = \sup \{d(x,y) \mid x,y \in P_t, t \geq 0\}$ , and analogously for  $d_{\min}$ . Therefore,  $\Delta$  is an upper bound on the aspect ratio over all updates and is known in advance. This is in accordance with a sequence of other works [1, 4, 3, 7, 10, 19].
The problem that we address is the dynamic  $k$ -center problem with  $z$  outliers or in short  $(k,z)$ -center problem: Given  $P_{t}$  and numbers  $k$  and  $z$ , produce a set  $C \subseteq P_{t}$  of size at most  $k$  such that the maximum distance of all but at most  $z$  points to their nearest neighbor is minimized. More formally,  $\min_{Z \subseteq P_{t}, |Z| \leq z} \max_{x \in P_{t} \setminus Z} \min_{c \in C} d(x, c)$  is minimized among all possible choices of  $C$  with  $|C| \leq k$ .<sup>1</sup>
1Notice the small detail that we do not ask to compute  $Z$  but only to provide centers that are good centers under the assumption that at most  $z$  points can be ignored. In the offline setting, finding  $Z$  can be done by assigning every point in  $P$  to its closest point in  $C$  and then ignoring the  $z$  points farthest away. In the dynamic setting and with our data structure, we could also produce the outliers. However, we believe that the outliers are less interesting than the centers, and since there could be substantially many outliers, we rather only report the cluster centers to avoid a linear dependence on  $z$ .
Table 2: Fully dynamic  $k$  -center algorithms with outliers and their update/query times.
Year	Ref.	Metric	Update time	Query time	Approx
2022	[4]	general	O(1/ε(k+z)2 log Δ)	O(1/ε(k+z)2 log Δ)	14 + ε
2023	[18]	doubl.	1/εO(dim) log Δ	1/εO(dim)(k+z)2 log Δ	3 + ε
2023	[7]	doubl.	O(k/εdim + z) log4(kΔ/εδ)	O((k/εdim + z)2k log(k/εdim + z))	3 + ε
now	Thm 1.1	doubl.	1/εO(dim) log n log Δ	1/εO(dim) k log n log log Δ	3 + ε
Our focus is on building a dynamic algorithm that has a low update time for insertion and deletions. We are allowed to store  $P$  (and additional information). Our aim is to achieve an update time for our data structure that is independent of  $k$  and a query time that is at most linear in  $k$ . For this, it is necessary to consider a restricted model, as Bateni et al. [1] showed that in arbitrary metric spaces, an update time of  $\Omega(k)$  is necessary. We consider metric spaces of bounded doubling dimension. Obviously, time  $\Theta(k)$  is necessary to return an actual solution, so we distinguish between update time for insertion and deletions of points and query time for obtaining a solution from the data structure.
Our result compared to related work We present a deterministic  $(3 + \varepsilon)$ -approximation algorithm for the  $k$ -center problem with  $z$  outliers in bounded doubling dimension that is fully dynamic, i.e., points can be inserted and deleted. Our algorithm is based on a data structure that requires space linear in  $n$  and does not need to know  $k$  and  $z$  in advance. As a result, we can answer queries for all  $k$  and  $z$ , or in other words,  $k$  and  $z$  can be part of the query. Moreover, our algorithm does not require knowledge of the doubling dimension dim of the underlying metric space, and it only appears in the analysis.
Theorem 1.1 (Main theorem). Let  $(M,d)$  be a metric space of bounded doubling dimension dim, and let  $\varepsilon >0$  be an error parameter. There exists a deterministic dynamic algorithm that allows the insertion or deletion of points from  $M$  using worst-case  $\varepsilon^{-O(\mathrm{dim})}\log n\log \Delta$  update time. Moreover, at any time  $t$ , it can be queried by parameters  $k$  and  $z$  to compute  $a(3 + \varepsilon)$ -approximate solution for the  $k$ -center problem with  $z$  outliers of  $P_{t}$  using worst-case  $\varepsilon^{-O(\mathrm{dim})}k\log n\log \log \Delta$  query time, where  $P_{t}$  is the set of points that are inserted but not deleted up to time  $t$ , and  $n$  is the size of  $P_{t}$ .
The first approximation algorithm for this problem has been obtained by Chan, Lattanzi, Sozio, and Wang in 2022 [4], which has an approximation ratio of  $14 + \varepsilon$ . Very recently and in independent work, Pellizzoni, Pietracaprina, and Pucci [18] derived an algorithm with approximation ratio  $3 + \varepsilon$ . For the problem of distributed  $k$ -center with outliers, De Berg, Biabani, and Monemizadeh [7] give a randomized dynamic algorithm that also achieves a guarantee of  $3 + \varepsilon$  with failure probability  $\delta$ . Both of these and also we develop algorithms that answer solution queries rather than membership queries, i.e., they can produce a center set at any time and this set is an approximately good solution for the current  $k$ -center with outliers instance. We state the update times (for insertions and deletions of points) and the query times (for reporting the current approximate solution) of these and our approaches in Table 2.
All three known results have a dependency on  $(k + z)^2$  in their query time which we improve to a linear dependence on  $k$  and no dependence on  $z$ . This gives our approach an advantage even if  $k$  or  $z$  are only mildly dependent on  $n$ : Already for  $k \in O(\log n)$  and  $z \in O(\sqrt{n})$ , the query complexity of [18] would be linear in  $n$  compared to the poly-logarithmic dependency in terms of  $n$  for our query time. More interestingly, for a realistic regime where  $z$  is an epsilon-fraction of  $n$ , the query time of the dynamic algorithm presented in [18] has quadratic dependency in terms of  $n$  while our query time still has a logarithmic dependency in terms of  $n$  and linear dependency in terms of  $k$ . [4] also states a bicriteria approximation where the number of outliers can be violated with improved running time, but the dependency on  $k$  is still quadratic. When comparing to [7], it should be noted that their algorithm is only applicable to the Euclidean metric space, is randomized and only works against oblivious adversaries. Further, the coreset size (and the space complexity) of the algorithm in [7] is  $x = O(k\epsilon^{-\dim}\log \Delta + z)$ . In the streaming model, in order to extract the solution, one needs to run an offline algorithm on the coreset. In this way, the query time of this algorithm is  $O(kx^2\log x) = O(k(z^2 + k^2)\epsilon^{-2dim}\log^2\Delta)$  which is significantly worse than the query time of our algorithm.
The main difference of our approach is that it makes the greedy algorithm for  $k$ -center with outliers dynamic while the previous dynamic algorithms extract a coreset and run the greedy algorithm on this coreset. Known dynamic algorithms [4, 7, 18] for the  $k$ -center problem with outliers maintain a coreset after every update. In particular, in [18], they extract the coreset by simply reading the solution from the cover tree. To extract an approximate solution for this problem, one needs to run a known (offline) greedy algorithm on this coreset. In this way, the query complexities of those algorithms are dominated by the running time of the greedy algorithm. The novelty of our dynamic algorithm is that we make the greedy algorithm itself dynamic. To this end, we use heap data structures to compute a ball that covers the maximum number of points, and dynamic neighborhood sets to obtain points in the expanded maximum ball and update their corresponding keys in the heap to recursively find the next maximum balls.
Navigating nets. The state-of-the-art algorithm for the fully dynamic  $k$ -center problem (without outliers) in doubling metrics, developed by Goranci et al. [10] uses navigating nets originally introduced by Krauthgamer and Lee [15]. The basic idea of a navigating net is to start with the point set  $P_{t}$  as the base net  $N_{1}$  and then compute coarser and coarser variants of it, resulting in a hierarchy of nets  $N_{1}, N_{2}, \ldots, N_{2^{i}}, \ldots$ . For every level in this hierarchy, the points are good representatives of the points in the level below but the higher we get in the net, we have fewer points that are better separated. More precisely, following Krauthgamer and Lee [15], every level is a so-called  $r$ -net of the level below it. An  $r$ -net on a set  $X$  is a subset  $Y \subseteq X$  that satisfies two properties:
1.  $\forall p\in X\exists y\in Y\colon d(p,y) <   r$  (Covering)
2.  $\forall x,y\in Y\colon d(x,y)\geq r$  (Packing)
In a navigating net, one always assures that  $N_{r}$  is an  $r$ -net of  $N_{r/2}$ . Assume that the minimum inter-point distance  $\mathrm{d}_{\min}$  satisfies  $\mathrm{d}_{\min} = 2$  and that the ratio between maximum and minimum inter-point distance is  $\Delta$ . Then a navigating net with  $O(\log \Delta)$  levels can be computed in a greedy fashion. Intriguingly, one can show that in the resulting navigating net, the lowest level with  $|N_{r}| \leq k$  provides an 8-approximation for the  $k$ -center problem (without outliers, i.e.,  $z = 0$ ). This was first observed (in the context of streaming algorithms and there known as the **doubling algorithm**) by Charikar, Chekuri, Feder, and Motwani [5]. An additional trick to improve the approximation ratio was discovered by McCutchen and Khuller [17] (again, in the setting of streaming). They observed that the quality of the solutions depends heavily on what the radius of the lowest level is, and that this radius can be shifted by small amounts, and one of the shifted versions will give a  $2 + \varepsilon$  approximation. By maintaining the navigating net structure and using a similar shifting trick, dynamic  $k$ -center algorithms maintain enough information to answer queries by checking one or a few levels of the resulting navigating net and reporting the points on that level as the solution, as for example in [1, 10].
Challenges with outliers. For the there is no level in the navigating net We know that an optimal  $(k,z)$  -center solution has a radius that lies between the radius of a  $k$  -center solution (because any  $k$  -center solution is feasible for the  $(k,z)$  -center problem) and the radius of a  $(k + z)$  -center solution (because any  $(k,z)$  -center solution is feasible for the  $(k + z)$  -center problem). But as Figure 1 shows, neither of the two choices may provide a good estimate for the optimum  $(k,z)$  -center solution. Say level  $r_1$  is the first to satisfy  $|N_{r_1}|\leq k + z$  and level  $r_2$  is the first to satisfy  $|N_{r_2}|\leq k$  Then the points in  $N_{r_2}$  are a feasible solution for the  $(k,z)$  -center problem, but their radius may be too large, while the radius of the solution in  $N_{r_1}$  is guaranteed to be small enough, but these points may not constitute a feasible solution for  $(k,z)$  -center. This is because every
$(k,z)$  -center problem, the challenge is that which provides a good solution in itself.
#
Figure 1: For  $k = 1$  and  $z = 3$ ,  $k$ -center and  $(k + z)$ -center provide bad estimations for the best  $(k, z)$ -center solution in this example. We see: (a) Dashed Ball: Optimum solution for  $(k, z)$ -center. (b) Small Balls (too small): Optimum solution for  $(k + z)$ -center. (c) Not drawn: the optimum solution for  $k$ -center is a ball around all points (too large).
point in  $N_{r}$  represents several points in the levels below it. So we cannot simply divide the  $k + z$
points into  $k$  centers and  $z$  outliers (see Figure 1). Therefore, there is no individual level which we can query to obtain a solution. Instead, we need to compute a solution from the navigating net levels while making sure not to lose too much in terms of quality and query time.
Pellizzoni et al. [18] base their approach on the so-called "cover tree" data structure, which is similar to that of a navigating net, to compute a coreset from which they construct the final solution. In their approach, they find the lowest level with  $|N_r| \leq 2^{O(\dim)} \varepsilon^{-\dim}(k + z)$  and demonstrate that such  $N_r$  serves as an  $\varepsilon$ -coreset for  $(k, z)$ -center. However, since it is not a feasible solution, they utilize a 3-approximation greedy algorithm to obtain a  $(3 + \varepsilon)$ -approximate clustering, which requires a time complexity of  $\Omega (\varepsilon^{-2\dim}(k + z)^2)$ . In this paper, we strategically integrate additional information at each level to directly emulate the offline 3-approximation greedy algorithm by Charikar et al. [6] in the dynamic setting. This approach enables us to achieve a time complexity that is linearly dependent on  $k$  and remarkably independent of  $z$ .
2 Our data structure
In this section, we describe our data structure, which has  $O(\log \Delta)$  levels. In each level  $r$ , we have net  $N_{r}$ , which is an  $r$ -net of  $N_{r/2}$ . For each point  $p \in N_{r}$ , we keep its weight  $w_{r}(p)$  and local neighborhoods  $B_{p,r}^{1}$  and  $B_{p,r}^{3}$ , which are balls of radius  $(1 + O(\varepsilon))r$  and  $(3 + O(\varepsilon))r$  respectively. Then we maintain max-heap  $H_{r}$  which is designed to store the total weights of neighborhoods  $B_{p,r}^{1}$  for all  $p \in N_{r}$ . We also keep  $\mathrm{rep}_{r}(p)$  for each point  $p \in P$ , which is the representative of  $p$  in  $N_{r}$ . Below, we provide a detailed explanation for elements of our data structure.
Input and set  $R$ . To start building our data structure, we require three input parameters:  $\Delta$ ,  $\varepsilon'$ , and  $\beta$ , where  $0 < \varepsilon' < 1$ , and  $\frac{1}{2} < \beta \leq 1$ .  $\Delta$  denotes the spread ratio of the underlying point set  $P$  and  $\varepsilon'$  is the given error parameter. We utilize  $\beta$  to define  $R := \{0\} \cup \{\beta \cdot 2^{\ell} \mid 0 \leq \ell \leq \lceil \log_2 \Delta \rceil + 1, \ell \in \mathbb{Z}\}$  as a set of levels. Additionally, we define  $\varepsilon := 2^{\lfloor \log \varepsilon' \rfloor}$ . This implies that  $\varepsilon'/2 < \varepsilon \leq \varepsilon'$ . Furthermore, if  $r \in R$ , then either  $\varepsilon r \in R$  or  $\varepsilon r < \beta$ .
Nets  $N_{r}$ . We maintain net  $N_{r}$  for each  $r \in R$ , satisfying the following conditions:  $N_{0} = N_{\beta} = P_{t}$ , where  $P_{t}$  is the set of points at time  $t$ , and for  $r > \beta$ ,  $N_{r}$  is an  $r$ -net of  $N_{r/2}$ . For  $p \in N_{r}$  and  $q \in N_{r/2}$ , we say  $p$  is a parent of  $q$  (and  $q$  is a child of  $p$ ) if  $d(p, q) \leq r$ . In our data structure, we store all the parents and children for each point for all the nets. It is important to note that each point can have at most  $2^{O(\dim)}$  parents or children. In the full version, we explain how to handle each insert/delete in time  $2^{O(\dim)} \log \Delta$ .
During the paper, we may refer to  $N_{\varepsilon r}$  for  $0 < \varepsilon r < \beta$ , which means  $\varepsilon r \notin R$ . Besides, note that since  $\mathrm{d}_{\min} = 2$  and  $\beta < 2$ ,  $P_t$  is an  $r$ -net for any  $0 \leq r \leq \beta$ . To solve this issue, we define  $N_r := P_t$  for any  $0 < r < \beta$ , however, we do not explicitly keep any  $r$ -nets for  $0 < r < \beta$  in our data structure to prevent redundancy.
Representatives  $\mathbf{rep}_r$  and weights  $w_r$ . For all  $r \in R$ , we inductively define functions  $\mathrm{rep}_r \colon P_t \to N_r$  such that each point  $p \in P_t$  has a unique representative  $\mathrm{rep}_r(p)$  in net  $N_r$ . To define  $\mathrm{rep}_r(p)$ , we consider three cases. First, if  $p \in N_r$ , then we define  $\mathrm{rep}_r(p) \coloneqq p$ . Second, if  $p \notin N_r$  but  $p \in N_{r/2}$ , we uniquely choose one of the parents of  $p$  in  $N_r$  as  $\mathrm{rep}_r(p)$ . Third, if  $p \notin N_{r/2}$  and therefore  $p \notin N_r$ , we define  $\mathrm{rep}_r(p) = \mathrm{rep}_r(\mathrm{rep}_{r/2}(p))$ . Again to avoid inconsistency, we define  $\mathrm{rep}_r(p) \coloneqq p$  for any  $0 < r < \beta$ , but we do not store this explicitly. We show in Lemma 2.1 that the distance between each point  $p$  and its representative in  $N_r$  is at most  $2r$ .
Next, we define a weight function  $w_{r}$  for every level. For every point  $p \in N_r$ , its weight  $w_{r}(p)$  denotes the number of points that  $p$  is representing in  $N_{r}$ . More formally,  $w_{r}(p) = \left|\{q \in P_{t} \mid \mathrm{rep}_{r}(q) = p\}\right|$ . For a set  $X \subseteq N_{r}$ , we define  $w_{r}(X) := \sum_{p \in X} w_{r}(p)$ . Equivalently,  $w_{r}(X) = \left|\{y \in P_{t} \mid \mathrm{rep}_{r}(y) \in X\}\right|$ . Note that the sum of weights of all representatives at any level is equal to the total number of points, so  $w_{r}(N_{r}) = |P_{t}|$  for all  $r \in R$ .
Since we store all the parents and children, we can easily update the representatives and therefore the weights after each insert/delete. The details are presented in the full version.
Balls  $B^{1}$  and  $B^{3}$ . Let  $r \in R$ . For every point  $p \in N_{\varepsilon r}$ , we maintain the close neighborhood  $B_{p,r}^{1} \coloneqq \operatorname{Ball}(p, (1 + 4\varepsilon)r) \cap N_{\varepsilon r}$  and the extended neighborhood  $B_{p,r}^{3} \coloneqq \operatorname{Ball}(p, (3 + 8\varepsilon)r) \cap N_{\varepsilon r}$ , where  $\operatorname{Ball}(p,\lambda) = \{q \in M \mid d(p,q) \leq \lambda\}$  denotes the ball of radius  $\lambda$  around  $p$  in the metric space  $M$ . Then  $B^{1} \coloneqq \{B_{p,r}^{1} \mid r \in R, p \in N_{\varepsilon r}\}$  and  $B^{3} \coloneqq \{B_{p,r}^{3} \mid r \in R, p \in N_{\varepsilon r}\}$  are the collections of local neighborhoods at different scales. As mentioned in Lemma 2.2, the size of each of these neighborhoods is at most  $2^{O(\dim)}/\varepsilon^{\dim}$ . We also show that we can update these neighborhoods in time  $2^{O(\dim)}\varepsilon^{-2\dim} \log \Delta$  after each insert/delete in the proof of Lemma 2.3.