SlowGuess/NeurIPS_2023_OCR huggingface dataset:
First, in linear  $(\tilde{O}(nd))$  time, the method of [22] computes an  $O(1)$ -approximate coreset  $C$  of size  $\mathrm{poly}(k,\log n)\cdot d$ . We can then project the data onto  $O(\log k)$  dimensions, with a linear map  $\Pi$ . In low dimensions, we can compute a smaller coreset  $C'$  of size  $\mathrm{poly}(k,\log n)$  of  $\Pi C$  in linear time, and then solve  $k$ -means on  $C'$  in time  $\mathrm{poly}(k,\log n)$ . This also implies an  $O(1)$ -approximation for  $\Pi C$ . Next, we can map every point in  $\Pi C$  to its closest center in  $d' = O(\log k)$  dimensions, to form an explicit clustering. This takes time  $O(|C|\cdot k\cdot d') = \mathrm{poly}(k,\log n)\cdot d$  time. By [57], every  $k$  clustering has its  $k$ -means objective preserved by a  $\Theta(1)$ -approximate when projected by  $\Pi$ , which means the same clustering should still be an  $O(1)$ -approximation in the original space. We can compute the mean of each cluster in linear time, so in the original  $d$ -dimensional space, we can find an  $O(1)$ -approximate  $k$ -means clustering in time  $\tilde{O}(nd) + \mathrm{poly}(k)\cdot d$ , as desired. Finally, in the  $k$ -median case, [57] is still applicable, and we can compute an approximate 1-median of each clustering in near-linear time as well [23].
Parallel computation. In the following, we briefly discuss how to implement our algorithm in the massively parallel computation (MPC) model [51, 11] when each machine has  $(kd\log(n)\cdot 1 / \varepsilon \cdot \log 1 / \delta \cdot \log(\Lambda / \rho))^C$  memory for a sufficiently large constant  $C > 0$ . Before we state our implementation, let us briefly describe the MPC model. In the MPC model, there are  $M$  machines where each machine has  $H$  local memory where  $H$  is sublinear in the input size and  $H = M^\gamma$  for an arbitrary constant  $\gamma > 0$ . At the beginning of the computation, the input is arbitrarily distributed in the machines. The computation proceeds in rounds. In each round, each machine performs some local computation. Then at the end of the round, each machine sends/recieves messages to/from other machines. However, the messages sent/received by a machine in a round cannot exceed its local memory  $H$ . At the end of the algorithm, the output should store in machines distributedly. The goal is to design an algorithm with small number of rounds. In the following, we show how to implement our algorithm in the MPC model using  $O(1)$  rounds.
Consider Algorithm 2. Computation of  $\{\tilde{x}_1, \tilde{x}_2, \dots, \tilde{x}_n\}$  only requires local computations. Then, we can run  $REP$  repetitions and each level  $\ell \in [0, L_2]$  of the loop in Algorithm 2 in parallel. For each instance, it only requires the counting and taking maximum which can be easily done in the MPC model in  $O(1)$  rounds [45]. Since a single machine has large enough local memory, we are able to send the entire  $F$  to a single machine. The total space required here is  $O(REP \cdot L_2 \cdot n \cdot d)$ . Next, let
us consider the implementation of Algorithm 1. Since each machine has enough local memory, we are able to make each machine holds a copy of  $F$ . This broadcasting process can be done in  $O(1)$  rounds (see e.g., [5]). Once each machine holds  $F$ , only local computation is required to determine a point  $\tilde{x}_i$  whether it should be in  $\tilde{X}_0$ . For points that are not belong to  $\tilde{X}_0$ , we are able to determine whether it belongs to  $\hat{X}_j$  by only local computations. For each  $\hat{X}_j$ , we run the MPC DP coreset algorithm of [25] to get an semi-coreset. This step also takes  $O(1)$  rounds. Finally, we can run any non-private MPC  $k$ -means algorithm (e.g., [37]) on  $\tilde{X}_0 \cup \tilde{X}_1 \cdots \cup \tilde{X}_{|F|}$  which takes  $O(1)$  rounds.
D A simple lower bound for dist-DP clustering
In this section, we prove the following simple proposition, showing a additive dependence on  $k \cdot \rho$  (resp.,  $k \cdot \rho^2$ ) is necessary for  $k$ -median (resp.,  $k$ -means), as long as the dimension is  $d = \Omega(\log k)$ .
Proposition D.1. Let  $X_0 = \{x_1, \ldots, x_{2k}\}$  be points in the ball of radius  $\rho$  around the origin, separated by at least  $\frac{\rho}{10}$  (for  $d = \Omega(\log k)$ , this is doable). Suppose  $X \subset X_0$  is a random subset of size  $k$ , and there exists an  $(\varepsilon, \delta, \rho)$ -dist-  $DP$  algorithm that outputs  $k$  centers  $C$  in terms of  $X$ , where  $\varepsilon, \delta \leq 0.1$ . Then, the expected cost  $\mathbb{E}[\mathrm{cost}(X; C)]$  is  $\Omega(k \cdot \rho^p)$ , where  $p = 1$  for  $k$ -median and  $p = 2$  for  $k$ -means. Yet, the optimum cost  $\mathrm{OPT}_k(X)$  is 0.
Hence, any  $(\varepsilon, \delta, \rho)$ -dist-  $DP$  algorithm with finite multiplicative ratio must incur additive error  $k \cdot \rho^p$ .
Proof. First, note that  $\mathrm{OPT}_k(X) = 0$  since  $|X| = k$ , so for  $C^* = X$ ,  $\mathrm{cost}(X,C^*) = 0$ . We now show that  $\mathbb{E}[\mathrm{cost}(X;C)] = \Omega (k\cdot \rho^p)$ .
For each  $i \leq 2k$ , let  $B_i$  be the ball of radius  $\frac{\rho}{100}$  around  $x_i$ . Let  $p_i$  be the probability over  $X$  and the randomness of the private algorithm that some point in  $C$  is in  $B_i$ . Let  $p_i^+$  be the probability of the same event conditioned on  $x_i \in X$ , and  $p_i^-$  be the same probability conditioned on  $x_i \notin X$ .
First, note that  $x_{i} \in X$  with probability  $1/2$ , since  $|X| = \frac{1}{2} \cdot |X_{0}|$ . So,  $p_{i} = \frac{1}{2} (p_{i}^{-} + p_{i}^{+})$ . Next, there exists a simple coupling between the events of  $x_{i} \in X$  and  $x_{i} \notin X$ , that changes at most 1 point. Namely, if  $X$  contains  $x_{i}$ , add in a random point in  $X_{0} \backslash X$ , and then remove  $x_{i}$ , to get a new set  $X'$ . If the distribution of  $X$  is uniform conditioned on  $x_{i} \in X$ , it is simple to see that the distribution of  $X'$  is uniform conditioned on  $x_{i} \notin X$ . Therefore,  $\mathbb{P}(C(X) \in B_{i}) = p_{i}^{+}$  and  $\mathbb{P}(C(X') \in B_{i}) = p_{i}^{-}$ .
Because we only changed one element  $x_{i}$  and moved it a distance at most  $\rho$ , this means that  $\mathbb{P}(C(X)\in B_i) = e^{\pm \varepsilon}\cdot \mathbb{P}(C(X')\in B_i)\pm \delta$ , or equivalently,  $p_i^+ = e^{\pm \varepsilon}\cdot p_i^- \pm \delta$ . Since  $\delta \leq \varepsilon \leq 0.1$ , this means  $|p_i^+ -p_i^- |\leq 0.3$ . Also, since  $p_i = \frac{1}{2} (p_i^- + p_i^+)$ , this means  $p_i^+ -p_i\leq 0.15$ .
Now, since the points in  $X_0$  are separated by  $\frac{\rho}{10}$ , the balls  $B_i$  are disjoint. So, for any fixed  $C$ , at most  $k$  of the events of some point in  $C$  is in  $B_i$  can hold. Therefore,  $\sum_{i=1}^{2k} p_i \leq k$ , which means  $\sum_{i=1}^{2k} p_i^+ \leq k + 0.15 \cdot 2k = 1.3k$ .
Now,  $\mathrm{cost}(X,C)$  is at least  $\sum_{i = 1}^{2k}\left(\frac{\rho}{10}\right)^p\cdot \mathbb{P}(x_i\in X)\cdot (1 - p_i^+)$ . This is because  $\mathbb{P}(x_i\in X)\cdot (1 - p_i^+)$  represents the probability that  $x_{i}\in X$  but no point in  $C$  is within  $\frac{\rho}{10}$ , so the point  $x_{i}$  itself contributes  $\left(\frac{\rho}{10}\right)^p$  to the cost. But this simply equals  $\left(\frac{\rho}{10}\right)^p\cdot (2k - \sum_{i = 1}^{2k}p_i^+) \geq \Omega (\rho^p\cdot k)$ , as desired.
E Additional Details of Experiments
E.1 Details of Implementations
More Implementation Details. Note that the privacy budget is consumed in 3 parts: (1) computing  $\bar{X}$ , (2) computing  $\text{count}(g)$  for cells  $g$  at level  $l$  for  $l \in [L_1 + 1, L_2]$ , and (3) computing DP semicoreset  $\tilde{X}_j$  in Algorithm 1. We split the privacy budget uniformly, i.e., each part takes  $\varepsilon/3$  and  $\delta/3$ .
Detailed implementation of Algorithm 2. Since we know each  $x$  is in  $[-1,1]^d$ , When we compute  $\tilde{x}_i$ , if any coordinate is outside  $[-2,2]$ , we project it to  $[-2,2]$ . We choose  $REP = 5$ . The random shifted vector is chosen uniformly random from  $[0,4]^d$  and thus the cell in the highest level of the
quadtree has side length 8. We choose  $L_{1} = 5$  and  $L_{2} = 10$ . When we compute count(g) of cell  $g$  at level  $l \in [L_1 + 1, L2]$ , we apply Gaussian thresholding mechanism<sup>5</sup>.
Detailed implementation of Algorithm 1. We set  $S = \sqrt{2\log(1.25) / (\delta / 6)} \cdot \sqrt{d} / (\varepsilon / 6)$ . We run the first loop of Algorithm 1 to obtain  $\hat{X}_0$ . We slightly modify the second loop as follows:  $\hat{X}_j = \{x_i \in X \setminus \tilde{X}_0 \mid d(x_i, f_j) = d(x_i, F)\}$ . We use Gaussian thresholding mechanism to compute  $\hat{n}_j$  to estimate  $|\hat{X}_j|$ . If  $\hat{n}_j \leq 0$ , we drop  $\hat{X}_j$ . Otherwise we run a semi-coreset for  $\hat{X}_j$ . It is easy to show that the above modified procedure is still DP. When we use the DP open-source library to compute the (semi)-coreset of  $\hat{X}_j$ , we specify the bounding ball is centered at  $f_j$  with radius  $\min(S, \sqrt{d}) \cdot \rho$ , i.e., the points in  $\hat{X}_j$  that are outside the ball are projected to the ball.
Finally, we use non-DP baseline  $k$ -means to run  $k$ -means over the union of (semi)-coresets  $\tilde{X}_1 \cup \dots \cup \tilde{X}_{\alpha \cdot k}$  and  $\tilde{X}_0$ .
E.2 Preprocessing Steps of the Datasets
Dataset gowalla contains 6,442,890 user check-ins of 107,092 different users and dataset brightkite contains 4,491,143 user check-ins of 51,406 different users. Each check-in record contains a location information (latitude and longitude). For each user, we use its latest check-in record, and thus we obtain a dataset of size  $107,092 \times 2$  for gowalla and a dataset of size  $51,406 \times 2$  for brightkite. For each latitude, we divided it by 90. For each longitude, we divided it by 180. Thus, each coordinate of a user is in [-1,1].
For other non-geographic datasets (shuttle, skin, rangequeries, s-sets), we follow the same preprocessing steps of experiments in [24]. In particular, we linearly rescale each dimension of each point to make the coordinate have value in [-1,1].
Our work developed distance based private algorithms for clustering problems. Distance based privacy provides provable standards of privacy but its use, like that of any privacy protection, is subject to limitations (we refer to standard textbooks on differential privacy such as [32] for the subject). We also stress that privacy is only one of the requirements of a responsible machine learning system. For this reason, we encourage anyone using the techniques developed in this paper in a real system, to review carefully the overall safety of their design.
<https://github.com/google/differential-privacy/blob/main/common_docs/Delta_For_Thresholding.pdf.
<sup>6</sup>https://ai.googleblog.com/2021/10/practical-differentially-private.html.