SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$k$ -Means Clustering with Distance-Based Privacy
Alessandro Epasto
aepasto@google.com
shyamsn@mit.edu
Peilin Zhong
peilinz@google.com
In this paper, we initiate the study of Euclidean clustering with Distance-based privacy. Distance-based privacy is motivated by the fact that it is often only needed to protect the privacy of exact, rather than approximate, locations. We provide constant-approximate algorithms for  $k$ -means and  $k$ -median clustering, with additive error depending only on the attacker's precision bound  $\rho$ , rather than the radius  $\Lambda$  of the space. In addition, we empirically demonstrate that our algorithm performs significantly better than previous differentially private clustering algorithms, as well as naive distance-based private clustering baselines.
Two of the most fundamental and widely studied problems in unsupervised machine learning are the  $k$ -means and  $k$ -median clustering problems. Solving these clustering problems can allow us to group together data efficiently, and hence extract valuable and concise information from massive datasets. The goal of the  $k$ -means (resp.,  $k$ -median) clustering problem is: given a dataset  $X$  of points, construct a set  $C$  of  $k$  centers to minimize the clustering cost  $\sum_{x \in X} d(x, C)^2$  (resp.,  $\sum_{x \in X} d(x, C)$ ), where  $d(x, C)$  represents the minimum distance between the data point  $x$  and the closest center in  $C$ .
In general, machine learning and data mining algorithms are prone to leaking sensitive information about individuals who contribute data points. In certain scenarios, this can lead to severe consequences, including losses of billions of dollars [60] or even the loss of human lives [10]. Thus, providing accurate algorithms that protect data privacy has become crucial in algorithm design. Over the past decade, the notion of differential privacy (DP) [31] has emerged as the gold standard for privacy-preserving algorithms, both in theory and in practice, and has been implemented by several major companies and the US Census [34, 68, 30, 1]. Informally, DP requires the output distribution of the algorithm to remain almost identical whenever a single data point is altered. (See Section 2 for a formal definition.) Hence, even the knowledge of all but one data point, along with the output of the algorithm, still cannot reveal significant information about the final data point.
The importance of  $k$ -means and  $k$ -median clustering, as well as preserving data privacy, has led to a large interest in designing differentially private clustering algorithms in Euclidean space [14, 62, 36, 47, 59, 73, 64, 65, 71, 38, 9, 63, 48, 70, 69, 44, 49, 17, 61, 19, 13, 24, 33, 25, 56]. Here, the goal is to design a differentially private set of  $k$  centers, such that the clustering cost with respect to these centers is only a small factor larger than the optimal (non-private) clustering cost. Importantly, the work of [70, 44, 25] led to efficient polynomial-time and differentially private algorithms that achieve constant multiplicative approximation ratios.
While we can obtain DP algorithms with low multiplicative error, all such algorithms also require an additional additive error. If  $\Lambda$  is the radius of a ball that is promised to contain all data points, even the
best private clustering algorithms are known to have an additive error proportional to  $\mathrm{poly}(k,d)\cdot \Lambda^p$ , where  $p = 2$  for  $k$ -means and  $p = 1$  for  $k$ -median. This factor of  $\Lambda^p$  is in fact unavoidable [47], as a single individual datapoint can be moved up to distance  $\Lambda$  and the algorithm must preserve privacy with respect to this change. If we do not have a good bound of  $\Lambda$ , this factor may dominate the error, and may make the clustering algorithm highly inaccurate. Even if the bound is known exactly, errors scaling with  $\Lambda$  may however be unnecessary and unacceptable in certain situations.
The additive error depending on  $\Lambda$  is necessary because standard differential privacy requires us to protect learning anything about the location of any point. However, in practice this may not be necessary as it might be enough to not know the location of a point up to a certain error. For instance, in address data, the risk is leaking the actual location, but uncertainty within a few miles in a city is sufficient to protect the privacy of the person [20]. Another motivation is in smart meters [20, Section 6.1], where accurately learning the fine-grained consumption can result in spectacular privacy leaks (e.g. learning which TV channel is being watched [46, 52]) but slight uncertainty on the measurements is sufficient to protect from such attacks. Moreover, when differential privacy is used to protect the algorithm from adversarial inputs, it is often sufficient to protect against small perturbations as large perturbations can be detected or removed otherwise [53].
These cases can be modeled by variants of differential privacy, such as dX privacy (a.k.a. extended differential privacy) [20, 40], and pixelDP [53]. All such models are adaptations or generalizations of DP which take into account a metric over the datasets.
In this paper, we study a concrete formulation of distance-based privacy which we call  $\rho$ -dist-DP. Roughly speaking, an algorithm is  $\rho$ -dist-DP if the algorithm protects privacy of a single data point if it is moved by at most  $\rho$  in a metric space. (See Section 2 for a formal definition, where we define  $(\varepsilon, \delta, \rho)$ -dist-DP.) This is a less restrictive version of DP, as usually the neighboring datasets are defined to be any two datasets with a single point allowed to move anywhere. While we remark that although this notion is well-defined for any metric space, our results in this paper focus entirely on Euclidean space.
The main question we study in this paper is the following: can we obtain much better approximation results (and algorithms better in practice) if we allow the algorithm to resist small movements, as opposed to arbitrary movements, of a point for instance for clustering? In other words, can we design  $\rho$ -dist-DP algorithms that perform significantly better than the state of the art regular DP algorithms for  $k$ -means or  $k$ -median clustering?
In this work, we answer the above question affirmatively, by providing an efficient and accurate theoretical algorithm, and showing empirically that our algorithm outperforms clustering algorithms with standard differential privacy.
1.1.1 Theoretical Results
From a theoretical perspective, we are able to obtain  $O(1)$ -approximate algorithms for  $k$ -means and  $k$ -median clustering with  $\rho$ -dist-DP, and with additive error essentially only depending on the smaller distance  $\rho$  as opposed to the full radius  $\Lambda$ . More precisely, our main theorem is the following.
Theorem 1.1. Let  $n, k, d$  be integers,  $\rho \in (0, \Lambda], \varepsilon, \delta \in (0, 1]$  be privacy parameters, and  $p \in \{1, 2\}$ . Then, given a dataset  $X = \{x_1, \ldots, x_n\}$  of points in a given ball of radius  $\Lambda$  in Euclidean space  $\mathbb{R}^d$ , there exists a polynomial-time  $(\varepsilon, \delta, \rho)$ -dist-  $DP$  algorithm  $\mathcal{A}$  that outputs a set of centers  $C = \{c_1, \ldots, c_k\}$ , such that
$$
\sum_{i = 1}^{n}d(x_{i},C)^{p}\leq O(1)\cdot \min_{\substack{C^{*}\subset \mathbb{R}^{d}\\ |C^{*}| = k}}\sum_{i = 1}^{n}d(x_{i},C^{*})^{p} + \text{poly}\left(k,d,\log n,\frac{1}{\varepsilon},\log \frac{1}{\delta},\log \frac{\Lambda}{\rho}\right)\cdot \rho^{p}.
$$
Here,  $p = 1$  for  $k$ -median and  $p = 2$  for  $k$ -means.
For more precise dependences on the parameters  $k, d, 1 / \varepsilon$ , please see Theorem C.1.
Qualitatively, Theorem 1.1 has similar guarantees to [70], who also provided an  $(\varepsilon, \delta)$ -differentially private algorithm with an  $O(1)$ -approximation algorithm and additive error that was
poly  $(k,d,\log n,\frac{1}{\varepsilon},\log \frac{1}{\delta})\cdot \Lambda^p$  . The main difference is that we drastically reduce the additive error by reducing the dependence on  $\Lambda$  to a dependence on the distance privacy parameter  $\rho$
Running time and parallel computation. The runtime of a straightforward implementation of our algorithm is  $\tilde{O}(nkd) + \mathrm{poly}(k) \cdot d$ , if we also ignore polynomial factors in  $\log \frac{\Lambda}{\rho}$ . By using approximate near neighbor algorithms, we can improve this further to  $\tilde{O}(nd) + \mathrm{poly}(k) \cdot d$ , which for  $k$  at most a small polynomial in  $n$ , is nearly linear. In addition, the algorithm can be easily implemented in the massively parallel computation (MPC) model [51, 11] (an abstraction of MapReduce [28]) using  $O(1)$  rounds and near linear total space where each machine has sublinear space. We discuss above in-memory algorithms and MPC algorithms further at the end of Appendix C.
Finally we remark that the  $\rho^p$  dependence in the additive error is required for ensuring  $\rho$ -dist-DP. In fact, we prove in Appendix D that any  $(\varepsilon, \delta, \rho)$ -dist-DP algorithm, with any finite multiplicative error, must incur  $\Omega(k \cdot \rho^2)$ -additive error for  $k$ -means and  $\Omega(k \cdot \rho)$ -additive error for  $k$ -median.
1.1.2 Empirical Results
We empirically studied the performance of our algorithm on public and real-world datasets. We compare the approximation guarantee of our algorithm with the standard DP clustering algorithm and the standard non-private  $k$ -clustering algorithm. Experiments show that our algorithm outperforms the DP clustering algorithm and is only slightly worse than the non-private algorithm. In addition, we show that smaller  $\rho$  provides a better approximation guarantee, which aligns with our theoretical study. We refer readers for more details of our empirical study to Section 6.
1.2 Other Related Work
Distance-based Privacy: The literature on distance-based privacy explored different data protection schemes which we now describe in more detail. A general notion is known as dX privacy [20] (a.k.a. Extended differential privacy) which includes as a special case differential privacy. This privacy notion bounds the distinguishability of two statistical datasets, not just by the number of different users' inputs (i.e., their Hamming distance) but by an arbitrary  $d_{\chi}$  distance between them accounting for the magnitude of the changes to each user entry. Similar notions, such as pixelDP [53] and perceptual indistinguishability [21], are also formalization of DP where adjacent datasets differ in a single feature of the input (e.g. a pixel) or some custom function of the data. Several algorithms have been defined for these notions, including LSH algorithms [40].
From an application point of view, much work has focused on geo-indistinguishability [3, 6, 15], i.e. preventing an adversary from distinguishing two close locations (by ensuring that close location have similar probabilities to generate a certain output). Other areas of applicability has been protecting textual data [39, 41], private smart meters sensing [27], image obfuscation [35, 21] and mobile crowsensing [74].
$k$ -Clustering:  $k$ -Means and  $k$ -median clustering have seen a large body of work over the past few decades. While both problems are known to be NP-hard [58], a significant amount of work has given various  $O(1)$ -approximation algorithms for both problems [18, 8, 50, 7, 55, 16, 2, 26]. The state-of-the-art approximation is a 5.912-approximation for Euclidean  $k$ -means and a 2.406-approximation for Euclidean  $k$ -median [26]. As noted in previously, there has also been significant work in specifically studying differentially private  $k$ -means and  $k$ -median clustering, though to our knowledge we are the first to study distance-based private clustering.
We present some basic definitions and setup that will be sufficient for explaining our algorithms for the main body of the paper. We defer some additional preliminaries to Appendix A.
$^1\tilde{O} (f(n))$  denotes  $O(f(n)\log f(n))$
First, we recall the definition of differential privacy.
Definition 2.1. [31] A (randomized) algorithm  $\mathcal{A}$  is said to be  $(\varepsilon, \delta)$ -differentially private  $((\varepsilon, \delta)$ -DP for short) if for any two datasets  $X$  and  $X'$  that differ in exactly one data point and any subset  $S$  of the output space of  $\mathcal{A}$ , we have
$$
\mathbb {P} (\mathcal {A} (X) \in S) \leq e ^ {\varepsilon} \cdot \mathbb {P} (\mathcal {A} (X ^ {\prime}) \in S) + \delta .
$$
In standard differential privacy, two datasets  $X$  and  $X'$  are adjacent if we can convert  $X$  to  $X'$  either by adding, removing, or changing a single data point. Notably, the change in the single data point may be arbitrary.
In distance-based privacy, however, we only allow two datasets to be adjacent if they differ by changing (not adding or removing) a single data point, by moving it up to distance  $\rho$ . Formally, we define the following.
Definition 2.2. Let  $X, X'$  be  $\rho$ -adjacent if they have the same number of points and differ in exactly one data point, where the distance between the two differing data points is  $\rho$ . Then, a (randomized) algorithm  $\mathcal{A}$  is  $(\varepsilon, \delta, \rho)$ -dist-DP if for any two  $\rho$ -adjacent datasets  $X$  and  $X'$  and any subset  $S$  of the output space of  $\mathcal{A}$ , we have
We remark that in all of our theoretical guarantees, we implicitly assume that  $\varepsilon, \delta \leq \frac{1}{2}$ .
The Laplace Mechanism is one of the most common primitives used to ensure privacy. Simply put, for a non-private statistic, the Laplace Mechanism adds noise  $\mathrm{Lap}(t)$  to the statistic for some  $t > 0$  where  $\mathrm{Lap}(t)$  has the probability density function (PDF) equal to  $\frac{1}{2t} \cdot e^{-|x| / t}$ . It is well-known that if  $f(X)$  is a statistic such that  $|f(X) - f(X')| \leq \Delta$  for any two adjacent datasets  $X, X'$ , then  $f(X) + \mathrm{Lap}(\Delta / \varepsilon)$  is  $(\varepsilon, 0)$ -DP. Likewise, if  $|f(X) - f(X')| \leq \Delta$  between two  $\rho$ -adjacent datasets  $X, X'$ , then  $f(X) + \mathrm{Lap}(\Delta / \varepsilon)$  is  $(\varepsilon, 0, \rho)$ -dist-DP.
Similar to the Laplace Mechanism, we can also implement the Truncated Laplace mechanism [43] for approximating functions  $f: X \to \mathbb{R}$ . The Truncated Laplace Mechanism outputs  $f(X) + \mathrm{T}\mathrm{Lap}(\Delta, \varepsilon, \delta)$ , where  $\mathrm{T}\mathrm{Lap}(\Delta, \varepsilon, \delta)$  is the distribution with PDF proportional to  $e^{-|x| \cdot \varepsilon / \Delta}$  on the region  $[-A, A]$ , where  $A = \frac{\Delta}{\varepsilon} \cdot \log \left(1 + \frac{e^{\varepsilon} - 1}{2\delta}\right)$ , and PDF 0 outside the region  $[-A, A]$ . Assuming  $0 < \varepsilon$  and  $0 < \delta \leq \frac{1}{2}$ , it is known that if  $|f(X) - f(X')| \leq \Delta$  for all adjacent  $X, X'$ , then this mechanism is  $(\varepsilon, \delta)$ -DP, and if  $\varepsilon \leq \frac{1}{2}$  this is accurate up to error  $\frac{\Delta}{\varepsilon} \cdot \log \frac{1}{\delta}$ , with probability 1.
Likewise, a nearly identical result holds for distance-based privacy. Namely, if  $|f(X) - f(X')| \leq \Delta$  for any  $\rho$ -adjacent datasets  $X, X'$ , then  $f(X) + \mathrm{TLap}(\Delta, \varepsilon, \delta)$  is  $(\varepsilon, \delta, \rho)$ -dist-DP.
We defer some additional preliminaries to Appendix A.
2.2  $k$ -Means and  $k$ -Median Clustering
We define  $d(x,y)$  to be the Euclidean distance between two points  $x$  and  $y$ , and for a finite subset  $C \subset \mathbb{R}^d$ , we define  $d(x,C) = d(C,x)$  to be  $\min_{c \in C} d(x,c)$ . Given a dataset  $X = \{x_1, \ldots, x_n\}$  of points in  $\mathbb{R}^d$ , and a set of centers  $C = \{c_1, \ldots, c_k\}$ , we define the  $k$ -means/  $k$ -median cost as
$$
\operatorname {c o s t} (X; C) := \sum_ {x \in X} d (x, C) ^ {p}.
$$
Above,  $p = 2$  for  $k$ -means and  $p = 1$  for  $k$ -median. Finally, we define  $\mathrm{OPT}_k(X)$  to be the minimum value of  $\mathrm{cost}(X; C)$  for any set of  $k$  points  $C$ .
We further assume that the points in  $X$  are in  $B(0,\Lambda)$ , which is the ball of radius  $\Lambda$  about the origin in  $\mathbb{R}^d$ . Our goal in  $k$ -means (resp.,  $k$ -median) clustering is to find a subset  $C$  of  $k$ -points that minimizes  $\mathrm{cost}(X;C)$ , i.e., where  $\mathrm{cost}(X;C)$  is as close to  $\mathrm{OPT}_k(X)$  as possible. Occasionally, we may assign each point  $x_i \in X$  a positive weight  $w_i$ , in which case we define  $\mathrm{cost}(X;C) := \sum_{x_i \in X} w_i \cdot d(x_i, C)^p$ .
Our goal in differentially private clustering is to produce a set of  $k$  centers  $C$  such that  $C$  is  $(\varepsilon, \delta)$ -DP with respect to  $X$ , and such that  $\mathrm{cost}(X; C) \leq \beta \cdot \mathrm{OPT}(X) + V \cdot \Lambda^p$  (where  $p = 2$  for  $k$ -means and  $p = 1$  for  $k$ -median), where  $\beta$  and  $V$  are not too large. In distance-based privacy, we wish to replace the factor  $\Lambda$  with some smaller  $\rho$ , i.e., we want  $\mathrm{cost}(X; C) \leq \beta \cdot \mathrm{OPT}(X) + V \cdot \rho^p$ . However, our algorithm only has to be private up to changing a single data point by up to  $\rho$ . If we obtain this guarantee, we say that we have a  $(\beta, V)$ -approximate and  $(\varepsilon, \delta, \rho)$ -dist-DP solution.
3 Technical Overview and Roadmap
We focus on proving Theorem 1.1 in Sections 4 and 5, and discuss our experimental results in Section 6. In Sections 4 and 5, we will only describe the algorithms, and we defer all formal proofs to the Supplementary sections. For simplicity, in this overview we focus on  $k$ -median and assume the dimension  $d = (\log n)^{O(1)}$ , and can be hidden in  $\tilde{O}$  notation.
Our approach follows two high-level steps, inspired by the work of [22, 25]. The insight used in [25], which proved highly efficient private clustering algorithms, is to start by generating a crude but private solution that may use a large number of centers and have a large approximation, but has small additive error. Then, one can apply the crude solution to partition the Euclidean space  $\mathbb{R}^d$  into smaller regions, and apply some regular differentially private clustering algorithm in the regions. We follow a similar high-level template to [25]. However, we still need to implement each of these steps, which require several technical insights to ensure we maintain privacy while only losing additive error roughly proportional to  $\mathrm{poly}(k,d)\cdot \rho$ .
To obtain a crude approximation, we use a technique based on partitioning the space  $\mathbb{R}^d$  into randomly shifted grids at various levels (also known as the Quadtree). In the Quadtree, the 0th level is a very coarse grid containing the large ball of radius  $\Lambda$ , and each subsequent level refines the previous level with smaller grid cells. For a single grid and knowledge of which point lies in which grid cell, a natural approach for minimizing cost would be to output the centers of the "heaviest" cells, i.e., those with the most number of points. Indeed, it is known that outputting the  $O(k)$  heaviest cells at each grid level provides a good approximation, at the cost of having more than  $k$  centers.
While this is not DP, a natural way of ensuring privacy would be to add Laplace noise to each count and add the heaviest cells after this. Unfortunately, doing so will lead to error depending on the full radius  $\Lambda$ , due to the coarser levels of the quadtree (i.e., levels with grid length close to  $\Lambda$  rather than  $\rho$ ). For example, if there was only a single data point, there will be at least  $e^d$  cells even at coarse levels, and several of them may have large noisy counts. Hence, we are likely to choose completely random cells, which will cause additive error to behave like  $\Lambda$  as opposed to  $\rho$ . Another option is to add noise to the points first and then compute the heaviest cells. While this avoids additive dependence on  $\Lambda$ , the additive dependence will behave like  $n \cdot \rho$  where  $n$  is the full size of the dataset.
Surprisingly, we show that we can combine both of these observations in the right way. Namely, for coarse cells (i.e., with length larger than  $\tilde{O}(\rho)$ ), we add noise (of distance proportional to  $\tilde{O}(\rho)$ ) to the data points directly to generate private points  $\tilde{x}_i$ , and then compute the heaviest cells without adding noise to the counts. For fine cells (length smaller than  $\tilde{O}(\rho)$ ), we do not add noise to the data points, but we add Laplace noise to the cell counts.
To explain the intuition behind this, suppose that the  $n$  data points happen to be perfectly divided into  $n / k$  clusters, where every point has distance  $r$  to its nearest cluster center. If  $r \gg \rho$ , then even if we add  $\tilde{O}(\rho)$  noise to each data point, we will still find cluster centers that are within  $\tilde{O}(r)$  of each correct center. So, the  $k$ -means cost should only blow up by a small multiplicative factor, without additive error. Alternatively, if  $r \ll \rho$ , then the grid cells of side length  $\tilde{O}(r)$  should contain the entire cluster, and hence have  $n / k$  points in them. Assuming  $n \gg d \cdot k$ , even if we add Laplace noise to each of  $e^d$  cells, none of them will exceed  $n / k$ . Alternatively, if  $n \ll d \cdot k$ , then our approach of simply adding noise to the points and obtaining  $n \cdot \rho$  error will be only  $O(dk) \cdot \rho$ , which is small.
In summary, we can generate a crude approximation  $F$  with roughly  $O(k)$  cells per grid level (and  $\tilde{O}(k)$  centers total), with small additive ratio. But we desire for the number of centers to be exactly  $k$ , and the multiplicative ratio to be  $O(1)$ , whereas ours will end up being  $d^{O(1)}$ . To achieve such an accurate result, we use  $F$  to partition the data into regions, and apply a private coreset algorithm on
each. By combining these coresets together, we may obtain a private coreset of the full data, and then we can apply an  $O(1)$ -approximate non-private algorithm on the coreset.
A first attempt, inspired by [22, 25], is to send each  $x_{i}$  to a region  $S_{j}$  if  $f_{j} \in F$  is the closest center to  $x_{i}$ , and then compute a standard (i.e., not dist-DP) private coreset on each region  $S_{j}$ . To avoid dealing with large additive errors depending on  $\Lambda$ , we further split each region into a close and far region, depending on whether the distance from  $x_{i}$  to  $f_{j}$  is more than or less than  $S \cdot \rho$  for some parameter  $S$ .
This attempt will still suffer from a large additive cost. For instance, if a point moves, even by distance  $\rho$ , it may move from a close region to a far region. Hence, the far region may have 1 more point, and since the far regions have diameter  $\Lambda$ , an algorithm that is private to adding or deleting a point must incur error proportional to  $\Lambda$ .
Our fix for this is to assign each  $x_{i}$  to a region not based on its closest point and distance, but instead based on  $\tilde{x}_i$ 's closest point and distance, where we recall that  $\tilde{x}_i$  is the noisy version of  $x_{i}$ . For the points  $\{x_i\}$  that are mapped to a far region (meaning  $\tilde{x}_i$  is far from its nearest  $f_{j}$ ), we will simply use  $\{\tilde{x}_i\}$  as the coreset, as  $\tilde{x}_i$  is already dist-DP. However, for points that are mapped to a close region, while we use  $\tilde{x}_i$  to determine which region the point  $x_{i}$  is mapped to, we compute a private coreset using [70] on the points  $x_{i}$ , rather than use the points  $\tilde{x}_i$ .