SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alexander Knop*
alexanderknop@google.com
Thomas Steinke*
steinke@google.com
We study the problem of counting the number of distinct elements in a dataset subject to the constraint of differential privacy. We consider the challenging setting of person-level DP (a.k.a. user-level DP) where each person may contribute an unbounded number of items and hence the sensitivity is unbounded.
Our approach is to compute a bounded-sensitivity version of this query, which reduces to solving a max-flow problem. The sensitivity bound is optimized to balance the noise we must add to privatize the answer against the error of the approximation of the bounded-sensitivity query to the true number of unique elements.
An elementary data analysis task is to count the number of distinct elements occurring in a dataset. The dataset may contain private data and even simple statistics can be combined to leak sensitive information about people [DN03]. Our goal is to release (an approximation to) this count in a way that ensures the privacy of the people who contributed their data. As a motivating example, consider a collection of internet browsing histories, in which case the goal is to compute the total number of websites that have been visited by at least one person.
Differential privacy (DP) [DMNS06] is a formal privacy standard. The simplest method for ensuring DP is to add noise (from either a Laplace or Gaussian distribution) to the true answer, where the scale of the noise corresponds to the sensitivity of the true answer – i.e., how much one person's data can change the true value.
If each person contributes a single element to the dataset, then the sensitivity of the number of unique elements is one. However, a person may contribute multiple elements to the dataset and our goal is to ensure privacy for all of these contributions simultaneously. That is, we seek to provide person-level DP (a.k.a. user-level  $\mathrm{DP}^2$ ).
This is the problem we study: We have a dataset  $D = (u_{1}, u_{2}, \dots, u_{n})$  of person records. Each person  $i \in [n]$  contributes a finite dataset  $u_{i} \in \Omega^{*}$ , where  $\Omega$  is some (possibly infinite) universe of potential elements (e.g., all finite-length binary strings) and  $\Omega^{*} := \bigcup_{\ell \in \mathbb{N}} \Omega^{\ell}$  denotes all subsets of  $\Omega$  of finite size. Informally, our goal is to compute the number of unique elements
$$
\mathrm {D C} (D) := \left| \bigcup_ {i \in [ n ]} u _ {i} \right| \tag {1}
$$
*Alphabetical author order.
2We prefer the term "person" over "user," as the latter only makes sense in some contexts and could be confusing in others.
in a way that preserves differential privacy. A priori, the sensitivity of this quantity is infinite, as a single person can contribute an unbounded number of unique elements.
In particular, it is not possible to output a meaningful upper bound on the number of distinct elements subject to differential privacy. This is because a single person could increase the number of distinct elements arbitrarily and differential privacy requires us to hide this contribution. It follows that we cannot output a differentially private unbiased estimate of the number of distinct elements with finite variance. However, it is possible to output a lower bound. Thus our formal goal is to compute a high-confidence lower bound on the number of distinct elements that is as large as possible and which is computed in a differentially private manner.
Given a dataset  $D = (u_{1},\dots ,u_{n})\in (\Omega^{*})^{n}$  and an integer  $\ell \geq 1$ , we define
$$
\mathrm {D C} (D; \ell) := \max  \left\{\left| \bigcup_ {i \in [ n ]} v _ {i} \right|: \forall i \in [ n ] v _ {i} \subseteq u _ {i} \wedge | v _ {i} | \leq \ell \right\}. \tag {2}
$$
That is,  $\mathrm{DC}(D;\ell)$  is the number of distinct elements if we restrict each person's contribution to  $\ell$  elements. We take the maximum over all possible restrictions.
It is immediate that  $\mathrm{DC}(D;\ell)\leq \mathrm{DC}(D)$  for all  $\ell \geq 1$ . Thus we obtain a lower bound on the true number of unique elements. The advantage of  $\mathrm{DC}(D;\ell)$  is that its sensitivity is bounded by  $\ell$  (see Lemma A.1 for a precise statement) and, hence, we can estimate it in a differentially private manner. Specifically,
$$
\mathcal {M} _ {\ell , \varepsilon} (D) := \mathrm {D C} (D; \ell) + \operatorname {L a p} (\ell / \varepsilon) \tag {3}
$$
defines an  $\varepsilon$ -DP algorithm  $M_{\ell, \varepsilon} : (\Omega^{*})^{n} \to \mathbb{R}$ , where  $\mathrm{Lap}(b)$  denotes Laplace noise scaled to have mean 0 and variance  $2b^{2}$ . This forms the basis of our algorithm. Two challenges remain: Setting the sensitivity parameter  $\ell$  and computing  $\mathrm{DC}(D; \ell)$  efficiently.
To obtain a high-confidence lower bound on the true distinct count, we must compensate for the Laplace noise, which may inflate the reported value. We can obtain such a lower bound from  $\mathcal{M}_{\ell}(D)$  using the cumulative distribution function (CDF) of the Laplace distribution: That is,  $\forall b > 0 \forall \beta \in (0,1/2] \mathbb{P}\left[\mathrm{Lap}(b) \geq b \cdot \log\left(\frac{1}{2\beta}\right)\right] = \beta$ , so
$$
\mathbb {P} \left[ \underbrace {\mathcal {M} _ {\ell , \varepsilon} (D) - \frac {\ell}{\varepsilon} \cdot \log \left(\frac {1}{2 \beta}\right)} _ {\text {l o w e r b o u n d}} \leq \mathrm {D C} (D) \right] \geq \underbrace {1 - \beta} _ {\text {c o n f i d e n c e}}. \tag {4}
$$
Choosing the sensitivity parameter  $\ell$ . Any choice of  $\ell \geq 1$  gives us a lower bound:  $\mathrm{DC}(D;\ell) \leq \mathrm{DC}(D)$ . Since  $\forall D \lim_{\ell \to \infty} \mathrm{DC}(D;\ell) = \mathrm{DC}(D)$ , this lower bound can be arbitrarily tight. However, the larger  $\ell$  is, the larger the sensitivity of  $\mathrm{DC}(D;\ell)$  is. That is, the noise we add scales linearly with  $\ell$ .
Thus there is a bias-variance tradeoff in the choice of  $\ell$ . To make this precise, suppose we want a lower bound on  $\mathrm{DC}(D)$  with confidence  $1 - \beta \in [\frac{1}{2}, 1)$ , as in Equation (4). To obtain the tightest possible lower bound with confidence  $1 - \beta$ , we want  $\ell$  to maximize the expectation
$$
q (D; \ell) := \operatorname {D C} (D; \ell) - \frac {\ell}{\varepsilon} \cdot \log \left(\frac {1}{2 \beta}\right) = \underset {\mathcal {M} _ {\ell , \varepsilon}} {\mathbb {E}} \left[ \mathcal {M} _ {\ell , \varepsilon} (D) - \frac {\ell}{\varepsilon} \cdot \log \left(\frac {1}{2 \beta}\right) \right]. \tag {5}
$$
We can use the exponential mechanism [MT07] to privately select  $\ell$  that approximately maximizes  $q(D;\ell)$ . However, directly applying the exponential mechanism is problematic because each score has a different sensitivity – the sensitivity of  $q(\cdot ;\ell)$  is  $\ell$ . Instead, we apply the Generalized Exponential Mechanism (GEM) of Raskhodnikova and Smith [RS15] (see Algorithm 3). Note that we assume some a priori maximum value of  $\ell$  is supplied to the algorithm; this is  $\ell_{\mathrm{max}}$ .
Our main algorithm attains the following guarantees.
Theorem 1.1 (Theoretical Guarantees of Our Algorithm). Let  $\varepsilon >0$  and  $\beta \in (0,\frac{1}{2})$  and  $\ell_{\mathrm{max}}\in \mathbb{N}$  Define  $\mathcal{M}:(\Omega^{*})^{*}\to \mathbb{N}\times \mathbb{R}$  to be  $\mathcal{M}(D) = \mathrm{DPDISTINCTCOUNT}(D;\ell_{\mathrm{max}},\varepsilon ,\beta)$  from Algorithm 1. Then  $\mathcal{M}$  satisfies all of the following properties.
- Privacy:  $\mathcal{M}$  is  $\varepsilon$ -differentially private.
- Lower bound: For all  $D \in (\Omega^{*})^{n}$ ,
$$
\underset {(\hat {\ell}, \hat {\nu}) \leftarrow \mathcal {M} (D)} {\mathbb {P}} [ \hat {\nu} \leq \mathrm {D C} (D) ] \geq 1 - \beta . \tag {6}
$$
- Upper bound: For all  $D \in (\Omega^{*})^{n}$ ,
$$
\underset {(\hat {\ell}, \hat {\nu}) \leftarrow \mathcal {M} (D)} {\mathbb {P}} \left[ \hat {\nu} \geq \max  _ {\ell \in [ \ell_ {\max } ]} \mathrm {D C} (D; \ell) - \frac {1 0 \ell + 1 8 \ell_ {A} ^ {*}}{\varepsilon} \log \left(\frac {\ell_ {\max }}{\beta}\right) \right] \geq 1 - 2 \beta , \tag {7}
$$
$$
w h e r e \ell_ {A} ^ {*} = \arg \max  _ {\ell \in [ \ell_ {\max } ]} \mathrm {D C} (D; \ell) - \frac {\ell}{\varepsilon} \log \left(\frac {1}{2 \beta}\right).
$$
- Computational efficiency:  $\mathcal{M}(D)$  has running time  $O\left(|D|^{1.5} \cdot \ell_{\max}^2\right)$ , where  $|D| := \sum_{i} |u_i|$ .
The upper bound guarantee (7) is somewhat difficult to interpret. However, if the number of items per person is bounded by  $\ell_{*}$ , then we can offer a clean guarantee: If  $D = (u_{1},\dots ,u_{n})\in (\Omega^{*})^{n}$  satisfies  $\max_{i\in [n]}|u_i|\leq \ell_*\leq \ell_{\mathrm{max}}$ , then combining the upper and lower bounds of Theorem 1.1 gives
$$
\underset {(\hat {\ell}, \hat {\nu}) \leftarrow \mathcal {M} (D)} {\mathbb {P}} \left[ \mathrm {D C} (D) \geq \hat {\nu} \geq \mathrm {D C} (D) - \frac {2 8 \ell_ {*}}{\varepsilon} \log \left(\frac {\ell_ {\operatorname* {m a x}}}{\beta}\right) \right] \geq 1 - 3 \beta . \tag {8}
$$
Note that  $\ell_{*}$  is not assumed to be known to the algorithm, but the accuracy guarantee is able to adapt. We only assume  $\ell_{*} \leq \ell_{\max}$ , where  $\ell_{\max}$  is the maximal sensitivity considered by the algorithm.
In addition to proving the above theoretical guarantees, we perform an experimental evaluation of our algorithm.
Algorithm 1 Distinct Count Algorithm
1: procedure SENSITIVE DISTINCT COUNT  $(D = (u_{1},\dots ,u_{n})\in (\Omega^{*})^{n};\ell \in \mathbb{N})$ $\triangleright$  DC(D;l)   
2: Let  $U_{\ell} = \bigcup_{i\in [n]}\left(\{i\} \times [\min \{\ell ,|u_i|\} ]\right)\subset [n]\times [\ell ]$    
3: Let  $V = \bigcup_{i\in [n]}u_{i}\subset \Omega$    
4: Define  $E_{\ell}\subseteq U\times V$  by  $((i,j),v)\in E\iff v\in u_i$    
5: Let  $G_{\ell}$  be a bipartite graph with vertices partitioned into  $U_{\ell}$  and  $V$  and edges  $E_{\ell}$    
6:  $m_{\ell}\gets \mathrm{MAXIMUMMATCHINGSIZE}(G).$  7: return  $m_{\ell}\in \mathbb{N}$    
8: end procedure   
9: procedure DPDISTINCTCOUNT  $(D = (u_{1},\dots ,u_{n})\in (\Omega^{*})^{n};\ell_{\max}\in \mathbb{N},\varepsilon >0,\beta \in (0,\frac{1}{2}))$    
10: for  $\ell \in [\ell_{\mathrm{max}}]$  do   
11: Define  $q_{\ell}(D)\coloneqq \mathrm{SENSITIVE DISTINCT COUNT}(D;\ell) - \frac{2\ell}{\varepsilon}\cdot \log \left(\frac{1}{2\beta}\right)$    
12: end for   
13:  $\hat{\ell}\gets \mathrm{GEM}(D;\{q_\ell \}_{\ell \in [\ell_{\mathrm{max}}]},\{\ell \}_{\ell \in [\ell_{\mathrm{max}}]},\varepsilon /2,\beta).$  14:  $\hat{\nu}\gets q_{\hat{\ell}}(D) + \mathrm{Lap}\left(2\hat{\ell} /\varepsilon\right)$    
15: return  $(\hat{\ell},\hat{\nu})\in [\ell_{\mathrm{max}}]\times \mathbb{R}$    
16: end procedure
Efficient computation. The main computational task for our algorithm is to compute  $\mathrm{DC}(D;\ell)$ . By definition (2), this is an optimization problem. For each person  $i\in [n]$ , we must select a subset  $v_{i}$  of that person's data  $u_{i}$  of size at most  $\ell$  so as to maximize the size of the union of the subsets  $\left|\bigcup_{i\in [n]}v_i\right|$ .
We can view the dataset  $D = (u_{1},\dots ,u_{n})\in (\Omega^{*})^{n}$  as a bipartite graph. On one side we have the  $n$  people and on the other side we have the elements of the data universe  $\Omega$ . There is an edge between  $i\in [n]$  and  $x\in \Omega$  if and only if  $x\in u_{i}$ .
We can reduce computing  $\mathrm{DC}(D;\ell)$  to a max-flow problem: Each edge in the bipartite graph has capacity one. We add a source vertex  $s$  which is connected to each person  $i\in [n]$  by an edge with capacity  $\ell$ . Finally we add a sink  $t$  that is connected to each  $x\in \Omega$  by an edge with capacity 1. The max flow through this graph is precisely  $\mathrm{DC}(D;\ell)$ .
Alternatively, we can reduce computing  $\mathrm{DC}(D;\ell)$  to bipartite maximum matching. For  $\ell = 1$ ,  $\mathrm{DC}(D;1)$  is exactly the maximum cardinality of a matching in the bipartite graph described above. For  $\ell \geq 2$ , we simply create  $\ell$  copies of each person vertex  $i\in [n]$  and then  $\mathrm{DC}(D;\ell)$  is the maximum cardinality of a matching in this new bipartite graph. $^4$
Using this reduction, standard algorithms for bipartite maximum matching [HK73; Kar73] allow us to compute  $\mathrm{DC}(D;\ell)$  with  $O(|D|^{1.5}\cdot \ell)$  operations. We must repeat this computation for each  $\ell \in [\ell_{\max}]$ .
Algorithm 2 Linear-Time Approximate Distinct Count Algorithm
1: procedure DPAPPROXDISTINCTCOUNT  $(D = (u_{1},\dots ,u_{n})\in (\Omega^{*})^{n};\ell_{\max}\in \mathbb{N},\varepsilon >0,\beta \in (0,\frac{1}{2}))$   
2:  $S\gets \emptyset$   
3: for  $\ell \in [\ell_{\mathrm{max}}]$  do  
4: for  $i\in [n]$  with  $u_{i}\setminus S\neq \emptyset$  do  
5: Choose lexicographically first  $v\in u_i\setminus S$  ▷ Match  $(i,\ell)$  to  $v$   
6: Update  $S\leftarrow S\cup \{v\}$   
7: end for  
8: Define  $q_{\ell}(D):= |S| - \frac{2\ell}{\varepsilon}\cdot \log \left(\frac{1}{2\beta}\right)$  ▷ This loop computes  $\{q_{\ell}(D)\}_{\ell \in [\ell_{\mathrm{max}}]}$   
9: end for  
10:  $\hat{\ell}\gets \mathrm{GEM}(D;\{q_{\ell}\}_{\ell \in [\ell_{\mathrm{max}}]},\{\ell \}_{\ell \in [\ell_{\mathrm{max}}]},\varepsilon /2,\beta)$  ▷ Algorithm 3  
11:  $\hat{\nu}\gets q_{\hat{\ell}}(D) + \mathrm{Lap}\left(2\hat{\ell} /\varepsilon\right)$   
12: return  $(\hat{\ell},\hat{\nu})\in [\ell_{\mathrm{max}}]\times \mathbb{R}$   
13: end procedure
Linear-time algorithm. Our algorithm above is polynomial-time. However, for many applications the dataset size  $|D|$  is enormous. Thus we also propose a linear-time variant of our algorithm. However, we must trade accuracy for efficiency.
There are two key ideas that differentiate our linear-time algorithm (Algorithm 2) from our first algorithm (Algorithm 1) above: First, we compute a maximal bipartite matching instead of a maximum bipartite matching. This can be done using a linear-time greedy algorithm and gives a 2-approximation to the maximum matching. (Experimentally we find that the approximation is better than a factor of 2.) Second, rather than repeating the computation from scratch for each  $\ell \in [\ell_{\max}]$ , we incrementally update our a maximal matching while increasing  $\ell$ . The main challenge here is ensuring that the approximation to  $\mathrm{DC}(D;\ell)$  has low sensitivity - i.e., we must ensure that our approximation algorithm doesn't inflate the sensitivity. Note that  $\mathrm{DC}(D;\ell)$  having low sensitivity does not automatically ensure that the approximation to it has low sensitivity.
Theorem 1.2 (Theoretical Guarantees of Our Linear-Time Algorithm). Let  $\varepsilon > 0$  and  $\beta \in (0, \frac{1}{2})$  and  $\ell_{\max} \in \mathbb{N}$ . Define  $\mathcal{M}: (\Omega^{*})^{*} \to \mathbb{N} \times \mathbb{R}$  to be  $\widehat{\mathcal{M}}(D) =$
3The data universe  $\Omega$  may be infinite, but we can restrict the computation to the finite set  $\bigcup_{i\in [n]}u_i$ . Thus there are at most  $n + \mathrm{DC}(D)\leq n + |D|$  item vertices in the graph.
<sup>4</sup>We need only create  $\min \{ \ell, |u_i| \}$  copies of the person  $i \in [n]$ . Thus the number of person vertices is at most  $\min \{ n\ell, |D| \}$ .
To clarify the confusing terminology: A matching is a subset of edges such that no two edges have a vertex in common. A maximum matching is a matching of the largest possible size. A maximal matching is a matching such that no edge could be added to the matching without violating the matching property. A maximum matching is also a maximal matching, but the reverse is not true.
Data Set	Vocabulary Size	Estimated Vocabulary Size
10th Percentile	Median	90th Percentile
Amazon Fashion	1450	1220.6	1319.1	1394.2
Amazon Industrial and Scientific	36665	35970.5	36198.9	36326.7
Reddit	102835	102379.7	102512.6	102643.9
IMDB	98726	98555.6	98670.4	98726.8
Table 1: True and estimated (using DPDistinctCount with  $\varepsilon = 1$ ,  $\beta = 0.05$  and  $\ell_{\max} = 100$ ) counts per data set.
DPAPPROXDISTINCTCOUNT  $(D;\ell_{\mathrm{max}},\varepsilon ,\beta)$  from Algorithm 2. Then  $\widehat{\mathcal{M}}$  satisfies all of the following properties.
- Privacy:  $\widehat{\mathcal{M}}$  is  $\varepsilon$ -differentially private.
- Lower bound: For all  $D \in (\Omega^{*})^{n}$ ,
$$
\underset {(\hat {\ell}, \hat {\nu}) \leftarrow \widehat {\mathcal {M}} (D)} {\mathbb {P}} [ \hat {\nu} \leq \mathrm {D C} (D) ] \geq 1 - \beta . \tag {9}
$$
- Upper bound: If  $D = (u_{1}, \dots, u_{n}) \in (\Omega^{*})^{n}$  satisfies  $\max_{i \in [n]} |u_{i}| \leq \ell_{*} \leq \ell_{\max}$ , then
$$
\underset {(\hat {\ell}, \hat {\nu}) \leftarrow \widehat {\mathcal {M}} (D)} {\mathbb {P}} \left[ \hat {\nu} \geq \frac {1}{2} \mathrm {D C} (D) - \frac {2 8 \ell_ {*}}{\varepsilon} \log \left(\frac {\ell_ {\operatorname* {m a x}}}{\beta}\right) \right] \geq 1 - 2 \beta . \tag {10}
$$
- Computational efficiency:  $\mathcal{M}(D)$  has running time  $O(|D| + \ell_{\max} \log \ell_{\max})$ , where  $|D| := \sum_{i} |u_i|$ .
The factor  $\frac{1}{2}$  in the upper bound guarantee (10) is the main loss compared to Theorem 1.1. (The win is  $O(|D|)$  runtime.) This is a worst-case bound and our experimental result show that for realistic data the performance gap is not so bad.
The proofs of Theorems 1.1 and 1.2 are in Appendix A.
Counting the number of distinct elements in a collection is one of the most fundamental database computations. This is supported as the COUNT(DISTINCT ...) operation in SQL. Hence, unsurprisingly, the problem of computing the number of unique elements in a differentially private way has been extensively investigated.
In the case where we assume each person contributes only one element (a.k.a. event-level privacy or item-level privacy), the number of distinct elements has sensitivity 1 and, hence, we can simply use Laplace (or Gaussian) noise addition to release. However, it may not be possible to compute the number of distinct elements exactly due to space, communication, or trust constraints (e.g. in the local model of DP [KLNRS11]).
Most efforts have been focused on creating differentially private algorithms for counting distinct elements under space constraints (and assuming each person contributes a single element). To save space, we wish to compute a small summary of the dataset (called a sketch) that allows us to estimate the number of distinct elements and which can be updated as more elements are added. Smith, Song, and Thakurta [SST20] proved that a variant of the Flajolet-Martin sketch is private and Pagh and Stausholm [PS20] analyzed a sketch over the binary finite field. Dickens, Thaler, and Ting [DTT22] proved a general privacy result for order-invariant cardinality estimators. Hehir, Ting, and Cormode [HTC23] provided a mergeable private sketch (i.e. two sketches can be combined to obtain a sketch of the union of the two datasets). In contrast, Desfontaines, Lochbihler, and Basin [DLB19] proved an impossibility result for mergeable sketches, which shows that privacy or accuracy must degrade as we merge sketches.
Figure 1: Performance of different algorithms estimating distinct count assuming that each person can contribute at most  $\ell$  elements (e.g., these algorithms are estimating  $\mathrm{DC}(D;\ell)$ ). (These algorithms have bounded sensitivity, but we do not add noise for privacy yet.)
Counting unique elements has been considered in the pan-private streaming setting [DNPRY10] (the aforementioned algorithms also work in the pan-private setting) and in the continual release streaming setting [GKNM23]. (In the continual release setting the approximate count is continually updated, while in the pan-private setting the approximate count is only revealed once, but at an unknown point in time.) Kreuter, Wright, Skvortsov, Mirisola, and Wang [KWSMW20] give private algorithms for counting distinct elements in the setting of secure multiparty computation. In the local and shuffle models, the only known results are communication complexity bounds [CGKM21].
A closely related problem is that of identifying as many elements as possible (rather than just counting them); this is known as "partition selection," "set union," or "key selection" [SDH23; DVGM22; KKMN09; CWG22; RCR22; GGKSSY20; ZDKSTMAS23]. Note that, by design, DP prevents us from identifying elements that only appear once in the dataset, or only a few times. Thus we can only output items that appear frequently.
The most closely related work to ours is that of Dong, Fang, Yi, Tao, and Machanavajjhala [DFYTM22] and Fang, Dong, and Yi [FDY22]. These papers present two different algorithms for privately approximating the distinct count (and other statistics). We discuss these below and present an experimental comparison in Table 2. We also remark that both papers prove instance optimality guarantees for their algorithms.
Figure 2: Performance of different algorithms estimating distinct count in a differentially private way for different values of  $\varepsilon$ ; for all of them  $\beta = 0.05$  and  $\ell_{\max} = 100$ . The values between 10th and 90th percentile of each algorithms estimation are shaded into corresponding colors. For the shifted inverse algorithm, the first two plots contain the results for  $\beta = 0.05$  and  $D$  equal to the true number of distinct elements in the dataset. The later two datasets are lacking the results for shifted inverse algorithm due to the computational constraints.
User Attribute	Supplier	Customer
PS.AQ	L.EP	O.OD	L.RD
R2T [DFYTM22]	0.0658	0.1759	0.0061	0.150
(Approx)ShiftedInverse [FDY22]	0.0553	0.0584	0.005	0.0061
DPApproxDistinctCount	0.0140	0.0110	0.0008	0.0037
DPDistinctCount	0.0100	0.0096	0.0008	0.0001
Table 2: Average relative absolute error of algorithms described in this paper and in [DFYTM22; FDY22] on the TPC-H dataset. For each algorithm we executed it 100 times, removed 20 top and 20 bottom values and computed average error for the rest of 60 values.
Most similar to our algorithm is the Race-to-the-Top (R2T) algorithm [DFYTM22]; R2T is a generic framework and the original paper did not specifically consider counting distinct elements, but the approach can easily be applied to  $\mathrm{DC}(D;\ell)$ . While we use the generalized exponential mechanism [RS15] to select the sensitivity  $\ell$ , R2T computes multiple lower bounds with different sensitivities  $\ell$  and then outputs the maximum of the noisy values. This approach incurs the cost of composition across the multiple evaluations. To manage this cost, R2T only evaluates  $\ell = 2,4,8,\dots ,2^{\log \ell_{\max}}$ . Compared to our guarantee (8) with an error  $O\left(\frac{\ell_*}{\varepsilon}\log \left(\frac{\ell_{\max}}{\beta}\right)\right)$ , R2T has a slightly worse theoretical error guarantee of  $O\left(\frac{\ell_*}{\varepsilon}\log (\ell_{\max})\log \left(\frac{\log\ell_{\max}}{\beta}\right)\right)$  [DFYTM22, Theorem 5.1].
The shifted inverse mechanism [FDY22] takes a different approach to the problem. Rather than relying on adding Laplace noise (as we do), it applies the exponential mechanism with an ingenious loss function (see [Ste23] for additional discussion). When applied to counting distinct elements, the shifted inverse mechanism gives an accuracy guarantee comparable to ours (8). The downside of the shifted inverse mechanism is that computing the loss function is, in general, NP-hard. Fang, Dong, and Yi [FDY22] propose polynomial-time variants for several specific tasks, including counting distinct elements. However, the algorithm is still relatively slow.
3 Technical Background on Differential Privacy
For detailed background on differential privacy, see the survey by Vadhan [Vad17] or the book by Dwork and Roth [DR14]. We briefly define pure DP and some basic mechanisms and results.