SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 3 Generalized Exponential Mechanism [RS15]
1: procedure  $\mathbf{GEM}(D\in \mathcal{X}^*;q_i:\mathcal{X}^*\to \mathbb{R}$  for  $i\in [m],\Delta_i > 0$  for  $i\in [m],\varepsilon >0,\beta >0)$    
2: Require:  $q_{i}$  has sensitivity sup  $\sup_{x,x'\in \mathcal{X}^*}\left|q(x) - q(x')\right|\leq \Delta_i$  for all  $i\in [m]$    
3: Let  $t = \frac{2}{\varepsilon}\log \left(\frac{m}{\beta}\right)$    
4: for  $i\in [m]$  do   
5:  $s_i\gets \min_{j\in [m]}\frac{(q_i(D) - t\Delta_i) - (q_j(D) - t\Delta_j)}{\Delta_i + \Delta_j}.$    
6: end for   
7: Sample  $\hat{i}\in [m]$  from the Exponential Mechanism using the normalized scores  $s_i$  ; i.e.,  $\forall i\in [m]\quad \mathbb{P}\left[\hat{i} = i\right] = \frac{\exp\left(\frac{1}{2}\varepsilon s_i\right)}{\sum_{k\in[m]}\exp\left(\frac{1}{2}\varepsilon s_k\right)}.$    
8: return  $\hat{i}\in [m]$    
9: end procedure
Definition 3.1 (Differential Privacy (DP) [DMNS06]). A randomized algorithm  $M: \mathcal{X}^* \to \mathcal{Y}$  satisfies  $\varepsilon$ -DP if, for all inputs  $D, D' \in \mathcal{X}^*$  differing only by the addition or removal of an element and for all measurable  $S \subset \mathcal{Y}$ , we have  $\mathbb{P}[M(D) \in S] \leq e^{\varepsilon} \cdot \mathbb{P}[M(D') \in S]$ .
We refer to pairs of inputs that differ only by the addition or removal of one person's data as neighboring. Note that it is common to also consider replacement of one person's data; for simplicity, we
Data Set	Size	Words per Person	Vocabulary Size
People	Records	Min	Median	Max
Amazon Fashion	404	8533	1	14.0	139	1450
Amazon Industrial and Scientific	11041	1446031	0	86	2059	36665
Reddit	223388	7117494	0	18.0	1724	102835
IMDB	50000	6688844	5	110.0	925	98726
Table 3: Data sets details.
do not do this. We remark that there are also variants of DP such as approximate DP [DKMMN06] and concentrated DP [DR16; BS16], which quantitatively relax the definition, but these are not relevant in our application. A key property of DP is that it composes and is invariant under postprocessing.
Lemma 3.2 (Composition & Postprocessing). Let  $M_1: \mathcal{X}^* \to \mathcal{Y}$  be  $\varepsilon_1$ -DP. Let  $M_2: \mathcal{X}^* \times \mathcal{Y} \to \mathcal{Z}$  be such that, for all  $y \in \mathcal{Y}$ , the restriction  $M(\cdot, y): \mathcal{X}^* \to \mathcal{Z}$  is  $\varepsilon_2$ -DP. Define  $M_{12}: \mathcal{X}^* \to \mathcal{Z}$  by  $M_{12}(D) = M_2(D, M_1(D))$ . Then  $M_{12}$  is  $(\varepsilon_1 + \varepsilon_2)$ -DP.
A basic DP tool is the Laplace mechanism [DMNS06]. Note that we could also use the discrete Laplace mechanism [GRS09; CKS20].
Lemma 3.3 (Laplace Mechanism). Let  $q:\mathcal{X}^*\to \mathbb{R}$ . We say  $q$  has sensitivity  $\Delta$  if  $|q(D) - q(D')|\leq \Delta$  for all neighboring  $D,D^{\prime}\in \mathcal{X}^{*}$ . Define  $M:\mathcal{X}^{*}\rightarrow \mathbb{R}$  by  $M(D) = q(D) + \mathrm{Lap}(\Delta /\varepsilon)$ , where  $\mathrm{Lap}(b)$  denotes laplace noise with mean 0 and variance  $2b^{2} - i.e.$ ,  $\mathbb{P}_{\xi \leftarrow \mathrm{Lap}(b)}[\xi >t] =$
$$
\underset {\xi \leftarrow \operatorname {L a p} (b)} {\mathbb {P}} \left[ \xi <   - t \right] = \frac {1}{2} \exp \left(\frac {t}{b}\right) f o r a l l t > 0. T h e n M i s \varepsilon - D P.
$$
Another fundamental tool for DP is the exponential mechanism [MT07]. It selects the approximately best option from among a set of options, where each option  $i$  has a quality function  $q_{i}$  with sensitivity  $\Delta$ . The following result generalizes the exponential mechanism by allowing each of the quality functions to have a different sensitivity.
Theorem 3.4 (Generalized Exponential Mechanism [RS15, Theorem 1.4]). For each  $i \in [m]$ , let  $q_i: \mathcal{X}^* \to \mathbb{R}$  be a query with sensitivity  $\Delta_i$ . Let  $\varepsilon, \beta > 0$ . The generalized exponential mechanism (GEM( $\because\{\{q_i\}_{i \in [m]}, \{\Delta_i\}_{i \in [m]}, \varepsilon, \beta\}$  in Algorithm 3) is  $\varepsilon$ -DP and has the following utility guarantee. For all  $D \in \mathcal{X}^*$ , we have
$$
\underset {\hat {i} \leftarrow \mathrm {G E M} (D; \{q _ {i} \} _ {i \in [ m ]}, \{\Delta_ {i} \} _ {i \in [ m ]}, \varepsilon , \beta)} {\mathbb {P}} \left[ q _ {i} (D) \geq \max  _ {j \in [ m ]} q _ {j} (D) - \Delta_ {j} \cdot \frac {4}{\varepsilon} \log \left(\frac {m}{\beta}\right) \right] \geq 1 - \beta .
$$
We empirically validate the performance of our algorithms using data sets of various sizes from different text domains. We focus on the problem of computing vocabulary size with person-level DP. Section 4.1 describes the data sets and Section 4.2 discusses the algorithms we compare.
We used four publicly available datasets to assess the accuracy of our algorithms compared to baselines. Two small datasets were used: Amazon Fashion 5-core [NLM19] (reviews of fashion products on Amazon) and Amazon Industrial and Scientific 5-core [NLM19] (reviews of industrial and scientific products on Amazon). Two large data sets were also used: Reddit [She20] (a data set of posts collected from r/AskReddit) and IMDb [N20; MDPHNP11] (a set of movie reviews scraped from IMDb). See details of the datasets in Table 3.
4.2 Comparisons
Computing the number of distinct elements using a differentially private mechanism involves two steps: selecting a contribution bound ( $\ell$  in our algorithms) and counting the number of distinct elements in a way that restricts each person to only contribute the given number of elements.
Selection: We examine four algorithms for determining the contribution limit:
1. Choosing the true maximum person contribution (due to computational restrictions this was only computed for Amazon Fashion data set).
2. Choosing the 90th percentile of person contributions.
3. Choosing the person contribution that exactly maximizes the utility function  $q_{\ell}(D) = \mathrm{DC}(D;\ell) - \frac{\ell}{\varepsilon}\log \left(\frac{1}{2\beta}\right)$ , where  $\varepsilon = 1$ , and  $\beta = 0.001$ .
4. Choosing the person contribution that approximately maximizes the utility function using the generalized exponential mechanism with  $\epsilon = 1$ .
Note that only the last option is differentially private, but we consider the other comparison points nonetheless.
Counting: We also consider three algorithms for estimating the number of distinct elements for a given sensitivity bound  $\ell$ :
1. For each person, we uniformly sample  $\ell$  elements without replacement and count the number of distinct elements in the union of the samples.
2. The linear-time greedy algorithm (Algorithm 2) with  $\varepsilon = 1$  and  $\beta = 0.001$ .
3. The matching-based algorithm (Algorithm 1) with  $\varepsilon = 1$  and  $\beta = 0.001$ .
All of these can be converted into DP algorithms by adding Laplace noise to the result.
In all our datasets "true maximum person contribution" and "90th percentile of person contributions" output bounds that are much larger than necessary to obtain true distinct count; hence, we only consider DP versions of the estimation algorithm for these selection algorithms.
Figure 1 shows the dependency of the result on the contribution bound for each of the algorithms for computing the number of distinct elements with fixed person contribution. It is clear that matching and greedy algorithms vastly outperform the sampling approach that is currently used in practice.
Tables 4 to 7 show the performance of algorithms for selecting optimal person contribution bounds on different data sets. For all bound selection algorithms and all data sets, the sampling approach to estimating the distinct count performs much worse than the greedy and matching-based approaches. The greedy approach performs worse than the matching-based approach, but the difference is about  $10\%$  for Amazon Fashion and is almost negligible for other data sets since they are much larger. As for the matching-based algorithm, it performs as follows on all the data sets:
1. The algorithm that uses the bound equal to the maximal person contribution overestimates the actual necessary bound. Therefore, we only consider the DP algorithms for counts estimation. It is easy to see that while the median of the estimation is close to the actual distinct count, the amount of noise is somewhat large.
2. The algorithm that uses the bound equal to the 99th percentile of person contributions also overestimates the necessary bound and behaves similarly to the one we just described (though the spread of the noise is a bit smaller).
3. The algorithms that optimize the utility function are considered: one non-private and one private. The non-private algorithm with non-private estimation gives the answer that is very close to the true number of distinct elements. The private algorithm with non-private estimation gives the answer that is worse, but not too much. Finally, the private algorithm with the private estimation gives answers very similar to the results of the non-private estimation.
We would like to thank Badih Ghazi, Andreas Terzis, and four anonymous reviewers for their constructive feedback and valuable suggestions. We thank Markus Hasenohrl for helpful discussions, which helped us identify the problem. In addition, we are grateful to Ke Yi, Wei Dong, and Juanru Fang for bringing their related work [DFYTM22; FDY22] to our attention.
[BS16] M. Bun and T. Steinke. “Concentrated differential privacy: Simplifications, extensions, and lower bounds”. In: Theory of Cryptography: 14th International Conference, TCC 2016-B, Beijing, China, October 31-November 3, 2016, Proceedings, Part I. Springer. 2016, pp. 635-658. URL: https://arxiv.org/abs/1605.02065 (cit. on p. 9).
[CGKM21] L. Chen, B. Ghazi, R. Kumar, and P. Manurangi. "On Distributed Differential Privacy and Counting Distinct Elements". In: 12th Innovations in Theoretical Computer Science Conference, ITCS 2021, January 6-8, 2021, Virtual Conference. Ed. by J. R. Lee. Vol. 185. LIPIcs. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2021, 56:1-56:18. URL: https://doi.org/10.4230/LIPIcs.ITCS.2021.56 (cit. on p. 6).
[CKS20] C. L. Canonne, G. Kamath, and T. Steinke. “The discrete gaussian for differential privacy”. In: Advances in Neural Information Processing Systems 33 (2020), pp. 15676–15688. URL: https://arxiv.org/abs/2004.00010 (cit. on p. 9).
[CGW22] R. S. Carvalho, K. Wang, and L. S. Gondara. "Incorporating Item Frequency for Differentially Private Set Union". In: Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022. AAAI Press, 2022, pp. 9504-9511. URL: https://ojs.aaai.org/index.php/AAAI/article/view/21183 (cit. on p. 6).
[DDKPPWXZ23] Z. Ding, J. Durrell, D. Kifer, P. Protivash, G. Wang, Y. Wang, Y. Xiao, and D. Zhang. “A Floating-Point Secure Implementation of the Report Noisy Max with Gap Mechanism”. In: arXiv preprint arXiv:2308.08057 (2023). URL: https://arxiv.org/abs/2308.08057 (cit. on p. 21).
[DFYTM22] W. Dong, J. Fang, K. Yi, Y. Tao, and A. Machanavajhala. “R2T: Instance-Optimal Truncation for Differentially Private Query Evaluation with Foreign Keys”. In: Proceedings of the 2022 International Conference on Management of Data. SIGMOD '22. Philadelphia, PA, USA: Association for Computing Machinery, 2022, 759–772. ISBN: 9781450392495. URL: https://cse.hkust.edu.hk/~yike/R2T.pdf (cit. on pp. 6, 8, 11).
[DKMMN06] C. Dwork, K. Kenthapadi, F. McSherry, I. Mironov, and M. Naor. “Our data, ourselves: Privacy via distributed noise generation”. In: Advances in Cryptology-EUROCRYPT 2006: 24th Annual International Conference on the Theory and Applications of Cryptographic Techniques, St. Petersburg, Russia, May 28-June 1, 2006. Proceedings 25. Springer. 2006, pp. 486-503 (cit. on p. 9).
[DKSWXZ+21] Z. Ding, D. Kifer, T. Steinke, Y. Wang, Y. Xiao, D. Zhang, et al. "The permute-and-flip mechanism is identical to report-noisy-max with exponential noise". In: arXiv preprint arXiv:2105.07260 (2021). URL: https://arxiv.org/abs/2105.07260 (cit. on p. 21).
[DLB19] D. Desfontaines, A. Lochbihler, and D. A. Basin. "Cardinality Estimators do not Preserve Privacy". In: Proc. Priv. Enhancing Technol. 2019.2 (2019), pp. 26-46. URL: https://doi.org/10.2478/popets-2019-0018 (cit. on p. 5).
[SDH23] M. Swanberg, D. Desfontaines, and S. Haney. "DP-SIPS: A simpler, more scalable mechanism for differentially private partition selection". In: CoRR abs/2301.01998 (2023). arXiv: 2301.01998. URL: https://doi.org/10.48550/arXiv.2301.01998 (cit. on p. 6).
[She20] J. H. Shen. Ask Reddit. English. 2020. URL: https : / / github . com / heyyjudes / differentially - private - set - union / tree / ea7b39285dace35cc9e9029692802759f3e1c8e8/data (cit. on p. 9).
[SST20] A. D. Smith, S. Song, and A. Thakurta. “The Flajolet-Martin Sketch Itself Preserves Differential Privacy: Private Counting with Minimal Space”. In: Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. Ed. by H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin. 2020. URL: https://proceedings.neurips.cc/paper/2020/ hash/e3019767b1b23f82883c9850356b71d6 - Abstract .html (cit. on p. 5).
[Ste23] T. Steinke. Beyond Local Sensitivity via Down Sensitivity. DifferentialPrivacy.org. https://differentialprivacy.org/down-sensitivity/. Sept. 2023 (cit. on p. 8).
[Vad17] S. Vadhan. “The complexity of differential privacy”. In: Tutorials on the Foundations of Cryptography: Dedicated to Oded Goldreich (2017), pp. 347–450. URL: https://privacymools.seas.harvard.edu/files/privacytools/files/manuscript_2016.pdf (cit. on p. 8).
[ZDKSTMAS23] B. Zhang, V. Doroshenko, P. Kairouz, T. Steinke, A. Thakurta, Z. Ma, H. Apte, and J. Spacek. "Differentially Private Stream Processing at Scale". In: arXiv preprint arXiv:2303.18086 (2023). URL: https://arxiv.org/abs/2303.18086 (cit. on p. 6).
Selection	Counting	Person Contribution Bound	Distinct Count
10th PC	Median	90th PC	10th PC	Median	90th PC
Max Contrib	DP Sampling	-	139	-	1196.8	1407.5	1649.1
Max Contrib	DP Greedy	-	139	-	1174.2	1439.2	1646.5
Max Contrib	DP Matching	-	139	-	1222.2	1460.9	1631.0
90th PC Contrib	DP Sampling	-	48	-	1225.4	1296.2	1377.9
90th PC Contrib	DP Greedy	-	48	-	1367.0	1432.6	1516.3
90th PC Contrib	DP Matching	-	48	-	1365.3	1444.7	1524.8
Max Utility	Sampling	-	41	-	1247.0	1259.0	1270.0
Max Utility	Greedy	-	20	-	-	1376	-
Max Utility	Matching	-	17	-	-	1428	-
DP Max Utility	Sampling	8.9	16.0	28.0	661.6	892.5	1124.5
DP Max Utility	Greedy	8.0	11.0	17.0	1148.0	1241.0	1348.0
DP Max Utility	Matching	7.0	9.0	14.0	1252.0	1317.0	1400.0
DP Max Utility	DP Sampling	9.0	16.0	27.1	702.4	899.1	1145.1
DP Max Utility	DP Greedy	8.0	10.0	19.0	1128.5	1224.4	1370.8
DP Max Utility	DP Matching	6.9	9.0	13.1	1220.6	1319.1	1394.2
Table 4: Amazon Fashion: the comparison is for  ${\ell }_{\max } = {100}$  .
Selection	Counting	Person Contribution Bound	Distinct Count
10th PC	Median	90th PC	10th PC	Median	90th PC
90th PC Contrib	DP Sampling	-	297	-	32458.1	32943.8	33452.6
90th PC Contrib	DP Greedy	-	297	-	36270.3	36669.5	37019.0
90th PC Contrib	DP Matching	-	297	-	36236.2	36651.7	37102.7
Max Utility	Sampling	-	99	-	24967.0	25039.0	25121.2
Max Utility	Greedy	-	79	-	-	36246	-
Max Utility	Matching	-	42	-	-	36364	-
DP Max Utility	Sampling	85.9	96.0	99.0	23852.8	24739.0	25049.8
DP Max Utility	Greedy	34.0	49.0	66.1	35393.0	35839.0	36116.9
DP Max Utility	Matching	22.9	30.5	43.2	36026.8	36243.5	36371.2
DP Max Utility	DP Sampling	87.0	95.0	99.0	23997.6	24701.1	25067.7
DP Max Utility	DP Greedy	32.9	47.5	68.0	35336.6	35776.2	36136.6
DP Max Utility	DP Matching	22.0	28.0	38.0	35970.5	36198.9	36326.7
Table 5: Amazon Industrial and Scientific: the comparison is for  $\ell_{\mathrm{max}} = 100$ .
Lemma A.1 (Sensitivity of  $\mathrm{DC}(D;\ell)$ ). As in Equation (2), for  $D\in (\Omega^{*})^{n}$  and  $\ell \in \mathbb{N}$ , define
$$
\operatorname {D C} (D; \ell) := \max  \left\{\left| \bigcup_ {i \in [ n ]} v _ {i} \right|: \forall i \in [ n ]   v _ {i} \subseteq u _ {i} \wedge | v _ {i} | \leq \ell \right\}.
$$
Let  $D, D' \in (\Omega^{*})^{*}$  be neighboring. That is,  $D$  and  $D'$  differ only by the addition or removal of one entry. Then  $|\mathrm{DC}(D; \ell) - \mathrm{DC}(D'; \ell)| \leq \ell$  for all  $\ell \in \mathbb{N}$ .
Proof. Without loss of generality  $D = (u_{1}, \dots, u_{n}) \in (\Omega^{*})^{n}$  and  $D' = (u_{1}, \dots, u_{n-1}) \in (\Omega^{*})^{n-1}$ . I.e.,  $D'$  is  $D$  with person  $n$  removed. Let  $\ell \in \mathbb{N}$ .
Selection	Counting	Person Contribution Bound	Distinct Count
10th PC	Median	90th PC	10th PC	Median	90th PC
90th PC Contrib	DP Sampling	-	75	-	92480.7	92654.8	92812.1
90th PC Contrib	DP Greedy	-	75	-	102544.8	102665.7	102817.7
90th PC Contrib	DP Matching	-	75	-	102651.1	102784.1	102907.8
Max Utility	Sampling	-	99	-	95606.9	95692.0	95750.3
Max Utility	Greedy	-	52	-	-	102543	-
Max Utility	Matching	-	32	-	-	102685	-
DP Max Utility	Sampling	89.0	96.0	99.0	94549.9	95394.5	95656.5
DP Max Utility	Greedy	26.0	33.0	50.0	102015.0	102253.0	102527.0
DP Max Utility	Matching	14.0	18.5	30.0	102357.0	102501.5	102671.0
DP Max Utility	DP Sampling	88.8	96.0	99.0	94665.2	95375.5	95693.5
DP Max Utility	DP Greedy	27.0	34.0	53.0	102053.2	102289.6	102531.2
DP Max Utility	DP Matching	14.9	18.5	28.0	102379.7	102512.6	102643.9
Table 6: Reddit: the comparison is for  $\ell_{\mathrm{max}} = 100$
Selection	Counting	Person Contribution Bound	Distinct Count
10th PC	Median	90th PC	10th PC	Median	90th PC
90th PC Contrib	DP Sampling	-	238	-	95264.5	95593.5	95966.1
90th PC Contrib	DP Greedy	-	238	-	98411.0	98734.0	99120.0
90th PC Contrib	DP Matching	-	238	-	98354.2	98729.4	99164.2
Max Utility	Sampling	-	29	-	49907.8	50036.5	50195.3
Max Utility	Greedy	-	29	-	-	98459	-
Max Utility	Matching	-	19	-	-	98712	-
DP Max Utility	Sampling	29.0	29.0	29.0	49899.6	50070.5	50220.9
DP Max Utility	Greedy	22.0	25.0	29.0	98244.0	98364.0	98459.0
DP Max Utility	Matching	13.0	16.0	21.0	98586.0	98674.0	98721.0
DP Max Utility	DP Sampling	29.0	29.0	29.0	49924.2	50053.7	50211.9
DP Max Utility	DP Greedy	20.0	26.0	29.0	98126.7	98369.6	98451.8
DP Max Utility	DP Matching	12.0	16.0	21.0	98555.6	98670.4	98726.8
Table 7:IMDB: the comparison is for  $\ell_{\mathrm{max}} = 30$
Let  $v_{1} \subseteq u_{1}, \dots, v_{n} \subseteq u_{n}$  and  $v_{1}' \subseteq u_{1}, \dots, v_{n-1}' \subseteq u_{n-1}$  satisfy
$$
\forall i \in [ n ] \left| v _ {i} \right| \leq \ell \quad \text {a n d} \quad \left| \bigcup_ {i \in [ n ]} v _ {i} \right| = \mathrm {D C} (D; \ell)
$$
$$
\forall i \in [ n - 1 ] \left| v _ {i} ^ {\prime} \right| \leq \ell \quad \text {a n d} \quad \left| \bigcup_ {i \in [ n - 1 ]} v _ {i} ^ {\prime} \right| = \mathrm {D C} (D ^ {\prime}; \ell).
$$
We can convert the "witness"  $v_{1}^{\prime}, \dots, v_{n-1}^{\prime}$  for  $D^{\prime}$  into a "witness" for  $D$  simply by adding the empty set. Define  $v_{n}^{\prime} = \emptyset$ , which satisfies  $v_{n}^{\prime} \subseteq u_{n}$  and  $|v_{n}^{\prime}| \leq \ell$ . Then
$$
\begin{array}{l} \mathrm {D C} (D; \ell) = \max  \left\{\left| \bigcup_ {i \in [ n ]} v _ {i} \right|: \forall i \in [ n ] v _ {i} \subseteq u _ {i} \wedge | v _ {i} | \leq \ell \right\} \\ \geq \left| \bigcup_ {i \in [ n ]} v _ {i} ^ {\prime} \right| \\ = \operatorname {D C} \left(D ^ {\prime}; \ell\right). \\ \end{array}
$$
Similarly, we can convert the "witness"  $v_{1}, \dots, v_{n}$  for  $D$  into a "witness" for  $D'$  simply by discarding  $v_{n}$ :
$$
\begin{array}{l} \mathrm {D C} (D ^ {\prime}; \ell) = \max  \left\{\left| \bigcup_ {i \in [ n - 1 ]} v _ {i} ^ {\prime} \right|: \forall i \in [ n - 1 ] v _ {i} ^ {\prime} \subseteq u _ {i} \wedge | v _ {i} ^ {\prime} | \leq \ell \right\} \\ \geq \left| \bigcup_ {i \in [ n - 1 ]} v _ {i} \right| \\ \geq \left| \bigcup_ {i \in [ n ]} v _ {i} \right| - | v _ {n} | \\ \geq \operatorname {D C} (D; \ell) - \ell . \\ \end{array}
$$
Thus  $|\mathrm{DC}(D;\ell) - \mathrm{DC}(D';\ell)| \leq \ell$ , as required.
Proof of Theorem 1.1. Privacy: First note that  $q_{\ell}(D) = \mathrm{DC}(D;\ell) - \frac{2\ell}{\varepsilon}\log (1 / 2\beta)$  has sensitivity  $\ell$ . Algorithm 1 accesses the dataset via  $q_{\ell}(D)$  in two ways: First it runs the generalized exponential mechanism to select  $\hat{\ell}$  and second it computes  $\hat{\nu}\gets q_{\hat{\ell}}(D) + \mathrm{Lap}\left(2\hat{\ell} /\varepsilon\right)$ . Since the generalized exponential mechanism is  $\varepsilon /2$ -DP and adding Laplace noise is also  $\varepsilon /2$ -DP, the overall algorithm is  $\varepsilon$ -DP by composition.
Lower bound: Since  $\hat{\nu}\gets q_{\hat{\ell}}(D) + \mathrm{Lap}\left(2\hat{\ell} /\varepsilon\right)$ , we have
$$
\mathbb {P} _ {\hat {\nu}} \left[ \hat {\nu} \leq q _ {\hat {\ell}} (D) + \frac {2 \hat {\ell}}{\varepsilon} \log \left(\frac {1}{2 \beta}\right) \right] = \mathbb {P} _ {\hat {\nu}} \left[ \hat {\nu} \geq q _ {\hat {\ell}} (D) - \frac {2 \hat {\ell}}{\varepsilon} \log \left(\frac {1}{2 \beta}\right) \right] = 1 - \beta . \tag {11}
$$
Since  $\mathrm{DC}(D;\hat{\ell})\leq \mathrm{DC}(D)$  and  $q_{\ell}(D) = \mathrm{DC}(D;\ell) - \frac{2\ell}{\varepsilon}\log (1 / 2\beta)$ , Equation (11) gives
$$
\mathbb {P} _ {\hat {\nu}} [ \hat {\nu} \leq \mathrm {D C} (D) ] \geq \mathbb {P} _ {\hat {\nu}} \left[ \hat {\nu} \leq \mathrm {D C} (D; \hat {\ell}) \right] = 1 - \beta .
$$
This is the guarantee of Equation (6);  $\hat{\nu}$  is a lower bound on  $\mathrm{DC}(D)$  with probability  $\geq 1 - \beta$ .
Upper bound: The accuracy guarantee of the generalized exponential mechanism (Theorem 3.4) is
$$
\mathbb {P} _ {\hat {\ell}} \left[ q _ {\hat {\ell}} (D) \geq \max  _ {\ell \in [ \ell_ {\max } ]} q _ {\ell} (D) - \ell \cdot \frac {4}{\varepsilon / 2} \log \left(\ell_ {\max } / \beta\right) \right] \geq 1 - \beta . \tag {12}
$$
Combining Equations (11) and (12) with a union bound yields
$$
\underset {(\hat {\ell}, \hat {\nu}) \leftarrow \mathcal {M} (D)} {\mathbb {P}} \left[ \hat {\nu} \geq \max  _ {\ell \in [ \ell_ {\max } ]} \mathrm {D C} (D; \ell) - \frac {2 \ell + 2 \hat {\ell}}{\varepsilon} \log \left(\frac {1}{2 \beta}\right) - \frac {8 \ell}{\varepsilon} \log \left(\frac {\ell_ {\max }}{\beta}\right) \right] \geq 1 - 2 \beta . \tag {13}
$$
To interpret Equation (13) we need a high-probability upper bound on  $\hat{\ell}$ . Let  $A > 0$  be determined later and define
$$
\ell_ {A} ^ {*} := \underset {\ell \in [ \ell_ {\max } ]} {\arg \max } \mathrm {D C} (D; \ell) - \frac {A \ell}{\varepsilon}, \tag {14}
$$
so that  $\mathrm{DC}(D;\ell)\leq \mathrm{DC}(D;\ell_A^*) + (\ell -\ell_A^*)\frac{A}{\varepsilon}$  for all  $\ell \in [\ell_{\max}]$ .
Assume the event in Equation (12) happens. We have
$$
\mathrm {D C} (D; \hat {\ell}) - \frac {2 \hat {\ell}}{\varepsilon} \log \left(\frac {1}{2 \beta}\right) \leq \mathrm {D C} (D; \ell_ {A} ^ {*}) + (\hat {\ell} - \ell_ {A} ^ {*}) \frac {A}{\varepsilon} - \frac {2 \hat {\ell}}{\varepsilon} \log \left(\frac {1}{2 \beta}\right),
$$
(by Equation (14))
$$
\begin{array}{l} \mathrm {D C} (D; \hat {\ell}) - \frac {2 \hat {\ell}}{\varepsilon} \log \left(\frac {1}{2 \beta}\right) = q _ {\hat {\ell}} (D) \geq \max  _ {\ell \in [ \ell_ {\max } ]} q _ {\ell} (D) - \ell \cdot \frac {4}{\varepsilon / 2} \log (\ell_ {\max } / \beta) \quad (\text {b y a s s u m p t i o n}) \\ = \max  _ {\ell \in [ \ell_ {\max } ]} \mathrm {D C} (D; \ell) - \frac {2 \ell}{\varepsilon} \log \left(\frac {1}{2 \beta}\right) - \frac {8 \ell}{\varepsilon} \log \left(\frac {\ell_ {\max }}{\beta}\right) \\ \geq \mathrm {D C} (D; \ell_ {A} ^ {*}) - \frac {2 \ell_ {A} ^ {*}}{\varepsilon} \log \left(\frac {1}{2 \beta}\right) - \frac {8 \ell_ {A} ^ {*}}{\varepsilon} \log \left(\frac {\ell_ {\max}}{\beta}\right). \\ \end{array}
$$
Combining inequalities yields