SlowGuess/NeurIPS_2023_OCR huggingface dataset:
On a more technical front, our transformation for approximate-DP (Theorem 10) has an item-level privacy loss (roughly) of the form  $\frac{\varepsilon^2}{\sqrt{m\log(1 / \delta)^3}}$  while it seems plausible that  $\frac{\varepsilon}{\sqrt{m\log(1 / \delta)}}$  can be achieved; an obvious open question is to give such an improvement, or show that it is impossible. We note that the extra factor of  $\frac{\log(1 / \delta)}{\varepsilon}$  comes from the fact that Algorithm 1 uses LDDP with distance  $O(\kappa) = O\left(\frac{\log(1 / \delta)}{\varepsilon}\right)$ . It is unclear how one could avoid this, given that the propose-test-release framework requires checking input at distance  $O\left(\frac{\log(1 / \delta)}{\varepsilon}\right)$ .
Another limitation of our algorithms is their computational inefficiency, as their running time grows linearly with the size of the possible outputs. In fact, the approximate-DP algorithm could even be slower than this since we need to check whether  $\mathbb{A}$  is LDDP at a certain input  $x$  (which involves computing  $d_{\varepsilon}(\mathbb{A}_{-S} \parallel \mathbb{A}_{-S'})$ ). Nevertheless, given the generality of our results, it is unlikely that a generic efficient algorithm exists with matching user complexity; for example, even the algorithm for user-level DP-SCO in  $[\mathrm{GKK}^{+}23]$  has a worst case running time that is not polynomial. Thus, it remains an interesting direction to devise more efficient algorithms for specific tasks.
Finally, while our work has focused on the central model of DP, where the algorithm gets the access to the raw input data, other models of DP have also been studied in the context of user-level DP, such as the local DP model [GKM21, ALS23]. Here again, [GKM21,  $\mathrm{BGH}^{+}23]$  give a generic transformation that sometimes drastically reduces the user complexity in the example-rich case. Another interesting research direction is to devise a refined transformation for the example-sparse case (like one presented in our paper for the central model) but for the local model.
$^{8}$ E.g., for PAC learning, Theorem 4 provides a better user complexity than applying the approximate-DP transformation (Theorem 10) to item-level approximate-DP learners for a large regime of parameters and concept classes; the former has user complexity that decreases with  $1 / m$  whereas the latter decreases with  $1 / \sqrt{m}$ .
[Ab018] John M Abowd. The US Census Bureau adopts differential privacy. In KDD, pages 2867-2867, 2018.
[ALS23] Jayadev Acharya, Yuhan Liu, and Ziteng Sun. Discrete distribution estimation under user-level local differential privacy. In AISTATS, 2023.
[App17] Apple Differential Privacy Team. Learning with privacy at scale. *Apple Machine Learning Journal*, 2017.
[ASZ21] Jayadev Acharya, Ziteng Sun, and Huanyu Zhang. Differentially private Assouad, Fano, and Le Cam. In ALT, pages 48-78, 2021.
[BBG18] Borja Balle, Gilles Barthe, and Marco Gaboardi. Privacy amplification by subsampling: Tight analyses via couplings and divergences. In NeurIPS, pages 6280-6290, 2018.
[BF16] Raef Bassily and Yoav Freund. Typicality-based stability and privacy. CoRR, abs/1604.03336, 2016.
$\left[\mathrm{BGH}^{+}23\right]$  Mark Bun, Marco Gaboardi, Max Hopkins, Russell Impagliazzo, Rex Lei, Toniann Pitassi, Satchit Sivakumar, and Jessica Sorrell. Stability is stable: Connections between replicability, privacy, and adaptive generalization. In STOC, 2023.
[BKSW19] Mark Bun, Gautam Kamath, Thomas Steinke, and Zhiwei Steven Wu. Private hypothesis selection. In NeurIPS, pages 156–167, 2019.
[BNS19] Amos Beimel, Kobbi Nissim, and Uri Stemmer. Characterizing the sample complexity of pure private learners. JMLR, 20:146:1-146:33, 2019.
$\left[\mathrm{CLN}^{+}16\right]$  Rachel Cummings, Katrina Ligett, Kobbi Nissim, Aaron Roth, and Zhiwei Steven Wu. Adaptive learning with robust generalization guarantees. In  $COLT$ , pages 772-814, 2016.
[DHS15] Ilias Diakonikolas, Moritz Hardt, and Ludwig Schmidt. Differentially private learning of structured discrete distributions. In NIPS, pages 2566-2574, 2015.
$\left[\mathrm{DKM}^{+}06\right]$  Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In EUROCRYPT, pages 486-503, 2006.
[DKY17] Bolin Ding, Janardhan Kulkarni, and Sergey Yekhanin. Collecting telemetry data privately. In NeurIPS, pages 3571-3580, 2017.
[DL09] Cynthia Dwork and Jing Lei. Differential privacy and robust statistics. In STOC, pages 371-380, 2009.
[DMNS06] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam D. Smith. Calibrating noise to sensitivity in private data analysis. In TCC, pages 265–284, 2006.
[DMR20] Luc Devroye, Abbas Mehrabian, and Tommy Reddad. The minimax learning rates of normal and Ising undirected graphical models. Electronic Journal of Statistics, 14(1):2338 - 2361, 2020.
[GGKM21] Badih Ghazi, Noah Golowich, Ravi Kumar, and Pasin Manurangsi. Sample-efficient proper PAC learning with approximate differential privacy. In STOC, 2021.
$\left[\mathrm{GKK}^{+}23\right]$  Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Raghu Meka, and Chiyuan Zhang. On user-level private convex optimization. In ICML, 2023.
[GKM21] Badih Ghazi, Ravi Kumar, and Pasin Manurangsi. User-level differentially private learning via correlated sampling. In NeurIPS, pages 20172-20184, 2021.
[Gre16] Andy Greenberg. Apple's "differential privacy" is about collecting your data – but not your data. Wired, June, 13, 2016.
[ILPS22] Russell Impagliazzo, Rex Lei, Toniann Pitassi, and Jessica Sorrell. Reproducibility in learning. In STOC, pages 818-831, 2022.
[KL21] Nitin Kohli and Paul Laskowski. Differential privacy for black-box statistical analyses. In TPDP, 2021.
[KLSU19] Gautam Kamath, Jerry Li, Vikrant Singhal, and Jonathan R. Ullman. Privately learning high-dimensional distributions. In  $COLT$ , pages 1853-1902, 2019.
[Kut02] Samuel Kutin. Extensions to McDiarmid's inequality when differences are bounded with high probability. *Dept. Comput. Sci.*, Univ. Chicago, Chicago, IL, USA, Tech. Rep. TR-2002-04, 2002.
$\left[\mathrm{LSA}^{+}21\right]$  Daniel Levy, Ziteng Sun, Kareem Amin, Satyen Kale, Alex Kulesza, Mehryar Mohri, and Ananda Theertha Suresh. Learning with user-level privacy. In NeurIPS, pages 12466-12479, 2021.
[LSY+20] Yuhan Liu, Ananda Theertha Suresh, Felix X. Yu, Sanjiv Kumar, and Michael Riley. Learning discrete distributions: user vs item-level privacy. In NeurIPS, 2020.
[MT07] Frank McSherry and Kunal Talwar. Mechanism design via differential privacy. In FOCS, pages 94-103, 2007.
[NME22] Shyam Narayanan, Vahab S. Mirrokni, and Hossein Esfandiari. Tight and robust private mean estimation with few users. In ICML, pages 16383-16412, 2022.
[RE19] Carey Radebaugh and Ulfar Erlingsson. Introducing TensorFlow Privacy: Learning with Differential Privacy for Training Data, March 2019. blog.tensorflow.org.
[Tal95] Michel Talagrand. Concentration of measure and isoperimetric inequalities in product spaces. *Publications Mathématiques de l'Institut des Hautes Études Scientifiques*, 81:73-205, 1995.
[TM20] Davide Testuggine and Ilya Mironov. PyTorch Differential Privacy Series Part 1: DP-SGD Algorithm Explained, August 2020. medium.com.
[Vad17] Salil P. Vadhan. The complexity of differential privacy. In Tutorials on the Foundations of Cryptography, pages 347-450. Springer International Publishing, 2017.
[War16] Lutz Warnke. On the method of typical bounded differences. Comb. Probab. Comput., 25(2):269-299, 2016.
Our results for pure-DP are summarized in Table 1.
For approximate-DP, our transformation (Theorem 10) is very general and can be applied to any statistical tasks. Due to this, we do not attempt to exhaustively list its corollaries. Rather, we give only a few examples of consequences of Theorem 10 in Table 2. We note that in all cases, we either improve upon previous results or match them up to  $\frac{\log(1 / \delta)^{O(1)}}{\varepsilon}$  factor for all values of  $m$ .
Table 2: Example results on approximate-DP. Each of our results is a combination of Theorem 10 and the previously known item-level DP bound. For PAC learning,  $d_{L}$  denotes the Littlestone's dimension of the concept class. We assume that  $\delta \ll \varepsilon$  and use  $\widetilde{O},\widetilde{\Theta}$  to also hide polylogarithmic factors in  $1 / \delta$ .
In this section, we list some additional material that is required for the remaining proofs.
For  $a, b \in \mathbb{R}$  such that  $a \leq b$ , let  $\operatorname{clip}_{a,b} : \mathbb{R} \to \mathbb{R}$  denote the function
$$
\operatorname {c l i p} _ {a, b} (x) = \left\{ \begin{array}{l l} a & \text {i f} x <   a, \\ b & \text {i f} x > b, \\ x & \text {o t h e r w i s e .} \end{array} \right.
$$
For two distributions  $A, B$ , we use  $d_{\mathrm{tv}}(A, B) \coloneqq \frac{1}{2} \sum_{x \in \operatorname{supp}(A) \cup \operatorname{supp}(B)} |A(x) - B(x)|$  to denote their total variation distance,  $d_{\mathrm{KL}}(A \| B) \coloneqq \sum_{x \in \operatorname{supp}(A)} A(x) \cdot \log (A(x) / B(x))$  to denote their Kullback-Leibler divergence, and  $d_{\chi^2}(A \| B) \coloneqq \sum_{x \in \operatorname{supp}(A)} \frac{(A(x) - B(x))^2}{B(x)}$  to denote the  $\chi^2$ -divergence. We use  $A^k$  to denote the distribution of  $k$  independent samples drawn from  $A$ . The following three well-known facts will be useful in our proofs.
Lemma 23. For any two distributions  $A$  and  $B$ ,
(i) (Pinsker's inequality)  $d_{\mathrm{tv}}(A,B)\leq \sqrt{\frac{1}{2}\cdot d_{\mathrm{KL}}(A\parallel B)}.$
(ii)  $d_{\mathrm{KL}}(A^k\parallel B^k) = k\cdot d_{\mathrm{KL}}(A\parallel B)$  for  $k\in \mathbb{N}$
(iii)  $d_{\mathrm{KL}}(A\parallel B)\leq d_{\chi^2}(A\parallel B).$
We use  $\| u \|_0$  to denote the Hamming "norm" of  $u$ , i.e., the number of non-zero coordinates of  $u$ . The following is a well-known fact (aka constant-rate constant-distance error correcting codes), which will be useful in our lower bound proofs.
Theorem 24. For  $d \in \mathbb{N}$ , there are  $u_1, \ldots, u_W \in \{0,1\}^d$  for  $W = 2^{\Omega(d)}$  such that  $\|u_i - u_j\|_0 \geq \Omega(d)$  for all distinct  $i, j \in [W]$ .
Next, we list a few concentration inequalities in convenient forms for subsequent proofs.
Chernoff bound. We start with the standard Chernoff bound.
Lemma 25 (Chernoff Bound). Let  $X_{1},\ldots ,X_{n}$  denote i.i.d. Bernoulli random variables with success probability  $p$ . Then, for all  $\delta >0$ , we have
$$
\Pr \left[ X _ {1} + \dots + X _ {n} \leq (1 - \delta) p n \right] \leq \exp \left(\frac {- \delta^ {2} p n}{2}\right),
$$
$$
\Pr \left[ X _ {1} + \dots + X _ {n} \geq (1 + \delta) p n \right] \leq \exp \left(\frac {- \delta^ {2} p n}{2 + \delta}\right).
$$
McDiarmid's inequality. For a distribution  $\mathcal{D}$  on  $\mathcal{Z}$ , and a function  $g: \mathcal{Z}^n \to \mathbb{R}$ , let  $\mu_{\mathcal{D}^n}(g) \coloneqq \mathbb{E}_{\boldsymbol{x} \sim \mathcal{D}^n}[g(\boldsymbol{x})]$ . We say that  $g$  is a  $c$ -bounded difference function iff  $|g(\boldsymbol{x}) - g(\boldsymbol{x}')| \leq c$  for all  $\boldsymbol{x}, \boldsymbol{x}' \in \mathcal{Z}^n$  that differ on a single coordinate (i.e.,  $\| \boldsymbol{x} - \boldsymbol{x}' \|_0 = 1$ ).
Lemma 26 (McDiarmid's inequality). Let  $g: \mathcal{Z}^n \to \mathbb{R}$  be any  $c$ -bounded difference function. Then, for all  $t > 0$
$$
\Pr \left[ | g (\boldsymbol {x}) - \mu_ {\mathcal {D} ^ {n}} (g) | > t \right] \leq 2 \exp \left(\frac {- 2 t ^ {2}}{n c ^ {2}}\right).
$$
We will also use the following consequence of Azuma's inequality.
Lemma 27. Let  $C_1, \ldots, C_n$  be sequence of random variables. Suppose that  $|C_i| \leq \alpha$  almost surely and that  $\gamma \leq \mathbb{E}[C_i \mid C_1 = c_1, \ldots, C_{i-1} = c_{i-1}] \leq 0$  for any  $c_1, \ldots, c_{i-1}$ . Then, for every  $G > 0$ , we have  $\operatorname*{Pr}\left[n\gamma - \alpha \sqrt{nG} \leq \sum_{i \in [n]} C_i \leq \alpha \sqrt{nG}\right] \geq 1 - 2\exp(-G/2)$ .
9This is not actually a norm, but often called as such.
B.2 Observations on Item-level vs User-level DP
Here we list a couple of trivial observations on the user / sample complexity in the item vs user-level DP settings.
Observation 28. Let  $\mathfrak{T}$  be any task and  $\gamma >0$  be a parameter. Then, for any  $\varepsilon ,\delta \geq 0$  , and  $m\in \mathbb{N},$  it holds that
1.  $n_{m}^{\mathfrak{T}}(\varepsilon, \delta; \gamma) \leq n_{1}^{\mathfrak{T}}(\varepsilon, \delta; \gamma)$ , and
2.  $n_m^{\xi}(\varepsilon, \delta; \gamma) \geq \lceil n_1^{\xi}(\varepsilon, \delta; \gamma) / m \rceil$ .
Proof. Both parts follow by simple reductions as given below.
1. Let  $\mathbb{A}$  be an  $(\varepsilon, \delta)$ -item-level DP  $\gamma$ -useful algorithm for  $\mathfrak{T}$ . Then, in the user-level DP algorithm, each user can simply throw away all-but-one example and run  $\mathbb{A}$ . Clearly, this algorithm is  $(\varepsilon, \delta)$ -user-level DP and is  $\gamma$ -useful for  $\mathfrak{T}$ . The user complexity remains the same.
2. Let  $\mathbb{A}$  be an  $(\varepsilon, \delta)$ -user-level DP  $\gamma$ -useful algorithm for  $\mathfrak{T}$  with user complexity  $n_m^{\mathfrak{T}}(\varepsilon, \delta; \gamma)$ . Then, in the item-level DP algorithm, if there are  $N = m \cdot n_m^{\mathfrak{T}}(\varepsilon, \delta; \gamma)$  users, we can just run  $\mathbb{A}$  by grouping each  $m$  examples together into a "super-user". Clearly, this algorithm is also  $(\varepsilon, \delta)$ -item-level DP and has user complexity  $m \cdot n_m^{\mathfrak{T}}(\varepsilon, \delta; \gamma)$ . Thus, we have  $m \cdot n_m^{\mathfrak{T}}(\varepsilon, \delta; \gamma) \leq n_1^{\mathfrak{T}}(\varepsilon, \delta; \gamma)$ .
C Missing Details from Section 3
C.1 DP Implies Perfect Generalization
We start by recalling the definition of perfectly generalizing algorithms.
Definition 29 (Perfect generalization  $[\mathrm{CLN}^+16,\mathrm{BF16}])$ . For  $\beta ,\varepsilon ,\delta >0$  , an algorithm  $\mathbb{A}:\mathcal{Z}^n\to \mathcal{O}$  is said to be  $(\beta ,\varepsilon ,\delta)$ -perfectly generalizing iff, for any distribution  $\mathcal{D}$  (over  $\mathcal{Z}$ ), there exists a distribution  $\mathrm{SIM}_{\mathcal{D}}$  such that
$$
\operatorname * {P r} _ {\boldsymbol {x} \sim \mathcal {D} ^ {n}} [ \mathbb {A} (\boldsymbol {x}) \approx_ {\varepsilon , \delta} \mathrm {S I M} _ {\mathcal {D}} ] \geq 1 - \beta .
$$
It is known that perfect generalization and sample perfect generalization (Definition 12) are equivalent:
Lemma 30 ([CLN $^{+}$ 16,  $\mathrm{BGH}^{+}23]$ ). Any  $(\beta, \varepsilon, \delta)$ -perfectly generalizing algorithm is also  $(\beta, 2\varepsilon, 3\delta)$ -sample perfectly generalizing. Any  $(\beta, \varepsilon, \delta)$ -sample perfectly generalizing algorithm is also  $(\sqrt{\beta}, \varepsilon, \delta + \sqrt{\beta})$ -perfectly generalizing; furthermore, this holds even when we set  $\mathrm{SIM}_{\mathcal{D}}$  to be the distribution of  $\mathbb{A}(\boldsymbol{x})$  where  $\boldsymbol{x} \sim \mathcal{D}^m$ .
Thus, by plugging the above into Theorem 13, we immediately arrive at the following theorem, which shows that any  $(\varepsilon, \delta)$ -DP algorithm is  $(\beta, \widetilde{O}(\varepsilon \sqrt{n}), O(n\delta))$ -perfectly generalizing. This resolves an open question of  $[\mathrm{CLN}^{+}16]$ .
Theorem 31 (DP implies perfect generalization). Suppose that  $\mathbb{A}$  is an  $(\varepsilon, \delta)$ -item-level DP algorithm. Further, assume that  $\varepsilon, \delta, \beta$ , and  $\varepsilon \sqrt{n \log(1 / \beta \delta)}$  are sufficiently small. Then,  $\mathbb{A}$  is  $(\beta, \varepsilon', \delta')$ -perfectly generalizing, where  $\varepsilon' \leq O(\varepsilon \sqrt{n \log(1 / \beta \delta)})$  and  $\delta' = O(n \delta)$ .
Furthermore, this holds even when we set  $\mathrm{SIM}_{\mathcal{D}}$  to be the distribution of  $\mathbb{A}(\pmb{x})$  where  $\pmb{x} \sim \mathcal{D}^n$ .
We remark that Bun et al.  $\mathrm{[BGH^{+}23]}$  showed that any  $(\varepsilon, \delta)$ -DP algorithm can be "repackaged" to an  $(\beta, \varepsilon', \delta')$ -perfectly generalizing algorithm with similar parameters as our theorem above. However, unlike our result, their proof does not show that the original algorithm is perfectly generalizing.
C.2 Achieving the Same  $\varepsilon$  via Subsampling: Proof of Theorem 11
Proof of Theorem 11. Let  $\varepsilon'' = \frac{\varepsilon^2}{\log(1/\delta)\sqrt{m\log(m/\delta)}}$ . We claim that  $n_1(\varepsilon'', \delta') \leq n_1(\varepsilon, \delta') \cdot O(\varepsilon/\varepsilon'')$ ; the lemma follows from this claim by simply plugging this bound into Theorem 10.
To see that this claim holds, by definition of  $n_1$ , there exists an  $(\varepsilon, \delta')$ -item-level DP algorithm  $\mathbb{A}_0$  that is  $\gamma$ -useful for  $\mathfrak{T}$  with sample complexity  $n_1(\varepsilon, \delta')$ . Let  $N \in \mathbb{N}$  be a smallest number such that
$$
(e ^ {\varepsilon} - 1) \frac {n _ {1} (\varepsilon , \delta^ {\prime})}{N} \leq e ^ {\varepsilon^ {\prime \prime}} - 1.
$$
Notice that we have  $N \leq n_{1}(\varepsilon, \delta') \cdot O(\varepsilon / \varepsilon'')$ . Let  $\mathbb{A}$  be an algorithm on  $N$  samples defined as follows: randomly sample (without replacement)  $n_{1}(\varepsilon, \delta')$  out of  $N$  samples, then run  $\mathbb{A}_{0}$  on this subsample. By the amplification-by-subsampling theorem (Theorem 7), we have that  $\mathbb{A}$  is  $(\varepsilon'', \delta')$ -item-level DP. It is also obvious that the output distribution of this algorithm is the same as that of  $\mathbb{A}_{0}$  on  $n_{1}(\varepsilon, \delta')$  examples, and this algorithm is thus also  $\gamma$ -useful for  $\mathfrak{T}$ . Thus, we have  $n_{1}(\varepsilon'', \delta') \leq N \leq n_{1}(\varepsilon, \delta') \cdot O(\varepsilon / \varepsilon'')$  as desired.
C.3 Proof of Lemma 14
C.3.1 "Robust" McDiarmid Inequality for Almost-Multiplicative-Lipschitz function
We derive the following concentration inequality, which may be viewed as a strengthened McDiarmid's inequality (Lemma 26) for the multiplicative-Lipschitz case. To state this, let us define an additional notation: for  $\pmb{x} \in \mathcal{Z}^n$ ,  $\tilde{x}_i \in \mathcal{Z}$ , we write  $\tilde{\pmb{x}}^{(i)}$  to denote  $\pmb{x}$  but with  $x_{i}$  replaced by  $\tilde{x}_i$ .
Lemma 32. Let  $f: \mathcal{Z}^n \to \mathbb{R}_{\geq 0}$  be any function and  $\mathcal{D}$  any distribution on  $\mathcal{Z}$ . Assume that  $\varepsilon, \delta > 0$ , and  $\varepsilon \sqrt{n \log(1 / \delta)} > 0$  are sufficiently small. Then, for  $\varepsilon' = O(\varepsilon \sqrt{n \log(1 / \delta)})$ , we have
$$
\mathbb {E} _ {\boldsymbol {x}, \boldsymbol {x} ^ {\prime} \sim \mathcal {D} ^ {n}} \left[ \left[ f (\boldsymbol {x}) - e ^ {\varepsilon^ {\prime}} \cdot f (\boldsymbol {x} ^ {\prime}) \right] _ {+} \right] \leq O \left(\mu_ {\mathcal {D} ^ {n}} (f) \cdot \delta\right) + O \left(\sum_ {i = 1} ^ {n} \mathbb {E} _ {\boldsymbol {x}, \tilde {\boldsymbol {x}} \sim \mathcal {D} ^ {n}} \left[ f (\boldsymbol {x}) - e ^ {\varepsilon} \cdot f (\tilde {\boldsymbol {x}} ^ {(i)}) \right] _ {+}\right).
$$
When the second term in the RHS (i.e.,  $O\left(\sum_{i=1}^{n}\mathbb{E}_{\boldsymbol{x},\tilde{\boldsymbol{x}}\sim \mathcal{D}^n}\left[f(\boldsymbol{x}) - e^\varepsilon \cdot f(\tilde{\boldsymbol{x}}^{(i)})\right]_+\right)$ ) is zero, our lemma is (essentially) the same as McDiarmid's inequality for multiplicative-Lipschitz functions, as proved, e.g., in [CLN+16]. However, our bound is more robust, in the sense that it can handle the case where the multiplicative-Lipschitzness condition fails sometimes; this is crucial for the proof of Lemma 14 as we only assume approximate-DP.
We note that, while there are (additive) McDiarmid's inequalities that does not require Lipschitzness everywhere [Kut02, War16], we are not aware of any version that works for us due to the regime of parameters we are in and the assumptions we have. This is why we prove Lemma 32 from first principles. The remainder of this subsection is devoted to the proof of Lemma 32.
Scalar Random Variable Adjustment for Bounded Ratio. For the rest of this section, we write  $z \approx_{\varepsilon} z'$  where  $z, z' \in \mathbb{R}_{\geq 0}$  to indicate that  $z \in [e^{-\varepsilon}z', e^{\varepsilon}z']$ . We extend the notion similarly to  $\mathbb{R}_{\geq 0}$ -valued random variables: for a  $\mathbb{R}_{\geq 0}$ -valued random variable  $Z$  and  $z \in \mathbb{R}_{\geq 0}$ , we write  $Z \approx_{\varepsilon} z'$  if  $z \approx_{\varepsilon} z'$  for each  $z \in \operatorname{supp}(Z)$ . Furthermore, for (possibly dependent)  $\mathbb{R}_{\geq 0}$ -valued random variables  $Z, Z'$ , we write  $Z \approx_{\varepsilon} Z'$  to denote  $z \approx_{\varepsilon} z'$  for all  $(z, z') \in \operatorname{supp}((Z, Z'))$ .
Suppose  $Z$  is a  $\mathbb{R}_{\geq 0}$ -valued random variable such that any two values  $z, \tilde{z} \in \mathrm{supp}(Z)$  satisfies  $z \approx_{\varepsilon} \tilde{z}$ . Then, we also immediately have that their mean  $\mu$  is in this range and thus  $Z \approx_{\varepsilon} \mu$ . We start by showing a "robust" version of this statement, which asserts that, even if the former condition fails sometimes, we can still "move" the random variable a little bit so that the second condition holds (albeit with weaker  $2\varepsilon$  bound). The exact statement is presented below.
Lemma 33. Let  $\varepsilon, \mu^{*} \geq 0$ . Let  $Z$  be any  $\mathbb{R}_{\geq 0}$ -valued random variable and  $\mu \coloneqq \mathbb{E}[Z]$ . Then, there exists a random variable  $Z'$ , which is a post-processing of  $Z$ , such that
(i)  $Z^{\prime}\approx_{2\varepsilon}\mu^{*}$
(ii)  $\mathbb{E}[Z'] = \mu^*$ .
(iii)  $\mathbb{E}_{(Z,Z^{\prime})}[|Z - Z^{\prime}|\leq |\mu -\mu^{*}| + 2(1 + e^{-\varepsilon})\cdot \mathbb{E}_{Z,\tilde{Z}}\left[\left[Z - e^{\varepsilon}\cdot \tilde{Z}\right]_{+}\right],$  where  $(Z,Z^{\prime})$  is the canonical coupling between  $Z,Z^{\prime}$  (from post-processing) and  $\tilde{Z}$  is an i.i.d. copy of  $Z$
Proof. Let  $\ell = e^{-\varepsilon} \cdot \mu, r = e^{\varepsilon} \cdot \mu$ . We start by defining  $\hat{Z} \coloneqq \operatorname{clip}_{\ell, r}(Z)$ . This also gives a canonical coupling between  $\hat{Z}$  and  $Z$ , which then yields
$$
\begin{array}{l} \mathbb {E} _ {(Z, \hat {Z})} [ | Z - \hat {Z} | ] = \mathbb {E} _ {Z} \left[ | Z - \operatorname {c l i p} _ {\ell , r} (Z) | \right] \\ = \mathbb {E} _ {Z} \left[ [ Z - r ] _ {+} + [ \ell - Z ] _ {+} \right] \\ = \mathbb {E} _ {Z} \left[ [ Z - e ^ {\varepsilon} \cdot \mu ] _ {+} \right] + \mathbb {E} _ {Z} \left[ \left[ e ^ {- \varepsilon} \cdot \mu - Z \right] _ {+} \right] \\ \end{array}
$$
$$
\begin{array}{l} = \mathbb {E} _ {Z} \left[ \left[ Z - e ^ {\varepsilon} \cdot \mathbb {E} _ {\tilde {Z}} [ \tilde {Z} ] \right] _ {+} \right] + \mathbb {E} _ {Z} \left[ \left[ e ^ {- \varepsilon} \cdot \mathbb {E} _ {\tilde {Z}} [ \tilde {Z} ] - Z \right] _ {+} \right] \\ \leq \mathbb {E} _ {Z, \tilde {Z}} \left[ \left[ Z - e ^ {\varepsilon} \cdot \tilde {Z} \right] _ {+} \right] + \mathbb {E} _ {Z, \tilde {Z}} \left[ \left[ e ^ {- \varepsilon} \cdot \tilde {Z} - Z \right] _ {+} \right] \\ = \left(1 + e ^ {- \varepsilon}\right) \cdot \mathbb {E} _ {Z, \tilde {Z}} \left[ \left[ Z - e ^ {\varepsilon} \tilde {Z} \right] _ {+} \right], \tag {2} \\ \end{array}
$$
where the inequality follows from the convexity of  $[\cdot ]_{+}$
Define  $\hat{\mu} \coloneqq \mathbb{E}_{\hat{Z}}[\hat{Z}]$ . Note that we have
$$
| \hat {\mu} - \mu | \leq \mathbb {E} _ {(Z, \hat {Z})} [ | Z - \hat {Z} | ]. \tag {3}
$$
Next, consider two cases based on  $\mu ,\mu^{*}$
- Case I:  $\mu > e^{\varepsilon} \cdot \mu^{*}$  or  $\mu < e^{-\varepsilon} \cdot \mu^{*}$ . We assume w.l.o.g. that  $\mu > e^{\varepsilon} \cdot \mu^{*}$ ; the case  $\mu < e^{-\varepsilon} \cdot \mu^{*}$  can be argued similarly. In this case, simply set  $Z' = \mu^{*}$  always. Obviously, items (i) and (ii) hold. Moreover, we have