SlowGuess/NeurIPS_2023_OCR huggingface dataset:
-  $\mathsf{P}_1$  emulates  $\mathsf{A}_1$  on input  $n$  for obtaining a distribution  $\mathcal{D}$  over  $\mathcal{X}$  (specified by a sampling procedure), and samples  $n$  i.i.d. samples  $S$ .
-  $\mathsf{P}_2$  initializes an emulation of  $\mathsf{A}_2$  on input  $n$ .
- For  $i = 1$  to  $\ell$ :
1.  $\mathsf{P}_2$  receives the  $i^{\mathrm{th}}$  query  $q_{i}$  from the emulated  $\mathsf{A}_2$  and sends it to  $\mathsf{P}_1$ .
2.  $\mathsf{P}_1$  computes  $y_{i} = \frac{1}{n}\sum_{x\in S}q_{i}(x)$ , and sends it to  $\mathsf{P}_2$ .
3.  $\mathsf{P}_2$  sends  $y_{i}$  as the  $i^{\mathrm{th}}$  answer to the emulated  $\mathsf{A}_2$ .
-  $\mathsf{P}_1$  and  $\mathsf{P}_2$  agree on  $\operatorname{E}_{x\sim \mathcal{D}}[q_{\ell}(x)]$ .
The agreement of Protocol 1.9 relies on the ability of  $\mathsf{P}_1$  and  $\mathsf{P}_2$  to compute  $\mathrm{E}_{x\sim \mathcal{D}}[q_{\ell}(x)]$ . Indeed,  $\mathsf{P}_1$  can accurately estimate it using the access to the sampling procedure, and  $\mathsf{P}_2$  can compute it based on the view of the analyst  $\mathsf{A}_2$  (follows by Property 1).
To prove the secrecy guarantee of Protocol 1.9, assume towards a contradiction that there exists a computationally bounded adversary  $G$  that given the transcript of the execution, can reveal  $\mathrm{E}_{x \sim \mathcal{D}}[q_{\ell}(x)]$ . Now consider the following mechanism for the ADA game: In the first  $\ell - 1$  queries, answer the empirical mean, but in the last query, apply  $G$  on the transcript and answer its output. By the assumption on  $G$ , the mechanism will be able to accurately answer the last query, in contradiction to Property 2.
We note that Property 1 can be relaxed by only requiring that  $\mathsf{A}$  is able to provide a "good enough" estimation of  $\mathrm{E}_{x\sim \mathcal{D}}[q_{\ell}(x)]$ . Namely, as long as the estimation provided in Property 1 is better than the estimation that an adversary can obtain in Property 2 (we prove that an  $n^{\Omega (1)}$  multiplicative gap suffices), this would imply that Protocol 1.9 is a weak key-agreement protocol, which can be amplified to a fully secure one using standard techniques.
We also note that by requiring in Game 1.3 that  $\mathsf{A}_1$  samples from  $\mathcal{D}$  according to the sampling procedure, we implicitly assume here that sampling from  $\mathcal{D}$  can be done efficiently (because  $\mathsf{A}_1$  is
$^4$ CDH is hard with respect to a group  $\mathbb{G}$  of order  $p$ , if given a random generator  $g$  along with  $g^a$  and  $g^b$ , for uniformly random  $a, b \in [p]$ , as inputs, the probability that a PPT algorithm can compute  $g^{ab}$  is negligible.
assumed to be computationally bounded). Our reduction to key-agreement relies on this property, since if sampling from  $\mathcal{D}$  could not be done efficiently, then  $\mathsf{P}_1$  would not have been a computationally bounded algorithm.
1.4 Perspective of Public Key Cryptography
Over the years, cryptographic research has proposed solutions to many different cryptographic tasks under a growing number of (unproven) computational hardness assumptions. To some extent, this state of affairs is unavoidable, since the security of almost any cryptographic primitive implies the existence of one-way functions [Impagliazzo and Luby, 1989] (which in particular implies that  $P \neq NP$ ). Yet, all various assumptions can essentially be divided into two main types: private key cryptography and public key cryptography [Impagliazzo, 1995]. The former type is better understood: A series of works have shown that the unstructured form of hardness guaranteed by one-way functions is sufficient to construct many complex and structured primitives such as pseudorandom generators [Håstad et al., 1999], pseudorandom functions [Goldreich et al., 1986] and permutations [Luby and Rackoff, 1988], commitment schemes [Naor, 1991, Haitner et al., 2009], universal one-way hash functions [Rompel, 1990], zero-knowledge proofs [Goldreich et al., 1987], and more. However, reductions are much less common outside the one-way functions regime, particularly when constructing public-key primitives. In the famous work of Impagliazzo and Rudich [1989] they gave the first evidence that public key cryptography assumptions are strictly stronger than one-way functions, by showing that key-agreement, which enables two parties to exchange secret messages over open channels, cannot be constructed from one-way functions in a black-box way.
Our work shows that a balanced adversary for the ADA game that has the structure of all known attacks, is a primitive that belongs to the public-key cryptography type. In particular, if public-key cryptography does not exist, it could be possible to construct a computationally bounded mechanism that can handle more than  $\Theta(n^{2})$  adaptive queries of a balanced adversary (i.e., we currently do not have a negative result that rules out this possibility).
1.5 Other Related Work
Nissim et al. [2018] presented a variant of the lower bound of Steinke and Ullman [2015a] that aims to reduce the number of queries used by the attacker. However, their resulting lower bound only holds for a certain family of mechanisms, and it does not rule out all computationally efficient mechanisms.
Dinur et al. [2023] revisited and generalized the lower bounds of Hardt and Ullman [2014] and Steinke and Ullman [2015a] by showing that they are a consequence of a space bottleneck rather than a sampling bottleneck. Yet, as in the works by Hardt, Steinke, and Ullman, the attack by Dinur et al. relies on the ability to choose the underlying distribution  $\mathcal{D}$  and inject secret trapdoors in it, and hence it utilizes an imbalanced adversary.
Recently, lower bounds constructions for the ADA problem were used as a tool for constructing (conditional) lower bounds for other problems, such as the space complexity of adaptive streaming algorithms [Kaplan et al., 2021] and the time complexity of dynamic algorithms Beimel et al. [2022]. Our lower bound for the ADA problem is qualitatively stronger than previous lower bounds (as the adversary we construct has less power). Thus, our lower bound could potentially yield new connections and constructions in additional settings.
1.6 Conclusion and Open Problems
In this work we present the balanced adversarial model for the ADA problem, and show that the existence of a balanced adversary that has the structure of all previously known attacks is equivalent to the existence of public-key cryptography. Yet, we do not know what is the truth outside of the public-key cryptography world. Can we present a different type of efficient attack that is based on weaker hardness assumptions (like one-way functions)? Or is it possible to construct an efficient mechanism that answer more than  $\Theta(n^2)$  adaptive queries assuming that public-key cryptography does not exist? We also leave open similar questions regarding the information theoretic case. We currently do not know whether it is possible to construct an unbounded mechanism that answers exponential number of queries for any distribution  $\mathcal{D}$  (regardless of its domain size).
In a broader perspective, lower bounds such as ours show that no general solution exists for a problem. They often use unnatural inputs or distributions and rely on cryptographic assumptions. They are important as guidance for how to proceed with a problem, e.g., search for mechanisms that would succeed if the underlying distribution is from a "nice" family of distributions.
2.1 Notations
We use calligraphic letters to denote sets and distributions, uppercase for random variables, and lowercase for values and functions. For  $n \in \mathbb{N}$ , let  $[n] = \{1, 2, \dots, n\}$ . Let  $\mathrm{neg}(n)$  stand for a negligible function in  $n$ , i.e., a function  $\nu(n)$  such that for every constant  $c > 0$  and large enough  $n$  it holds that  $\nu(n) < n^{-c}$ . For  $n \in \mathbb{N}$  we denote by  $1^n$  the  $n$ -size string  $1 \dots 1$  ( $n$  times). Let PPT stand for probabilistic polynomial time. We say that a pair of algorithms  $A = (A_1, A_2)$  is PPT if both  $A_1$  and  $A_2$  are PPT algorithms.
2.2 Distributions and Random Variables
Given a distribution  $\mathcal{D}$ , we write  $x \sim \mathcal{D}$ , meaning that  $x$  is sampled according to  $\mathcal{D}$ . For a multiset  $\mathcal{S}$ , we denote by  $\mathcal{U}_{\mathcal{S}}$  the uniform distribution over  $\mathcal{S}$ , and let  $x \gets \mathcal{S}$  denote that  $x \sim \mathcal{U}_{\mathcal{S}}$ . For a distribution  $\mathcal{D}$  and a value  $n \in \mathbb{N}$ , we denote by  $\mathcal{D}^n$  the distribution of  $n$  i.i.d. samples from  $\mathcal{D}$ . For a distribution  $\mathcal{D}$  over  $\mathcal{X}$  and a query  $q: \mathcal{X} \to [-1, 1]$ , we abuse notation and denote  $q(\mathcal{D}) := \operatorname{E}_{x \sim \mathcal{D}}[q(x)]$ , and similarly for  $\mathcal{S} = (x_1, \ldots, x_n) \in \mathcal{X}^*$  we abuse notation and denote  $q(\mathcal{S}) := \operatorname{E}_{x \gets \mathcal{S}}[q(x)] = \frac{1}{n} \sum_{i=1}^{n} x_i$ .
2.3 Cryptographic Primitives
2.3.1 Key Agreement Protocols
The most basic public-key cryptographic primitive is a (1-bit) key-agreement protocol, defined below.
Definition 2.1 (key-agreement protocol). Let  $\pi$  be a two party protocol between two interactive PPT algorithms  $\mathsf{P}_1$  and  $\mathsf{P}_2$ , each outputs 1-bit. Let  $\pi(1^n)$  denote a random execution of the protocol on joint input  $1^n$  (the security parameter), and let  $O_n^1, O_n^2$  and  $T_n$  denote the random variables of  $\mathsf{P}_1$ 's output,  $\mathsf{P}_2$ 's output, and the transcript (respectively) in this execution. We say that  $\pi$  is an  $(\alpha, \beta)$ -key-agreement protocol if the following holds for any PPT ("eavesdropper") A and any  $n \in \mathbb{N}$ :
Agreement:  $\operatorname*{Pr}\left[O_n^1 = O_n^2\right] \geq \alpha(n)$ , and
Secrecy:  $\operatorname*{Pr}\big[\mathsf{A}(T_n) = O_n^1\big]\leq \beta (n).$
We say that  $\pi$  is a fully-secure key-agreement protocol if it is an  $(1 - \mathrm{neg}(n), 1/2 + \mathrm{neg}(n))$ -key-agreement protocol.
2.3.2 Identity-Based Encryption
An Identity-Based Encryption (IBE) scheme [Shamir, 1984, Cocks, 2001, Boneh and Franklin, 2001] consists of four PPT algorithms (Setup, KeyGen, Encrypt, Decrypt) defined as follows:
Setup(1): given the security parameter  $\lambda$ , it outputs a master public key  $mpk$  and a master secret key  $msk$ .
KeyGen(msk, id): given the master secret key msk and an identity id  $\in [n]$ , it outputs a decryption key skid.
Encrypt(mpk, id, m): given the master public key mpk, and identity id  $\in [n]$  and a message m, it outputs a ciphertext ct.
Decrypt  $(\mathsf{sk}_{\mathrm{id}},\mathsf{ct})$  : given a secret key  $\mathsf{sk}_{\mathrm{id}}$  for identity id and a ciphertext ct, it outputs a string m.
The following are the properties of such an encryption scheme:
Completeness: For all security parameter  $\lambda$ , identity  $\mathrm{id} \in [n]$  and a message  $\mathfrak{m}$ , with probability 1 over (mpk, msk)  $\sim$  Setup(1 $\lambda$ ) and  $\mathrm{sk}_{\mathrm{id}} \sim$  KeyGen(msk, id) it holds that
$$
\operatorname {D e c r o p t} \left(\operatorname {s k} _ {\text {i d}}, \operatorname {E n c r o p t} (\operatorname {m p k}, \text {i d}, \mathrm {m})\right) = \mathrm {m}
$$
Security: For any PPT adversary  $\mathsf{A} = (\mathsf{A}_1,\mathsf{A}_2)$  it holds that:
$$
\Pr \left[ I N D _ {\mathrm {A}} ^ {I B E} \left(1 ^ {\lambda}\right) = 1 \right] \leq 1 / 2 + \operatorname {n e g} (\lambda)
$$
where  $IND_{\mathsf{A}}^{IBE}$  is shown in Experiment 2.2.5
Experiment 2.2  $(IND_{\mathsf{A}}^{IBE}(1^{\lambda}))$
1. (mpk, msk)  $\sim$  Setup(1).
2.  $(\mathsf{id}^*, (\mathsf{m}_1^0, \ldots, \mathsf{m}_k^0), (\mathsf{m}_1^1, \ldots, \mathsf{m}_k^1), \mathsf{st}) \sim \mathsf{A}_1^{\mathsf{KeyGen}(\mathsf{msk}, \cdot)}(\mathsf{mpk})$  where  $|\mathsf{m}_i^0| = |\mathsf{m}_i^1|$  for every  $i \in [k]$  and for each query id by  $\mathsf{A}_1$  to  $\mathsf{KeyGen}(\mathsf{msk}, \cdot)$  we have that  $\mathsf{id} \neq \mathsf{id}^*$ .
3. Sample  $b \gets \{0,1\}$ .
4. Sample  $\mathsf{ct}_i^*\sim \mathsf{Encrypt}(\mathsf{mpk},\mathsf{id}^*,\mathsf{m}_i^b)$  for every  $i\in [k]$ .
5.  $b' \sim A_2^{KeyGen(msk, \cdot)}(mpk, (ct_1^*, \ldots, ct_k^*), st)$  where for each query id by  $A_2$  to  $KeyGen(msk, \cdot)$  we have that  $id \neq id^*$ .
6. Output 1 if  $b = b'$  and 0 otherwise.
Namely, the adversary chooses two sequences of messages  $(\mathfrak{m}_1^0,\dots ,\mathfrak{m}_k^0)$  and  $(\mathfrak{m}_1^1,\dots ,\mathfrak{m}_k^1)$ , and gets encryptions of either the first sequence or the second one, where the encryptions made for identity  $\mathrm{id}^*$  that the adversary does not hold its key (not allowed to query KeyGen on input  $\mathrm{id}^*$ ). The security requirement means that she cannot distinguish between the two cases (except with negligible probability).
Theorem 2.3 (Döttling and Garg [2021]). Assume that the Computational Diffie-Hellman (CDH) Problem is hard. Then there exists an IBE scheme  $\mathcal{E} =$  (Setup, KeyGen, Encrypt, Decrypt) for  $n$  identities such that given a security parameter  $\lambda$ , the master keys and each decryption key are of size  $O(\lambda \cdot \log n)$ .<sup>6</sup>
2.4 Balanced Adaptive Data Analysis
As described in the introduction, the mechanism plays a game with a balanced adversary that consists of two (isolated) algorithms: a sampler  $\mathsf{A}_1$ , which chooses a distribution  $\mathcal{D}$  over a domain  $\mathcal{X}$  and provides  $n$  i.i.d. samples to  $\mathsf{M}$ , and an analyst  $\mathsf{A}_2$ , which asks the adaptive queries about the distribution. Let  $\mathsf{ADA}_{n,\ell,\mathcal{X}}[\mathsf{M},\mathsf{A} = (\mathsf{A}_1,\mathsf{A}_2)]$  denote Game 1.3 on public inputs  $n$ -the number of samples,  $\ell$ -the number of queries, and  $\mathcal{X}$ -the domain. We denote by output 1 the case that  $\mathsf{M}$  fails in the game, and 0 otherwise. Since this work mainly deals with computationally bounded algorithms that we would like to model as PPT algorithms, we provide  $n$  and  $\ell$  in unary representation. We also assume for simplicity that  $\mathcal{X}$  is finite, which allows to represent each element as a binary vector of dimension  $\lceil \log |\mathcal{X}| \rceil$ , and we provide the dimension in unary representation as well.
All previous negative results (Hardt and Ullman [2014], Steinke and Ullman [2015a], and Dinur et al. [2023]) where achieved by reduction to a restricted family of mechanisms, called natural mechanisms (Definition 1.8). These are algorithms that can only evaluate the query on the sample points they are given.
For natural mechanisms (even unbounded ones), the following was proven.
Theorem 2.4 (Hardt and Ullman [2014], Steinke and Ullman [2015a]). There exists a pair of PPT algorithms  $\widetilde{\mathsf{A}} = (\widetilde{\mathsf{A}}_1, \widetilde{\mathsf{A}}_2)$  such that for every natural mechanism  $\widetilde{\mathsf{M}}$  and every large enough  $n$  and  $\ell = \Theta(n^2)$  it holds that
$$
\Pr \left[ \mathsf {A D A} _ {n, \ell , \mathcal {X} = [ 2 0 0 0 n ]} [ \widetilde {\mathsf {M}}, \widetilde {\mathsf {A}} ] = 1 \right] > 3 / 4. \tag {1}
$$
5The IBE security experiment is usually described as Experiment 2.2 with  $k = 1$  (i.e., encrypting a single message). Yet, it can be extended to any sequence of messages using a simple reduction.
The construction can also be based on the hardness of factoring.
In particular,  $\widetilde{\mathsf{A}}_1$  always chooses the uniform distribution over  $[2000n]$ , and  $\widetilde{\mathsf{A}}_2$  uses only queries over the range  $\{-1, 0, 1\}$ .
3 Constructing a Balanced Adversary via IBE
We prove that if an IBE scheme exists, then there is an efficient reduction to natural mechanisms that holds against any PPT mechanism, yielding a general lower bound for the computational case.
Theorem 3.1 (Restatement of Theorem 1.5). Assume the existence of an IBE scheme  $\mathcal{E}$  that supports  $m = 2000n$  identities with security parameter  $\lambda = \lambda(n)$  s.t.  $n \leq \mathrm{poly}(\lambda)$  (e.g.,  $\lambda = n^{0.1}$ ) using keys of length  $k = k(n)$ . Then there exists a PPT balanced adversary  $\mathsf{A} = (\mathsf{A}_1, \mathsf{A}_2)$  and  $\ell = \Theta(n^2) + k$  such that for every PPT mechanism  $\mathsf{M}$  it holds that
$$
\Pr \left[ \mathsf {A D A} _ {n, \ell , \mathcal {X} = [ m ] \times \{0, 1 \} ^ {2 k}} [ \mathsf {M}, \mathsf {A} ] = 1 \right] > 3 / 4 - \operatorname {n e g} (n).
$$
The proof of the theorem is given in the supplementary material. Note that we use a domain  $\mathcal{X}$  with  $\log |\mathcal{X}| = 2k + \log n + O(1)$ , and by applying Theorem 2.3, the lower bound holds for  $k = O(\lambda \cdot \log n)$  under the CDH hardness assumption.
4 Reduction to Natural Mechanisms Implies Key Agreement
We prove that any PPT balanced adversary  $\mathsf{A} = (\mathsf{A}_1, \mathsf{A}_2)$  that has the structure of all known lower bounds (Hardt and Ullman [2014], Steinke and Ullman [2015a], Dinur et al. [2023] and ours in Section 3), can be used to construct a key-agreement protocol.
All known constructions use an adversary  $\mathsf{A}$  that wraps the adversary  $\widetilde{\mathsf{A}}$  for the natural mechanisms case (Theorem 2.4) by forcing every mechanism  $\mathsf{M}$  to behave naturally using cryptography. In particular, they all satisfy Properties 1 and 2 from Section 1.3.2.
The formal statement is given in the following theorem. The proof is provided in the supplementary material.
Theorem 4.1 (Restatement of Theorem 1.6). Assume the existence of a PPT adversary  $\mathsf{A} = (\mathsf{A}_1, \mathsf{A}_2)$  and functions  $\ell = \ell(n) \leq \mathrm{poly}(n)$  and  $\mathcal{X} = \mathcal{X}(n)$  with  $\log |\mathcal{X}| \leq \mathrm{poly}(n)$  such that the following holds: Let  $n \in \mathbb{N}$  and consider a random execution of  $\mathsf{ADA}_{n,\ell,\mathcal{X}}[\mathsf{M}, \mathsf{A}]$  where  $\mathsf{M}$  is the mechanism that given a sample  $\mathcal{S}$  and a query  $q$ , answers the empirical mean  $q(S)$ . Let  $D_n$  and  $Q_n$  be the (r.v.'s of the) values of  $\mathcal{D}$  and  $q = q_{\ell}$  (the last query) in the execution (respectively), let  $T_n$  be the transcript of the execution between the analyst  $\mathsf{A}_2$  and the mechanism  $\mathsf{M}$  (i.e., the queries and answers), and let  $V_n$  be the view of  $\mathsf{A}_2$  at the end of the execution (without loss of generality, its input, random coins and the transcript). Assume that
1.  $\exists$  PPT algorithm F s.t.  $\forall n\in \mathbb{N}:\operatorname *{Pr}\left[\left|\mathsf{F}(V_n) - Q_n(D_n)\right|\leq n^{-1 / 10}\right]\geq 1 - \mathrm{neg}(n),$  and
2.  $\forall$  PPT algorithm  $G$  and  $\forall n\in \mathbb{N}$  ..  $\operatorname *{Pr}[|G(T_n) - Q_n(D_n)|\leq 1 / 10]\leq 1 / 4 + \mathrm{neg}(n).$
Then using A and F it is possible to construct a fully-secure key-agreement protocol.
Note that Assumption 1 in Theorem 4.1 formalizes the first property in which the analyst knows a good estimation of the true answer, and the PPT algorithm  $\mathsf{F}$  is the assumed knowledge extractor. Assumption 2 in Theorem 4.1 formalizes the second property which states that the mechanism, which answers the empirical mean along the way, will fail in the last query, no matter how it chooses to act (this behavior is captured with the PPT algorithm G), and moreover, it is enough to assume that this requirement only holds with respect to to transcript of the execution, and not with respect to the view of the mechanism.
We refer to the full version of the paper, given in the supplementary material, for all the missing proofs.
Kobbi Nissim is partially supported by NSF grant No. CNS-2001041 and a gift to Georgetown University. Uri Stemmer is partially supported by the Israel Science Foundation (grant 1871/19) and by Len Blavatnik and the Blavatnik Family foundation. Eliad Tsfadia is partially supported by the Fulbright Program and a gift to Georgetown University.
R. Bassily, K. Nissim, A. D. Smith, T. Steinke, U. Stemmer, and J. Ullman. Algorithmic stability for adaptive data analysis. In Proceedings of the 48th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2016, pages 1046-1059, 2016.
A. Beimel, H. Kaplan, Y. Mansour, K. Nissim, T. Saranurak, and U. Stemmer. Dynamic algorithms against an adaptive adversary: generic constructions and lower bounds. In STOC, pages 1671-1684. ACM, 2022.
G. Blanc. Subsampling suffices for adaptive data analysis. CoRR, abs/2302.08661, 2023.
D. Boneh and M. K. Franklin. Identity-based encryption from the weil pairing. In J. Kilian, editor, Advances in Cryptology - CRYPTO 2001, 21st Annual International Cryptology Conference, Proceedings, volume 2139, pages 213-229, 2001.
C. C. Cocks. An identity based encryption scheme based on quadratic residues. In Cryptography and Coding, 8th IMA International Conference, Proceedings, volume 2260, pages 360-363, 2001.
Y. Dagan and G. Kur. A bounded-noise mechanism for differential privacy. In Conference on Learning Theory, volume 178, pages 625-661. PMLR, 2022.
W. Diffie and M. E. Hellman. New directions in cryptography. IEEE Trans. Inf. Theory, 22(6): 644-654, 1976.
I. Dinur, U. Stemmer, D. P. Woodruff, and S. Zhou. On differential privacy and adaptive data analysis with bounded space. CoRR, abs/2302.05707, 2023.
N. Döttling and S. Garg. Identity-based encryption from the diffie-hellman assumption. J. ACM, 68 (3):14:1-14:46, 2021.
C. Dwork, V. Feldman, M. Hardt, T. Pitassi, O. Reingold, and A. Roth. Generalization in adaptive data analysis and holdout reuse. In Advances in Neural Information Processing Systems (NIPS), 2015a.
C. Dwork, V. Feldman, M. Hardt, T. Pitassi, O. Reingold, and A. Roth. The reusable holdout: Preserving validity in adaptive data analysis. Science, 349(6248):636-638, 2015b.
C. Dwork, V. Feldman, M. Hardt, T. Pitassi, O. Reingold, and A. L. Roth. Preserving statistical validity in adaptive data analysis. In STOC, pages 117-126. ACM, 2015c.
S. Elder. Challenges in bayesian adaptive data analysis. CoRR, abs/1604.02492, 2016. URL http://arxiv.org/abs/1604.02492.
V. Feldman and T. Steinke. Generalization for adaptively-chosen estimators via stable median. In Proceedings of the 30th Conference on Learning Theory, COLT 2017, volume 65, pages 728-757. PMLR, 2017.
V. Feldman and T. Steinke. Calibrating noise to variance in adaptive data analysis. In Conference On Learning Theory, COLT 2018, volume 75, pages 535-544. PMLR, 2018.
B. Fish, L. Reyzin, and B. I. P. Rubinstein. Sampling without compromising accuracy in adaptive data analysis. In Algorithmic Learning Theory, ALT 2020, volume 117, pages 297-318. PMLR, 2020.
O. Goldreich, S. Goldwasser, and S. Micali. How to construct random functions. Journal of the ACM, 33(4):792-807, 1986.
A. D. Smith. Information, privacy and stability in adaptive data analysis. CoRR, abs/1706.00820, 2017.
T. Steinke and J. Ullman. Interactive fingerprinting codes and the hardness of preventing false discovery. In  $COLT$ , pages 1588-1628, 2015a.
T. Steinke and J. Ullman. Between pure and approximate differential privacy. CoRR, abs/1501.06095, 2015b.