SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kobbi Nissim
Department of Computer Science
Georgetown University
kobbi.nissim@georgetown.edu
Uri Stemmer
Blavatnik School of Computer Science
u@uri.co.il
Eliad Tsfadia
eliadtsfadia@gmail.com
In adaptive data analysis, a mechanism gets  $n$  i.i.d. samples from an unknown distribution  $\mathcal{D}$ , and is required to provide accurate estimations to a sequence of adaptively chosen statistical queries with respect to  $\mathcal{D}$ . Hardt and Ullman [2014] and Steinke and Ullman [2015a] showed that, in general, it is computationally hard to answer more than  $\Theta(n^2)$  adaptive queries, assuming the existence of one-way functions.
However, these negative results strongly rely on an adversarial model that significantly advantages the adversarial analyst over the mechanism, as the analyst, who chooses the adaptive queries, also chooses the underlying distribution  $\mathcal{D}$ . This imbalance raises questions with respect to the applicability of the obtained hardness results – an analyst who has complete knowledge of the underlying distribution  $\mathcal{D}$  would have little need, if at all, to issue statistical queries to a mechanism which only holds a finite number of samples from  $\mathcal{D}$ .
We consider more restricted adversaries, called balanced, where each such adversary consists of two separate algorithms: The sampler who is the entity that chooses the distribution and provides the samples to the mechanism, and the analyst who chooses the adaptive queries, but has no prior knowledge of the underlying distribution (and hence has no a priori advantage with respect to the mechanism).
We improve the quality of previous lower bounds by revisiting them using an efficient balanced adversary, under standard public-key cryptography assumptions. We show that these stronger hardness assumptions are unavoidable in the sense that any computationally bounded balanced adversary that has the structure of all known attacks, implies the existence of public-key cryptography.
Statistical validity is a widely recognized crucial feature of modern science. Lack of validity – popularly known as the replication crisis in science poses a serious threat to the scientific process and also to the public's trust in scientific findings.
One of the factors leading to the replication crisis is the inherent adaptivity in the data analysis process. To illustrate adaptivity and its effect, consider a data analyst who is testing a specific research
hypothesis. The analyst gathers data, evaluates the hypothesis empirically, and often finds that their hypothesis is not supported by the data, leading to the formulation and testing of more hypotheses. If these hypotheses are tested and formed based on the same data (as acquiring fresh data is often expensive or even impossible), then the process is of adaptive data analysis (ADA) because the choice of hypotheses depends on the data. However, ADA no longer aligns with classical statistical theory, which assumes that hypotheses are selected independently of the data (and preferably before gathering data). ADA may lead to overfitting and hence false discoveries.
Statistical validity under ADA is a fundamental problem in statistics, that has received only partial answers. A recent line of work, initiated by Dwork et al. [2015c] and includes [Hardt and Ullman, 2014, Dwork et al., 2015a,b, Steinke and Ullman, 2015a,b, Bassily et al., 2016, Rogers et al., 2016, Russo and Zou, 2016, Smith, 2017, Feldman and Steinke, 2017, Nissim et al., 2018, Feldman and Steinke, 2018, Shenfeld and Ligett, 2019, Jung et al., 2020, Fish et al., 2020, Dagan and Kur, 2022, Kontorovich et al., 2022, Dinur et al., 2023, Blanc, 2023] has resulted in new insights into ADA and robust paradigms for guaranteeing statistical validity in ADA. A major objective of this line of work is to design optimal mechanisms M that initially obtain a dataset  $S$  containing  $n$  i.i.d. samples from an unknown distribution  $\mathcal{D}$ , and then answers adaptively chosen queries with respect to  $\mathcal{D}$ . Importantly, all of M's answers must be accurate with respect to the underlying distribution  $\mathcal{D}$ , not just w.r.t. the empirical dataset  $S$ . The main question is how to design an efficient mechanism that provides accurate estimations to adaptively chosen statistical queries, where the goal is to maximize the number of queries M can answer. This objective is achieved by providing both upper- and lower-bound constructions, where the lower-bound constructions demonstrate how an adversarial analyst making a small number of queries to an arbitrary M can invariably force M to err. The setting for these lower-bound proofs is formalized as a two-player game between a mechanism M and an adversary A as in Game 1.1.
Game 1.1 (ADA game between a mechanism M and an adversarial analyst A).
- M gets a dataset  $S$  of  $n$  i.i.d. samples from an unknown distribution  $\mathcal{D}$  over  $\mathcal{X}$ .
- For  $i = 1, \dots, \ell$ :
- A sends a query  $q_{i} \colon \mathcal{X} \mapsto [-1, 1]$  to  $\mathsf{M}$ .
-  $\mathsf{M}$  sends an answer  $y_{i} \in [-1, 1]$  to  $\mathsf{A}$ .
(As A and M are stateful,  $q_{i}$  and  $y_{i}$  may depend on the previous messages.)
M fails if  $\exists i\in [\ell ]$  s.t.  $|y_{i} - \mathrm{E}_{x\sim \mathcal{D}}[q_{i}(x)]| > 1 / 10.$
A question that immediately arises from the description of Game 1.1 is to whom should the distribution  $\mathcal{D}$  be unknown, and how to formalize this lack of knowledge. Ideally, the mechanism M should succeed with high probability for every unknown distribution  $\mathcal{D}$  and against any adversary A.
In prior work, this property was captured by letting the adversary choose the distribution  $\mathcal{D}$  at the outset of Game 1.1. Namely, the adversary A can be seen as a pair of algorithms  $(\mathsf{A}_1,\mathsf{A}_2)$ , where  $\mathsf{A}_1$  chooses the distribution  $\mathcal{D}$  and sends a state st to  $\mathsf{A}_2$  (which may contain the entire view of  $\mathsf{A}_1$ ), and after that, M and  $\mathsf{A}_2(\mathrm{st})$  interacts in Game 1.1. In this adversarial model, Hardt and Ullman [2014] and Steinke and Ullman [2015a] showed that, assuming the existence of one way functions, it is computationally hard to answer more than  $\Theta(n^2)$  adaptive queries. These results match the state-of-the-art constructions [Dwork et al., 2015c,a,b, Steinke and Ullman, 2015b, Bassily et al., 2016, Feldman and Steinke, 2017, 2018, Dagan and Kur, 2022, Blanc, 2023]. In fact, each such negative result was obtained by constructing a single adversary A that fails all efficient mechanisms. This means that, in general, it is computationally hard to answer more than  $\Theta(n^2)$  adaptive queries even when the analyst's algorithm is known to the mechanism. On the other hand, in each of these negative results, the adversarial analyst has a significant advantage over the mechanism - their ability to select the distribution  $\mathcal{D}$ . This allows the analyst to inject random trapdoors in  $\mathcal{D}$  (e.g., keys of an encryption scheme) which are then used in forcing a computationally limited mechanism to fail, as the mechanism does not get a hold of the trapdoor information.
<sup>1</sup>Here is an example of a mechanism that handles  $\tilde{\Theta}(n^2)$  adaptive queries using differential privacy: Given a query  $q_i$ , the mechanism returns an answer  $y_i = \frac{1}{n} \sum_{x \in S} x + \nu_i$  where the  $\nu_i$ 's are independent Gaussian noises, each with standard deviation of  $\tilde{O}(\sqrt{\ell}/n)$ . The noises guarantee that the entire process is "private enough" for avoiding overfitting in the ADA game, and accuracy is obtained whenever  $\ell = \tilde{O}(n^2)$ .
For most applications, the above adversarial model seems to be too strong. For instance, a data analyst who is testing research hypotheses usually has no knowledge advantage about the distribution that the mechanism does not have. In this typical setting, even if the underlying distribution  $\mathcal{D}$  happens to have a trapdoor, if the analyst recovers the trapdoor then the mechanism should also be able to recover it and hence disable its adversarial usage.
In light of this observation, we could hope that in a balanced setting, where the underlying distribution is unknown to both the mechanism and the analyst, it would always be possible for  $\mathsf{M}$  to answer more than  $O(n^{2})$  adaptive queries. To explore this possibility, we introduce what we call a balanced adversarial model.
Definition 1.2 (Balanced Adversary). A balanced adversary  $\mathsf{A}$  consists of two isolated algorithms: The sampler  $\mathsf{A}_1$ , which chooses a distribution  $\mathcal{D}$  and provides i.i.d. samples to the mechanism  $\mathsf{M}$ , and the analyst  $\mathsf{A}_2$ , which asks the adaptive queries. No information is transferred from  $\mathsf{A}_1$  to  $\mathsf{A}_2$ . See Game 1.3.
Game 1.3 (The ADA game between a mechanism  $M$  and a balanced adversary  $A = (A_1, A_2)$ ).
-  $\mathsf{A}_1$  chooses a distribution  $\mathcal{D}$  over  $\mathcal{X}$  (specified by a sampling algorithm) and provides  $n$  i.i.d. samples  $\mathcal{S}$  to  $\mathsf{M}$  (by applying the sampling algorithm  $n$  times).  
 $/^* \mathsf{A}_1$  does not provide  $\mathsf{A}_2$  with any information  $/^*$
-  $\mathsf{M}$  and  $\mathsf{A}_2$  play Game 1.1 (with respect to  $\mathcal{D}$  and  $\mathcal{S}$ ).
-  $\mathsf{M}$  fails if and only if it fails in Game 1.1.
Note that the difference between the balanced model and the previous (imbalanced) one is whether  $\mathsf{A}_1$  can send a state to  $\mathsf{A}_2$  after choosing the distribution  $\mathcal{D}$  (in the imbalanced model it is allowed, in contrast to the balanced model).<sup>2</sup>
We remark that the main advantage of the balanced model comes when considering a publicly known sampler  $\mathsf{A}_1$  (as we do throughout this work). This way,  $\mathsf{A}_1$  captures the common knowledge that both the mechanism  $\mathsf{M}$  and the analyst  $\mathsf{A}_2$  have about the underlying distribution  $\mathcal{D}$ .
Question 1.4. Do the lower-bounds proved in prior work hold also for balanced adversaries?
In this work we answer Question 1.4 in the positive. We do that using a publicly known analyst  $A_2$  (which even makes it stronger than what is required for a lower bound). I.e., even though the sampler  $A_1$  and the analyst  $A_2$  are publicly known and cannot communicate with each other, they fail any computationally bounded mechanism. However, our lower-bound is based on stronger hardness assumptions than in prior work, namely, we use public-key cryptography.
1.1 Our Results
Our first result is a construction of a balanced adversary forcing any computationally bounded mechanism to fail in Game 1.3.
Theorem 1.5 (Computational lower bound, informal). There exists an efficient balanced adversary  $\mathsf{A} = (\mathsf{A}_1,\mathsf{A}_2)$  that fails any computationally bounded mechanism  $\mathsf{M}$  using  $\Theta (n^{2})$  adaptive queries. Moreover, it does so by choosing a distribution over a small domain.
Our construction in Theorem 1.5 uses the structure of previous attacks of Hardt and Ullman [2014] and Steinke and Ullman [2015a], but relies on a stronger hardness assumption of public-key cryptography. We prove that this is unavoidable.
Theorem 1.6 (The necessity of public-key cryptography for proving Theorem 1.5, informal). Any computationally bounded balanced adversary that follows the structure of all currently known attacks, implies the existence of public-key cryptography (in particular, a key-agreement protocol).
In Section 1.3 we provide proof sketches of Theorems 1.5 and 1.6, where the formal statements appear in Sections 3 and 4 (respectively) and the formal proofs appear in the supplementary material.
An additional (minor) difference is that we chose in our model to let  $A_1$  also provide the i.i.d. samples to  $M$ . This is only useful for Theorem 1.6 as we need there that choosing  $\mathcal{D}$  and sampling from  $\mathcal{D}$  are both computationally efficient (which are simply captured by saying that  $A_1$  is computationally bounded).
Potential Consequences for the Information Theoretic Setting. Theorem 1.6 has immediate implication to the information theoretic setting, and allow for some optimism regarding the possibility of constructing an inefficient mechanism that answers many adaptive queries.
It is known that an inefficient mechanism can answer exponentially many adaptive queries, but such results have a strong dependency on the domain size. For instance, the Private Multiplicative Weights algorithm of Hardt and Rothblum [2010] can answer  $2^{\tilde{O}\left(n / \sqrt{\log|\mathcal{X}|}\right)}$  adaptive queries accurately. However, this result is not useful whenever  $n \leq O(\sqrt{\log|\mathcal{X}|})$ . Indeed, Steinke and Ullman [2015a] showed that this dependency is unavoidable in general, by showing that large domain can be used for constructing a similar, unconditional, adversary that fails any computationally unbounded mechanism after  $\Theta (n^{2})$  queries. Our Theorem 1.6 implies that such an attack cannot be implemented in the balanced setting, which gives the first evidence that there might be a separation between the computational and information theoretic setting under the balanced adversarial model (in contrast with the imbalanced model).
Corollary 1.7. There is no balanced adversary that follows the structure of all currently known attacks, and fails any (computationally unbounded) mechanism.
In order to see why Corollary 1.7 holds, suppose that we could implement such kind of attack using a balanced adversary. Then by Theorem 1.6, this would imply that we could construct an information-theoretic key agreement protocol (i.e., a protocol between two parties that agree on a key that is secret from the eyes of a computationally unbounded adversary that only sees the transcript of the execution). But since the latter does not exist, we conclude that such a balanced adversary does not exist either. In other words, we do not have a negative result that rules out the possibility of constructing an inefficient mechanism that can answer many adaptive queries of a balanced adversary, and we know that if a negative result exists, then by Theorem 1.6 it cannot follow the structure of Hardt and Ullman [2014], Steinke and Ullman [2015a].
1.2 Comparison with Elder [2016]
The criticism about the lower bounds of Hardt and Ullman [2014], Steinke and Ullman [2015a] is not new and prior work has attempted at addressing them with only partial success.
For example, Elder [2016] presented a similar "balanced" model (called "Bayesian ADA"), where both the analyst and the mechanism receive a prior  $\mathcal{P}$  which is a family of distributions, and then the distribution  $\mathcal{D}$  is drawn according to  $\mathcal{P}$  (unknown to both the mechanism and the analyst).
From an information theoretic point of view, this model is equivalent to ours when the sampler  $\mathsf{A}_1$  is publicly known, since  $\mathsf{A}_1$  simply defines a prior. But from a computational point of view, defining the sampling process (i.e., sampling  $\mathcal{D}$  and the i.i.d. samples from it) in an algorithmic way is better when we would like to focus on computationally bounded samplers.
Elder [2016] only focused on the information-theoretic setting. His main result is that a certain family of mechanisms (ones that only use the posterior means) cannot answer more than  $\tilde{O}(n^4)$  adaptive queries. This, however, does not hold for any mechanism's strategy. In particular, it does not apply to general computationally efficient mechanisms. Our negative result is quantitatively stronger ( $n^2$  vs  $n^4$ ) and it applies for all computationally efficient mechanisms. $^3$
Table 1 summarizes the comparison between Theorem 1.5 and the prior lower bounds (ignoring computational hardness assumptions).
1.3 Techniques
We follow a similar technique to that used in Hardt and Ullman [2014] and Steinke and Ullman [2015a], i.e., a reduction to a restricted set of mechanisms, called natural.
Definition 1.8 (Natural mechanism [Hardt and Ullman, 2014]). A mechanism  $\mathsf{M}$  is natural if, when given a sample  $\mathcal{S} = (x_1,\ldots ,x_n)\in \mathcal{X}^n$  and a query  $q\colon \mathcal{X}\to [-1,1]$ ,  $\mathsf{M}$  returns an answer that is a function solely of  $(q(x_{1}),\dots,q(x_{n}))$ . In particular,  $\mathsf{M}$  does not evaluate  $q$  on other data points of its choice.
<sup>3</sup>Our result is not directly comparable to that of Elder [2016], because our negative result does not say anything for non-efficient mechanisms, while his result does rule out a certain family of non-efficient mechanisms.
Balanced?	Class of Mechanisms	# of Queries	Dimension (log|X|)
Steinke and Ullman [2015a]	No	PPT Algorithms	O(n2)	n0(1)
Steinke and Ullman [2015a]	No	All	O(n2)	O(n2)
Elder [2016]	Yes	Certain Family	O(n4)	O(n4)
Theorem 1.5	Yes	PPT Algorithms	O(n2)	n0(1)
Table 1: Comparison between the lower bounds for adaptive data analysis.
Hardt and Ullman [2014] and Steinke and Ullman [2015a] showed that there exists an adversarial analyst  $\widetilde{\mathsf{A}}$  that fails any natural mechanism M, even when M is computationally unbounded, and even when  $\mathcal{D}$  is chosen to be the uniform distribution over  $\{1,2,\dots,m = 2000n\}$  (i.e.,  $\mathcal{D}$  is known to everyone). While general mechanisms could simply use the knowledge of the distribution to answer any query, natural mechanisms are more restricted, and can only provide answers based on the  $n$ -size dataset  $\mathcal{S}$  that they get. The restriction to natural mechanisms allowed Steinke and Ullman [2015a] to use interactive fingerprinting codes, which enable to reveal  $\mathcal{S}$  using  $\Theta(n^2)$  adaptive queries when the answers are accurate and correlated with  $\mathcal{S}$ .
To construct an attacker A that fails any computationally bounded mechanism (and not just natural mechanisms), prior work forced the mechanism to behave naturally by using a private-key encryption scheme. More specifically, the adversary first samples  $m$  secret keys  $\mathsf{sk}_1, \ldots, \mathsf{sk}_m$ , and then defines  $\mathcal{D}$  to be the uniform distribution over the pairs  $\{(j, \mathsf{sk}_j)\}_{j=1}^m$ . The adversary then simulates an adversary  $\widetilde{\mathsf{A}}$  which fails natural mechanisms as follows: a query  $\tilde{q} \colon [m] \to [-1,1]$  issued by  $\widetilde{\mathsf{A}}$  is translated by A to a set of  $m$  encryptions  $\{\mathsf{ct}_j\}_{j=1}^m$  where each  $\mathsf{ct}_j$  is an encryption of  $\tilde{q}(j)$  under the key  $\mathsf{sk}_j$ . These encryptions define a new query  $q$  that on input  $(j, \mathsf{sk})$ , outputs the decryption of  $\mathsf{ct}_j$  under the key sk. However, since M is computationally bounded and has only the secret keys that are part of its dataset  $S$ , it can only decrypt the values of  $\tilde{q}$  on points in  $S$ , yielding that it effectively behaves naturally.
Note that the above attack A is imbalanced as it injects the secret keys  $\mathsf{sk}_1,\ldots ,\mathsf{sk}_m$  into  $\mathcal{D}$  and then uses these keys when it forms queries. In other words, even though the attacker A is known to the mechanism, A is able to fail M by creating a secret correlation between its random coins and the distribution  $\mathcal{D}$ .
1.3.1 Balanced Adversary via Identity Based Encryption Scheme
For proving Theorem 1.5, we replace the private-key encryption scheme with a public-key primitive called identity-based encryption (IBE) scheme [Shamir, 1984, Cocks, 2001, Boneh and Franklin, 2001]. Such a scheme enables to produce  $m$  secret keys  $\mathsf{sk}_1, \ldots, \mathsf{sk}_m$  along with a master public key  $\mathsf{mpk}$ . Encrypting a message to a specific identity  $j \in [m]$  only requires  $\mathsf{mpk}$ , but decrypting a message for identity  $j$  must be done using its secret key  $\mathsf{sk}_j$ . Using an IBE scheme we can achieve a reduction to natural mechanisms via a balanced adversary  $\mathsf{A} = (\mathsf{A}_1, \mathsf{A}_2)$  as follows:  $\mathsf{A}_1$  samples keys  $\mathsf{mpk}, \mathsf{sk}_1, \ldots, \mathsf{sk}_m$  according to the IBE scheme, and defines  $\mathcal{D}$  to be the uniform distribution over the triplets  $\{(j, \mathsf{mpk}, \mathsf{sk}_j)\}_{j=1}^m$ . The analyst  $\mathsf{A}_2$ , which does not know the keys, first asks queries of the form  $q(j, \mathsf{mpk}, \mathsf{sk}) = \mathsf{mpk}_k$  for every bit  $k$  of  $\mathsf{mpk}$  in order to reveal it. Then, it follows a strategy as in the previous section, i.e., it simulates an adversary  $\widetilde{\mathsf{A}}$  which foils natural mechanisms by translating each query query  $\tilde{q} : [m] \to [-1,1]$  issued by  $\widetilde{\mathsf{A}}$  by encrypting each  $\tilde{q}(j)$  for identity  $j$  using  $\mathsf{mpk}$ . Namely, the IBE scheme allowed the analyst to implement the attack of Hardt and Ullman [2014] and Steinke and Ullman [2015a], but without having to know the secret keys  $\mathsf{sk}_1, \ldots, \mathsf{sk}_m$ .
We can implement the IBE scheme using a standard public-key encryption scheme: in the sampling process, we sample  $m$  independent pairs of public and secret keys  $\{(\mathsf{pk}_j, \mathsf{sk}_j)\}_{j=1}^m$  of the original scheme, and define  $\mathsf{mpk} = (\mathsf{pk}_1, \ldots, \mathsf{pk}_m)$ . When encrypting a message for identity  $j \in [m]$ , we could simply encrypt it using  $\mathsf{pk}_j$  (part of  $\mathsf{mpk}$ ), which can only be decrypted using  $\mathsf{sk}_j$ . The disadvantage of this approach is the large master public key  $\mathsf{mpk}$  that it induces. Applying the encryption scheme with security parameter of  $\lambda$ , the master key  $\mathsf{mpk}$  will be of size  $\lambda \cdot m$  and not just  $\lambda$  as the sizes of the secret keys. This means that implementing our balanced adversary with such an encryption scheme would result in a distribution over a large domain  $\mathcal{X}$ , which would not rule out the possibility to construct a mechanism for distributions over smaller domains. Yet, Döttling and
Garg [2021] showed that it is possible to construct a fully secure IBE scheme using a small  $mpk$  of size only  $O(\lambda \cdot \log m)$  under standard hardness assumptions (e.g., the Computational Diffie Helman problem [Diffie and Hellman, 1976] $^4$  or the hardness of factoring).
1.3.2 Key-Agreement Protocol via Balanced Adversary
In order to prove Theorem 1.6, we first explain what type of adversaries the theorem applies to. Recall that in all known attacks (including ours), the adversary A wraps a simpler adversary  $\widetilde{\mathsf{A}}$  that fails natural mechanisms. In particular, the wrapper A has two key properties:
1. A knows  $\operatorname{E}_{x\sim \mathcal{D}}[q_{\ell}(x)]$  for the last query  $q_{\ell}$  that it asks (becausethequivalent to  $\frac{1}{m}\sum_{j = 1}^{m}\tilde{q}_{\ell}(j)$  where  $\tilde{q}_{\ell}$  is the wrapped query which is part of A's view), and
2. If the mechanism attempts to behave accurately in the first  $\ell - 1$  rounds (e.g., it answers the empirical mean  $\frac{1}{n} \sum_{x \in S} q(x)$  for every query  $q$ ), then A, as a wrapper of A, will be able to ask a last query  $q_\ell$  that would fail any computationally bounded last-round strategy for the mechanism.
We next show that any computationally bounded balanced adversary A that has the above two properties, can be used for constructing a key-agreement protocol. That is, a protocol between two computationally bounded parties  $\mathsf{P}_1$  and  $\mathsf{P}_2$  that enable them to agree on a value which cannot be revealed by a computationally bounded adversary who only sees the transcript of the execution. See Protocol 1.9.
Protocol 1.9 (Key-Agreement Protocol  $(\mathsf{P}_1,\mathsf{P}_2)$  via a balanced adversary  $\mathsf{A} = (\mathsf{A}_1,\mathsf{A}_2)$ ).
Input:  $1^n$ . Let  $\ell = \ell(n)$  and  $\mathcal{X} = \mathcal{X}(n)$  be the number queries and the domain that is used by the adversary A.
Operation: