SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Moni Naor*
Kobbi Nissim†
Uri Stemmer†
Chao Yan $\S$
A private learner is trained on a sample of labeled points and generates a hypothesis that can be used for predicting the labels of newly sampled points while protecting the privacy of the training set [Kasiviswannathan et al., FOCS 2008]. Past research uncovered that private learners may need to exhibit significantly higher sample complexity than non-private learners as is the case of learning of one-dimensional threshold functions [Bun et al., FOCS 2015, Alon et al., STOC 2019].
We explore prediction as an alternative to learning. A predictor answers a stream of classification queries instead of outputting a hypothesis. Earlier work has considered a private prediction model with a single classification query [Dwork and Feldman, COLT 2018]. We observe that when answering a stream of queries, a predictor must modify the hypothesis it uses over time, and in a manner that cannot rely solely on the training set.
We introduce private everlasting prediction taking into account the privacy of both the training set and the (adaptively chosen) queries made to the predictor. We then present a generic construction of private everlasting predictors in the PAC model. The sample complexity of the initial training sample in our construction is quadratic (up to polylog factors) in the VC dimension of the concept class. Our construction allows prediction for all concept classes with finite VC dimension, and in particular threshold functions over infinite domains, for which (traditional) private learning is known to be impossible.
A PAC learner for a concept class  $C$  is given labeled examples  $S = \{(x_{i},y_{i})\}_{i\in [n]}$  drawn i.i.d. from an unknown underlying probability distribution  $\mathcal{D}$  over a data domain  $X$  and outputs a hypothesis  $h$  that can be used for predicting the label of fresh points  $x_{n + 1},x_{n + 2},\ldots$  sampled from the same underlying probability distribution  $\mathcal{D}$  [Valiant, 1984]. It is well known that when points are labeled by a concept selected from a concept class  $C = \{c:X\to \{0,1\}\}$  then learning is possible with sample complexity proportional to  $\mathrm{VC}(C)$ .
*Department of Computer Science and Applied Math, Weizmann Institute of Science. moni.naor@weizmann.ac.il. Incumbent of the Judith Kleeman Professorial Chair. Research supported in part by grants from the Israel Science Foundation (no.2686/20), by the Simons Foundation Collaboration on the Theory of Algorithmic Fairness and by the Israeli Council for Higher Education (CHE) via the Weizmann Data Science Research Center.
$^{\dagger}$ Department of Computer Science, Georgetown University. kobbi.nissim@georgetown.edu. Research supported in part by by NSF grant No. CNS-2001041 and a gift to Georgetown University.
‡Blavatnik School of Computer Science, Tel Aviv University, and Google Research. u@uri.co.il. Partially supported by the Israel Science Foundation (grant 1871/19) and by Len Blavatnik and the Blavatnik Family foundation.
$\S$ Department of Computer Science, Georgetown University. cy399@georgetown.edu. Research supported in part by a gift to Georgetown University.
Learning often happens in settings where the underlying training data is related to individuals and privacy-sensitive. For legal, ethical, or other reasons, the learner is then required, to protect personal information from being leaked in the learned hypothesis  $h$ . Private learning was introduced by Kasiviswanathan et al. [2011] as a theoretical model for studying such tasks. A private learner is a PAC learner that preserves differential privacy with respect to its training set  $S$ . That is, the learner's distribution on outcome hypotheses must not depend too strongly on any single example in  $S$ . Kasiviswanathan et al. showed that any finite concept class can be learned privately and with sample complexity  $n = O(\log |C|)$ , a value that is sometimes significantly higher than the VC dimension of the concept class  $C$ .
It is now understood that the gap between the sample complexity of private and non-private learners is essential - an important example is private learning of threshold functions (defined over an ordered domain  $X$  as  $C_{thresh} = \{c_t\}_{t \in X}$  where  $c_t(x) = \mathbb{1}_{x \geq t}$ ), which requires sample complexity that is asymptotically higher than the (constant) VC dimension of  $C_{thresh}$ . In more detail, with pure differential privacy, the sample complexity of private learning is characterized by the representation dimension of the concept class [Beimel et al., 2013a]. The representation dimension of  $C_{thresh}$  (hence, the sample complexity of private learning thresholds) is  $\Theta(\log |X|)$  [Feldman and Xiao, 2015]. With approximate differential privacy, the sample complexity of learning threshold functions is  $\Theta(\log^* |X|)$  [Beimel et al., 2013b, Bun et al., 2015, Alon et al., 2019, Kaplan et al., 2020, Cohen et al., 2022]. Hence, whether with pure or with approximate differential privacy, the sample complexity of privately learning thresholds grows with the cardinality of the domain  $|X|$  and no private learner exists for this task over infinite domains. In contrast, non-private learning is possible with constant sample complexity (independent of  $|X|$ ).
Privacy preserving (black-box) prediction. Dwork and Feldman [2018] proposed privacy-preserving prediction as an alternative for private learning. Noting that "[i]t is now known that for some basic learning problems [. . .] producing an accurate private model requires much more data than learning without privacy," they considered a setting where "users may be allowed to query the prediction model on their inputs only through an appropriate interface". That is, a setting where the learned hypothesis is not made public and may be accessed only in a black-box manner via a privacy-preserving query-answering prediction interface. The prediction interface is required to preserve the privacy of its training set  $S$ :
Definition 1.1 (private prediction interface [Dwork and Feldman, 2018] (rephrased)). A prediction interface  $\mathcal{A}$  is  $(\epsilon, \delta)$ -differentially private if for every interactive query generating algorithm  $Q$ , the output of the interaction between  $Q$  and  $\mathcal{A}(S)$  is  $(\epsilon, \delta)$ -differentially private with respect to  $S$ .
Dwork and Feldman focused on the setting where the entire interaction between  $Q$  and  $\mathcal{A}(S)$  consists of issuing a single prediction query and answering it:
Definition 1.2 (Single query prediction [Dwork and Feldman, 2018]). Let  $\mathcal{A}$  be an algorithm that given a set of labeled examples  $S$  and an unlabeled point  $x$  produces a label  $y$ .  $\mathcal{A}$  is an  $(\epsilon, \delta)$ -differentially private prediction algorithm if for every  $x$ , the output  $\mathcal{A}(S, x)$  is  $(\epsilon, \delta)$ -differentially private with respect to  $S$ .
W.r.t. answering a single prediction query, Dwork and Feldman showed that the sample complexity of such predictors is proportional to the VC dimension of the concept class.
We extend private prediction to answering a sequence – unlimited in length – of prediction queries. We refer to this as private everlasting prediction (PEP). Our goal is to present a generic private everlasting predictor with low training sample complexity  $|S|$ .
1.1.1 Private prediction interfaces when applied to a large number of queries
We begin by examining private everlasting prediction under the framework of Definition 1.1. We prove:
Theorem 1.3 (informal version of Theorem 3.3). Let  $\mathcal{A}$  be a private everlasting prediction interface for concept class  $C$  and assume  $\mathcal{A}$  bases its predictions solely on the initial training set  $S$ , then there exists a private learner for concept class  $C$  with sample complexity  $|S|$ .
This means that everlasting predictors that base their prediction solely on the initial training set  $S$  are subject to the same complexity lower bounds as private learners. Hence, to avoid private learning lower bounds, private everlasting predictors need to rely on more than the initial training sample  $S$  as a source of information about the underlying probability distribution and the labeling concept.
In this work, we choose to allow the everlasting predictor to rely on the queries made – which are unlabeled points from the domain  $X$  – assuming the queries are drawn from the same distribution the initial training  $S$  is sampled from. This requires modifying the privacy definition, as Definition 1.1 does not protect the queries issued to the predictor.
1.1.2 A definition of private everlasting predictors
Our definition of private everlasting predictors is motivated by the observations above. Consider an algorithm  $\mathcal{A}$  that is first fed with a training set  $S$  of labeled points and then executes for an unlimited number of rounds, where in round  $i$  algorithm  $\mathcal{A}$  receives as input a query point  $x_{i}$  and produces a label  $\hat{y}_i$ . We say that  $\mathcal{A}$  is an everlasting predictor if, when the (labeled) training set  $S$  and the (unlabeled) query points are coming from the same underlying distribution,  $\mathcal{A}$  answers each query points  $x_{i}$  by invoking a good hypothesis  $h_i$  (i.e.,  $h_i$  has low generalization error), and hence the label  $\hat{y}_i$  produced by  $\mathcal{A}$  is correct with high probability. We say that  $\mathcal{A}$  is a private everlasting predictor if its sequence of predictions  $\hat{y}_1,\hat{y}_2,\hat{y}_3,\ldots$  protects both the privacy of the training set  $S$  and the query points  $x_{1},x_{2},x_{3},\ldots$  in face of any adversary that chooses the query points adaptively.
We emphasize that while private everlasting predictors need to exhibit average-case utility – as good prediction is required only for the case where the initial training set  $S$  and the queries  $x_{1},x_{2},x_{3},\ldots$  are selected i.i.d. from the same underlying distribution – our privacy requirement is worst-case, and holds in face of an adaptive adversary that chooses each query point  $x_{i}$  after receiving the prediction provided for  $(x_{1},\dots,x_{i - 1})$ , and not necessarily in accordance with any probability distribution.
1.1.3 A generic construction of private everlasting predictors
We show a reduction from private everlasting prediction of a concept class  $C$  to non-private PAC learning of  $C$ . Our Algorithm GenericBBL, presented in Section 6, executes in rounds.
Initialization. The input to the first round is a labeled training set  $S$  where  $|S| = O\big((\mathrm{VC}(C))^2\big)$ , assumed to be labeled consistently with some unknown target concept  $c \in C$ . Denote  $S_{1} = S$  and  $c_{1} = c$ .
Rounds. Each round begins with a collection  $S_{i}$  of labeled examples and ends with a newly generated collection of labeled examples  $S_{i+1}$  that feeds as input for the next round. The size of these collections grows by a constant factor at each round, to allow for the accumulated error of the predictor to converge. The construction ensures that each labeled set  $S_{i}$  is consistent with some concept in  $c_{i} \in C$ , which is not necessarily the original target concept  $c$ , but has a bounded generalization error with respect to  $c$ . In more detail, the construction ensures that  $\operatorname{Pr}_{x \sim \mathcal{D}}[c_{i+1}(x) \neq c_{i}(x)]$  decreases by a factor of two in every round, and hence, by the triangle inequality,  $\operatorname{Pr}_{x \sim \mathcal{D}}[c_{i}(x) \neq c(x)]$  is bounded.
We briefly describe the main computations performed in each round of GenericBBL.
- Round initialization: At the outset of a round, the labeled set  $S_{i}$  is partitioned into sub-sets, each with number of samples which is proportional to the VC dimension (so we have  $\approx \frac{|S_i|}{\mathrm{VC}(C)}$  sub-sets). Each of the sub-sets is used for training a classifier
5Important details, such as privacy amplification via sampling and management of the learning accuracy and error parameters are omitted from the description provided in this section.
non-privately, hence creating a collection of classifiers  $F_{i} = \{f:X\to \{0,1\} \}$  that are used throughout the round.
- **Query answering:** Queries are issued to the predictor in an online manner. A query is first labeled by each of the classifiers in  $F_{i}$ . Then the predicted label is computed by applying a privacy-preserving majority vote on these intermediate labels. By standard composition theorems for differential privacy, we could answer roughly  $|F_{i}|^{2} \approx \left(\frac{|S_{i}|}{\mathrm{VC}(C)}\right)^{2}$  queries without exhausting the privacy budget.
- Generating a labeled set for the following round: A round ends with the preparation of a collection  $S_{i+1}$  of labeled samples that is to be used in the initialization of next round. We explore alternatives for creating  $S_{i+1}$ :
- As at the end of a round the predictor has already labeled all the queries presented during the round's execution, one alternative is to let  $S_{i+1}$  consist of these queries and the labels provided for them. Note, however, that it is not guaranteed that the majority vote would result in a set of labels that are consistent with some concept in  $C$ , even if  $S_i$  is consistent with some concept in  $C$ . Hence, following this alternative would require the use of non-private agnostic learners instead. Furthermore, the error introduced by the majority vote can be larger than the generalization error of the classifiers in  $F_i$  by a constant factor greater than one. This may prevent the generalization error of the classifiers used in future rounds from converging.<sup>6</sup>
- To overcome this problem, we use Algorithm LabelBoost - a tool developed by Beimel et al. [2021] in the context of private semi-supervised learning. LabelBoost takes as input the sample  $S_{i}$  (labeled by a concept  $c_{i} \in C$ ) and the (unlabeled) queries made during the round. It labels them with a concept  $c_{i+1} \in C$  where the error of  $c_{i+1}$  with respect to  $c_{i}$ , i.e.,  $\operatorname{Pr}_{x \sim \mathcal{D}}[c_{i+1}(x) \neq c_{i}(x)]$  is bounded.
- Controlling the generalization error: Let  $S_{i+1}$  denote the (re)labeled query points obtained in the  $i$ th round. This is a collection of size  $|S_{i+1}| \approx \left( \frac{|S_i|}{\mathrm{VC}(C)} \right)^2$ . Hence, provided that  $|S_i| \gtrsim (\mathrm{VC}(C))^2$  we get that  $|S_{i+1}| > |S_i|$ . This allows to lower the accuracy parameters of the non-private learners in each round, and hence ensure that the total error converges.
Theorem 1.4 (informal version of Theorem 6.1). For every concept class  $C$ , Algorithm GenericBBL is a private everlasting predictor requiring an initial set of labeled examples which is (upto polylogarithmic factors) quadratic in the VC dimension of  $C$ .
Beyond the work of Dwork and Feldman [2018] on private prediction mentioned above, our work is related to private semi-supervised learning and joint differential privacy.
Semi-supervised private learning. As in the model of private semi-supervised learning of Beimel et al. [2021], our predictors depend on both labeled and unlabeled samples. Beyond the difference between outputting a hypothesis and providing black-box prediction, a major difference between the settings is that in the work of Beimel et al. [2021] all samples – labeled and unlabeled – are given at once at the outset of the learning process whereas in the setting of everlasting predictors the unlabeled samples are supplied in an online manner. Our construction of private everlasting predictors uses tools developed for the semi-supervised setting, and in particular Algorithm LabelBoost of beimel et al.
Joint differential privacy. Kearns et al. [2015] introduced joint differential privacy (JDP) as a relaxation of differential privacy applicable for mechanism design and games. For
To see that the majority vote can increase the generalization error by a constant factor consider three classifiers  $f_{1}, f_{2}, f_{3}$ , each with generalization error  $\alpha$ , and let  $A, B, C$  be three disjoint subsets of the support of the underlying distribution, each of probability weight  $\alpha / 2$ . If the classifier  $f_{1}$  errs on inputs in  $A \cup B$ , the classifier  $f_{2}$  errs on  $A \cup C$ , and  $f_{3}$  errs on  $B \cup C$  then the majority of  $f_{1}, f_{2}, f_{3}$  errs on inputs in  $A \cup B \cup C$  and hence the generalization error grows from  $\alpha$  to  $1.5\alpha$ .
every user  $u$ , JDP requires that the outputs jointly seen by all other users would preserve differential privacy w.r.t. the input of  $u$ . Crucially, in JDP users select their inputs ahead of the computation. In our settings, the inputs to a private everlasting predictor are prediction queries which are chosen in an online manner, and hence a query can depend on previous queries and their answers. Yet, similarly to JDP, the outputs provided to queries not performed by a user  $u$  should jointly preserve differential privacy w.r.t. the query made by  $u$ . Our privacy requirement hence extends JDP to an adaptive online setting.
Additional works on private prediction. Bassily et al. [2018] studied a variant of the private prediction problem where the algorithm takes a labeled sample  $S$  and is then required to answer  $m$  prediction queries (i.e., label a sequence of  $m$  unlabeled points sampled from the same underlying distribution). They presented algorithms for this task with sample complexity  $|S| \gtrsim \sqrt{m}$ . This should be contrasted with our model and results, where the sample complexity is independent of  $m$ . The bounds presented by Dwork and Feldman [2018] and Bassily et al. [2018] were improved by Dagan and Feldman [2020] and by Nandi and Bassily [2020] who presented algorithms with improved dependency on the accuracy parameter in the agnostic setting.
1.3 Discussion and open problems
We show how to transform any (non-private) learner for the class  $C$  (with sample complexity proportional to the VC dimension of  $C$ ) to a private everlasting predictor for  $C$ . Our construction is not polynomial time due to the use of Algorithm LabelBoost, and requires an initial set  $S$  of labeled examples which is quadratic in the VC dimension. We leave open the question whether  $|S|$  can be reduced to be linear in the VC dimension and whether the construction can be made polynomial time. A few remarks are in order:
1. While our generic construction is not computationally efficient, it does result in efficient learners for several interesting special cases. Specifically, algorithm LabelBoost can be implemented efficiently whenever given an input sample  $S$  it is possible to efficiently enumerate all possible dichotomies from the target class  $C$  over the points in  $S$ . In particular, this is the case for the class of 1-dim threshold functions  $C_{thresh}$ , as well as additional classes with constant VC dimension. Another notable example is the class  $C_{thresh}^{enc}$  which intuitively is an "encrypted" version of  $C_{thresh}$ . Bun and Zhandry [2016] showed that (under plausible cryptographic assumptions) the class  $C_{thresh}^{enc}$  cannot be learned privately and efficiently, while it can be learned efficiently non-privately. Our construction can be implemented efficiently for this class. This provides an example where private everlasting prediction can be done efficiently, while (standard) private learning is possible but necessarily inefficient.
2. It is now known that some learning tasks require the produced model to memorize parts of the training set in order to achieve good learning rates, which in particular disallows the learning algorithm from satisfying (standard) differential privacy [Brown et al., 2021]. Our notion of private everlasting prediction circumvents this issue, since the model is never publicly released and hence the fact that it must memorize parts of the sample is not of a direct privacy threat. In other words, our work puts forward a private learning model which, in principle, allows memorization. This could have additional applications in broader settings.
3. As mentioned above, in general, private everlasting predictors cannot base their predictions solely on the initial training set, and in this work we choose to rely on the queries presented to the algorithm (in addition to the training set). Our construction can be easily adapted to a setting where the content of the blackbox is updated based on fresh unlabeled samples (whose privacy would be preserved), instead of relying on the query points themselves. This might be beneficial to avoid poisoning attacks via the queries.
2.1 Preliminaries from differential privacy
Definition 2.1  $((\epsilon, \delta)$ -indistinguishability). Let  $R_0, R_1$  be two random variables over the same support. We say that  $R_0, R_1$  are  $(\epsilon, \delta)$ -indistinguishable if for every event  $E$  defined over the support of  $R_0, R_1$ ,
$$
\Pr \left[ R _ {0} \in E \right] \leq e ^ {\epsilon} \cdot \Pr \left[ R _ {1} \in E \right] + \delta \text {a n d} \Pr \left[ R _ {1} \in E \right] \leq e ^ {\epsilon} \cdot \Pr \left[ R _ {0} \in E \right] + \delta .
$$
Definition 2.2. Let  $X$  be a data domain. Two datasets  $x, x' \in X^n$  are called neighboring if  $|\{i : x_i \neq x_j'\}| = 1$ .
Definition 2.3 (differential privacy [Dwork et al., 2006]). A mechanism  $M: X^n \to Y$  is  $(\epsilon, \delta)$ -differentially private if  $M(x)$  and  $M(x')$  are  $(\epsilon, \delta)$ -indistinguishable for all neighboring  $x, x' \in X^n$ .
In our analysis, we use the post-processing and composition properties of differential privacy, that we cite in their simplest forms.
Proposition 2.4 (post-processing). Let  $M_1: X^n \to Y$  be an  $(\epsilon, \delta)$ -differentially private algorithm and  $M_2: Y \to Z$  be any algorithm. Then the algorithm that on input  $x \in X^n$  outputs  $M_2(M_1(x))$  is  $(\epsilon, \delta)$ -differentially private.
Proposition 2.5 (composition). Let  $M_1$  be a  $(\epsilon_1, \delta_1)$ -differentially private algorithm and let  $M_2$  be  $(\epsilon_2, \delta_2)$ -differentially private algorithm. Then the algorithm that on input  $x \in X^n$  outputs  $(M_1(x), M_2(x))$  is  $(\epsilon_1 + \epsilon_2, \delta_1 + \delta_2)$ -differentially private.
Theorem 2.6 (Advanced composition Dwork et al. [2010b]). Let  $M_1, \ldots, M_k : X \to Y$  be  $(\varepsilon, \delta)$ -differentially private algorithms. Then the algorithm that on input  $x \in X$  outputs  $(M_1(x), \ldots, M_k(x))$  is  $(\varepsilon', k\delta + \delta')$ -differentially private, where  $\varepsilon' = \sqrt{2k\ln(1/\delta')} \cdot \varepsilon$  for every  $\delta' > 0$ .
Definition 2.7 (Laplace mechanism [Dwork et al., 2006]). For  $f: X^n \to \mathbb{R}$  Let  $\Delta_f = \max(f(x) - f(x'))$ , where the maximum is taken over all neighboring  $x, x' \in X^n$ . The Laplace mechanism is  $M(x) = f(x) + Y$ , where  $Y$  is sampled from the laplace distribution  $\mathrm{Lap}(\bar{\Delta}_f / \varepsilon)$ . The Laplace mechanism is  $(\varepsilon, 0)$ -differentially private.
Definition 2.8 (Exponential mechanism [McSherry and Talwar, 2007]). Let  $q: X^n \times Y \to \mathbb{R}$  be a score function defined over data domain  $X$  and output domain  $Y$ . Define  $\Delta = \max(|q(x,y) - q(x',y)|)$  where the maximum is taken over all  $y \in Y$  and all neighbouring databases  $x, x' \in X^n$ . The exponential mechanism is the  $(\epsilon, 0)$ -differentially private mechanism which selects an output  $y \in Y$  with probability proportional to  $e^{\frac{\epsilon q(x,y)}{2\Delta}}$ .