Title: Private Everlasting Prediction
Venue: NeurIPS 2023
Accepted: yes
Keywords: Differential privacy

private learning

private prediction
Abstract: A private learner is trained on a sample of labeled points and generates a hypothesis that can be used for predicting the labels of newly sampled points while protecting the privacy of the training set [Kasiviswannathan et al., FOCS 2008]. Past research uncovered that private learners may need to exhibit significantly higher sample complexity than non-private learners as is the case of learning of one-dimensional threshold functions [Bun et al., FOCS 2015, Alon et al., STOC 2019].

We explore prediction as an alternative to learning. A predictor answers a stream of classification queries instead of outputting a hypothesis. 
Earlier work has considered a private prediction model with a single classification query [Dwork and Feldman, COLT 2018]. We observe that when answering a stream of queries, a predictor must modify the hypothesis it uses over time, and in a manner that  cannot rely solely on the training set.

We introduce {\em private everlasting prediction} taking into account the privacy of both the training set {\em and} the (adaptively chosen) queries made to the predictor. 
We then present a generic construction of private everlasting predictors in the PAC model.
The sample complexity of the initial training sample in our construction is quadratic (up to polylog factors) in the VC dimension of the concept class. Our construction allows prediction for all concept classes with finite VC dimension, and in particular threshold functions 
over infinite domains, for which (traditional) private learning is known to be impossible.

Private Everlasting Prediction

Abstract

A private learner is trained on a sample of labeled points and generates a hypothesis that can be used for predicting the labels of newly sampled points while protecting the privacy of the training set [Kasiviswannathan et al., FOCS 2008]. Past research uncovered that private learners may need to exhibit significantly higher sample complexity than non-private learners as is the case of learning of one-dimensional threshold functions [Bun et al., FOCS 2015, Alon et al., STOC 2019].

We explore prediction as an alternative to learning. A predictor answers a stream of classification queries instead of outputting a hypothesis. Earlier work has considered a private prediction model with a single classification query [Dwork and Feldman, COLT 2018]. We observe that when answering a stream of queries, a predictor must modify the hypothesis it uses over time, and in a manner that cannot rely solely on the training set.

We introduce private everlasting prediction taking into account the privacy of both the training set and the (adaptively chosen) queries made to the predictor. We then present a generic construction of private everlasting predictors in the PAC model. The sample complexity of the initial training sample in our construction is quadratic (up to polylog factors) in the VC dimension of the concept class. Our construction allows prediction for all concept classes with finite VC dimension, and in particular threshold functions over infinite domains, for which (traditional) private learning is known to be impossible.

1 Introduction

A PAC learner for a concept class C is given labeled examples S = { ( xi , y i ) } i ∈ [ n ] drawn i.i.d. from an unknown underlying probability distribution D over a data domain X and outputs a hypothesis h that can be used for predicting the label of fresh points xn +1 , x n +2 , . . . sampled from the same underlying probability distribution D [Valiant, 1984]. It is well known that when points are labeled by a concept selected from a concept class C = { c : X →{ 0 , 1 }} then learning is possible with sample complexity proportional to VC( C ).

Learning often happens in settings where the underlying training data is related to individuals and privacy-sensitive. For legal, ethical, or other reasons, the learner is then required, to protect personal information from being leaked in the learned hypothesis h . Private learning was introduced by Kasiviswanathan et al. [2011] as a theoretical model for studying such tasks. A private learner is a PAC learner that preserves di ff erential privacy with respect to its training set S . That is, the learner's distribution on outcome hypotheses must not depend too strongly on any single example in S . Kasiviswanathan et al. showed that any finite concept class can be learned privately and with sample complexity n = O (log | C | ), a value that is sometimes significantly higher than the VC dimension of the concept class C .

It is now understood that the gap between the sample complexity of private and non-private learners is essential - an important example is private learning of threshold functions (defined over an ordered domain X as Cthresh = { c t } t ∈ X where c t ( x ) = 1 x ≥ t ), which requires sample complexity that is asymptotically higher than the (constant) VC dimension of Cthresh . In more detail, with pure di ff erential privacy, the sample complexity of private learning is characterized by the representation dimension of the concept class [Beimel et al., 2013a]. The representation dimension of Cthresh (hence, the sample complexity of private learning thresholds) is Θ (log | X | ) [Feldman and Xiao, 2015]. With approximate di ff erential privacy, the sample complexity of learning threshold functions is Θ (log ∗ | X | ) [Beimel et al., 2013b, Bun et al., 2015, Alon et al., 2019, Kaplan et al., 2020, Cohen et al., 2022]. Hence, whether with pure or with approximate di ff erential privacy, the sample complexity of privately learning thresholds grows with the cardinality of the domain | X | and no private learner exists for this task over infinite domains. In contrast, non-private learning is possible with constant sample complexity (independent of | X | ).

Privacy preserving (black-box) prediction. Dwork and Feldman [2018] proposed privacypreserving prediction as an alternative for private learning. Noting that '[i]t is now known that for some basic learning problems [. . .] producing an accurate private model requires much more data than learning without privacy,' they considered a setting where 'users may be allowed to query the prediction model on their inputs only through an appropriate interface'. That is, a setting where the learned hypothesis is not made public and may be accessed only in a black-box manner via a privacy-preserving query-answering prediction interface. The prediction interface is required to preserve the privacy of its training set S :

Definition 1.1 (private prediction interface [Dwork and Feldman, 2018] (rephrased)) . A prediction interface A is ( ϵ,δ )-di ff erentially private if for every interactive query generating algorithm Q , the output of the interaction between Q and A ( S ) is ( ϵ,δ )-di ff erentially private with respect to S .

Dwork and Feldman focused on the setting where the entire interaction between Q and A ( S ) consists of issuing a single prediction query and answering it:

Definition 1.2 (Single query prediction [Dwork and Feldman, 2018]) . Let A be an algorithm that given a set of labeled examples S and an unlabeled point x produces a label y . A is an ( ϵ,δ )-di ff erentially private prediction algorithm if for every x , the output A ( S,x ) is ( ϵ,δ )-di ff erentially private with respect to S .

W.r.t. answering a single prediction query, Dwork and Feldman showed that the sample complexity of such predictors is proportional to the VC dimension of the concept class.

1.1 Our contributions

We extend private prediction to answering a sequence unlimited in length - of prediction queries. We refer to this as private everlasting prediction (PEP). Our goal is to present a generic private everlasting predictor with low training sample complexity | S | .

1.1.1 Private prediction interfaces when applied to a large number of queries

Webegin by examining private everlasting prediction under the framework of Definition 1.1. We prove:

Theorem 1.3 (informal version of Theorem 3.3) . Let A be a private everlasting prediction interface for concept class C and assume A bases its predictions solely on the initial training set S , then there exists a private learner for concept class C with sample complexity | S | .

This means that everlasting predictors that base their prediction solely on the initial training set S are subject to the same complexity lowerbounds as private learners. Hence, to avoid private learning lowerbounds, private everlasting predictors need to rely on more than the initial training sample S as a source of information about the underlying probability distribution and the labeling concept.

In this work, we choose to allow the everlasting predictor to rely on the queries made which are unlabeled points from the domain X - assuming the queries are drawn from the same distribution the initial training S is sampled from. This requires modifying the privacy definition, as Definition 1.1 does not protect the queries issued to the predictor.

1.1.2 Adefinition of private everlasting predictors

Our definition of private everlasting predictors is motivated by the observations above. Consider an algorithm A that is first fed with a training set S of labeled points and then executes for an unlimited number of rounds, where in round i algorithm A receives as input a query point xi and produces a label ˆ yi . We say that A is an everlasting predictor if, when the (labeled) training set S and the (unlabeled) query points are coming from the same underlying distribution, A answers each query points xi by invoking a good hypothesis hi (i.e., hi has low generalization error), and hence the label ˆ yi produced by A is correct with high probability. We say that A is a private everlasting predictor if its sequence of predictions ˆ y 1 , ˆ y 2 , ˆ y 3 , . . . protects both the privacy of the training set S and the query points x 1 , x 2 , x 3 , . . . in face of any adversary that chooses the query points adaptively.

We emphasize that while private everlasting predictors need to exhibit average-case utility - as good prediction is required only for the case where the initial training set S and the queries x 1 , x 2 , x 3 , . . . are selected i.i.d. from the same underlying distribution - our privacy requirement is worst-case, and holds in face of an adaptive adversary that chooses each query point xi after receiving the prediction provided for ( x 1 , . . . , x i -1 ), and not necessarily in accordance with any probability distribution.

1.1.3 Ageneric construction of private everlasting predictors

We show a reduction from private everlasting prediction of a concept class C to non-private PAC learning of C . Our Algorithm GenericBBL , presented in Section 6, executes in rounds.

Initialization. The input to the first round is a labeled training set S where | S | = O GLYPH<16> (VC( C )) 2 GLYPH<17> , assumed to be labeled consistently with some unknown target concept c ∈ C . Denote S 1 = S and c 1 = c .

Rounds. Each round begins with a collection Si of labeled examples and ends with a newly generated collection of labeled examples Si +1 that feeds as input for the next round. The size of these collections grows by a constant factor at each round, to allow for the accumulated error of the predictor to converge. The construction ensures that each labeled set Si is consistent with some concept in c i ∈ C , which is not necessarily the original target concept c , but has a bounded generalization error with respect to c . In more detail, the construction ensures that Pr x ∼D [ c i +1 ( x ) , c i ( x )] decreases by a factor of two in every round, and hence, by the triangle inequality, Pr x ∼D [ c i ( x ) , c ( x )] is bounded.

We briefly describe the main computations performed in each round of GenericBBL . 5

Round initialization: At the outset of a round, the labeled set Si is partitioned into sub-sets, each with number of samples which is proportional to the VC dimension (so we have ≈ | Si | VC( C ) sub-sets). Each of the sub-sets is used for training a classifier

non-privately, hence creating a collection of classifiers Fi = { f : X →{ 0 , 1 }} that are used throughout the round.

Query answering: Queries are issued to the predictor in an online manner. A query is first labeled by each of the classifiers in Fi . Then the predicted label is computed by applying a privacy-preserving majority vote on these intermediate labels. By standard composition theorems for di ff erential privacy, we could answer roughly | Fi | 2 ≈ GLYPH<16> | Si | VC( C ) GLYPH<17> 2 queries without exhausting the privacy budget.

Generating a labeled set for the following round: A round ends with the preparation of a collection Si +1 of labeled samples that is to be used in the initialization of next round. We explore alternatives for creating Si +1 :

-As at the end of a round the predictor has already labeled all the queries presented during the round's execution, one alternative is to let Si +1 consist of these queries and the labels provided for them. Note, however, that it is not guaranteed that the majority vote would result in a set of labels that are consistent with some concept in C , even if Si is consistent with some concept in C . Hence, following this alternative would require the use of non-private agnostic learners instead. Furthermore, the error introduced by the majority vote can be larger than the generalization error of the classifiers in Fi by a constant factor greater than one. This may prevent the generalization error of the classifiers used in future rounds from converging. 6

-To overcome this problem, we use Algorithm LabelBoost - a tool developed by Beimel et al. [2021] in the context of private semi-supervised learning. LabelBoost takes as input the sample Si (labeled by a concept c i ∈ C ) and the (unlabeled) queries made during the round. It labels them with a concept c i +1 ∈ C where the error of c i +1 with respect to c i , i.e., Pr x ∼D [ c i +1 ( x ) , c i ( x )] is bounded.

Controlling the generalization error: Let Si +1 denote the (re)labeled query points obtained in the i th round. This is a collection of size | Si +1 | ≈ GLYPH<16> | Si | VC( C ) GLYPH<17> 2 . Hence, provided that | Si | ≳ (VC( C )) 2 we get that | Si +1 | > | Si | . This allows to lower the accuracy parameters of the non-private learners in each round, and hence ensure that the total error converges.

Theorem 1.4 (informal version of Theorem 6.1) . For every concept class C , Algorithm GenericBBL is a private everlasting predictor requiring an initial set of labeled examples which is (upto polylogarithmic factors) quadratic in the VC dimension of C .

1.2 Related work

Beyond the work of Dwork and Feldman [2018] on private prediction mentioned above, our work is related to private semi-supervised learning and joint di ff erential privacy.

Semi-supervised private learning. As in the model of private semi-supervised learning of Beimel et al. [2021], our predictors depend on both labeled and unlabeled samples. Beyond the di ff erence between outputting a hypothesis and providing black-box prediction, a major di ff erence between the settings is that in the work of Beimel et al. [2021] all samples labeled and unlabeled - are given at once at the outset of the learning process whereas in the setting of everlasting predictors the unlabeled samples are supplied in an online manner. Our construction of private everlasting predictors uses tools developed for the semi-supervised setting, and in particular Algorithm LabelBoost of of Beimel et al.

Joint di ff erential privacy. Kearns et al. [2015] introduced joint di ff erential privacy (JDP) as a relaxation of di ff erential privacy applicable for mechanism design and games. For

every user u , JDP requires that the outputs jointly seen by all other users would preserve di ff erential privacy w.r.t. the input of u . Crucially, in JDP users select their inputs ahead of the computation. In our settings, the inputs to a private everlasting predictor are prediction queries which are chosen in an online manner, and hence a query can depend on previous queries and their answers. Yet, similarly to JDP, the outputs provided to queries not performed by a user u should jointly preserve di ff erential privacy w.r.t. the query made by u . Our privacy requirement hence extends JDP to an adaptive online setting.

Additional works on private prediction. Bassily et al. [2018] studied a variant of the private prediction problem where the algorithm takes a labeled sample S and is then required to answer m prediction queries (i.e., label a sequence of m unlabeled points sampled from the same underlying distribution). They presented algorithms for this task with sample complexity | S | ≳ √ m . This should be contrasted with our model and results, where the sample complexity is independent of m . The bounds presented by Dwork and Feldman [2018] and Bassily et al. [2018] were improved by Dagan and Feldman [2020] and by Nandi and Bassily [2020] who presented algorithms with improved dependency on the accuracy parameter in the agnostic setting.

1.3 Discussion and open problems

Weshow how to transform any (non-private) learner for the class C (with sample complexity proportional to the VC dimension of C ) to a private everlasting predictor for C . Our construction is not polynomial time due to the use of Algorithm LabelBoost , and requires an initial set S of labeled examples which is quadratic in the VC dimension. We leave open the question whether | S | can be reduced to be linear in the VC dimension and whether the construction can be made polynomial time. A few remarks are in order:

While our generic construction is not computationally e ffi cient, it does result in e ffi cient learners for several interesting special cases. Specifically, algorithm LabelBoost can be implemented e ffi ciently whenever given an input sample S it is possible to e ffi ciently enumerate all possible dichotomies from the target class C over the points in S . In particular, this is the case for the class of 1-dim threshold functions Cthresh , as well as additional classes with constant VC dimension. Another notable example is the class C enc thresh which intuitively is an 'encrypted' version of Cthresh . Bun and Zhandry [2016] showed that (under plausible cryptographic assumptions) the class C enc thresh cannot be learned privately and e ffi ciently, while it can be learned e ffi ciently non-privately. Our construction can be implemented e ffi ciently for this class. This provides an example where private everlasting prediction can be done e ffi ciently, while (standard) private learning is possible but necessarily ine ffi cient.

It is now known that some learning tasks require the produced model to memorize parts of the training set in order to achieve good learning rates, which in particular disallows the learning algorithm from satisfying (standard) di ff erential privacy [Brown et al., 2021]. Our notion of private everlasting prediction circumvents this issue, since the model is never publicly released and hence the fact that it must memorize parts of the sample is not of a direct privacy threat. In other words, our work puts forward a private learning model which, in principle, allows memorization. This could have additional applications in broader settings.

As mentioned above, in general, private everlasting predictors cannot base their predictions solely on the initial training set, and in this work we choose to rely on the queries presented to the algorithm (in addition to the training set). Our construction can be easily adapted to a setting where the content of the blackbox is updated based on fresh unlabeled samples (whose privacy would be preserved), instead of relying on the query points themselves. This might be beneficial to avoid poisoning attacks via the queries.

2 Preliminaries

2.1 Preliminaries from di ff erential privacy

Definition 2.1 (( ϵ,δ )-indistinguishability) . Let R 0 , R 1 be two random variables over the same support. We say that R 0 , R 1 are ( ϵ,δ )-indistinguishable if for every event E defined over the support of R 0 , R 1 ,

Definition 2.2. Let X be a data domain. Two datasets x,x ′ ∈ X n are called neighboring if |{ i : xi , x ′ i }| = 1.

Definition 2.3 (di ff erential privacy [Dwork et al., 2006]) . A mechanism M : X n → Y is ( ϵ,δ )-di ff erentially private if M ( x ) and M ( x ′ ) are ( ϵ,δ )-indistinguishable for all neighboring x,x ′ ∈ X n .

In our analysis, we use the post-processing and composition properties of di ff erential privacy, that we cite in their simplest forms.

Proposition 2.4 (post-processing) . Let M 1 : X n → Y be an ( ϵ,δ ) -di ff erentially private algorithm and M 2 : Y → Z be any algorithm. Then the algorithm that on input x ∈ X n outputs M 2 ( M 1 ( x )) is ( ϵ,δ ) -di ff erentially private.

Proposition 2.5 (composition) . Let M 1 be a ( ϵ 1 , δ 1 ) -di ff erentially private algorithm and let M 2 be ( ϵ 2 , δ 2 ) -di ff erentially private algorithm. Then the algorithm that on input x ∈ X n outputs ( M 1 ( x ) , M 2 ( x ) is ( ϵ 1 + ϵ 2 , δ 1 + δ 2 ) -di ff erentially private.

Theorem 2.6 (Advanced composition Dwork et al. [2010b]) . Let M 1 , . . . , M k : X → Y be ( ε,δ ) -di ff erentially private algorithms. Then the algorithm that on input x ∈ X outputs ( M 1 ( x ) , . . . , M k ( x )) is ( ε ′ , kδ + δ ′ ) -di ff erentially private, where ε ′ = p 2 k ln(1 /δ ′ ) · ε for every δ ′ > 0 .

Definition 2.7 (Laplace mechanism [Dwork et al., 2006]) . For f : X n → R Let ∆ f = max( f ( x ) -f ( x ′ )), where the maximum is taken over all neighboring x,x ′ ∈ X n . The Laplace mechanis is M ( x ) = f ( x ) + Y , where Y is sampled from the laplace distribution Lap( ∆ f /ε ). The Laplace mechanism is ( ε, 0)-di ff erentially private.

Definition 2.8 (Exponential mechanism [McSherry and Talwar, 2007]) . Let q : X n × Y → R be a score function defined over data domain X and output domain Y . Define ∆ = max( | q ( x,y ) -q ( x ′ , y ) | ) where the maximum is taken over all y ∈ Y and all neighbouring databases x,x ′ ∈ X n . The exponential mechanism is the ( ϵ, 0)-di ff erentially private mecha- nism which selects an output y ∈ Y with probability proportional to e ϵq ( x,y ) 2 ∆ .

Claim 2.9 (Privacy amplification by sub-sampling [Kasiviswanathan et al., 2011]) . Let A be an ( ε ′ , δ ′ ) -di ff erentially private algorithm operating on a database of size n . Let ε ≤ 1 and let t = n ε (3 + exp ( ε ′ )) . Construct an algorithm B operating the database D = ( zi ) t i =1 . Algorithm B randomly selects a subset J ⊆ { 1 , 2 , . . . , t } of size n , and executes A on DJ = ( zi ) i ∈ J . Then B is GLYPH<18> ε, 4 ε 3+ exp ( ε ′ ) δ ′ GLYPH<19> -di ff erentially private.

2.2 Preliminaries from PAC learning

A concept class C over data domain X is a set of predicates c : X →{ 0 , 1 } (called concepts) which label points of the domain X by either 0 or 1. A learner A for concept class C is given n examples sampled i.i.d. from an unknown probability distribution D over the data domain X and labeled according to an unknown target concept c ∈ C . The learner should output a hypothesis h : X → [0 , 1] that approximates c for the distribution D . More formally,

Definition 2.10 (generalization error) . The generalization error of a hypothesis h : X → [0 , 1] with respect to concept c and distribution D is defined as error D ( c,h ) = Exp x ∼D [ | h ( x ) -c ( x ) | ] .

Definition 2.11 (PAC learning [Valiant, 1984]) . Let C be a concept class over a domain X . Algorithm A is an ( α,β,n ) -PAC learner for C if for all c ∈ C and all distributions D on X ,

where the probability is over the sampling of ( x 1 , . . . , x n ) from D and the coin tosses of A . The parameter n is the sample complexity of A .

See Appendix A for additional preliminaries on PAC learning.

2.3 Preliminaties from private learning

Definition 2.12 (private PAC learning [Kasiviswanathan et al., 2011]) . Algorithm A is a ( α,β,ϵ,δ,n )-private PAC learner if (i) A is an ( α,β,n )-PAC learner and (ii) A is ( ϵ,δ ) di ff erentially private.

Kasiviswanathan et al. [2011] provided a generic private learner with O (log( | X | )) labeled samples. 7 Beimel et al. [2013a] introduced the representation dimension and showed that any concept class C can be privately learned with Θ (RepDim( C )) samples. For the sample complexity of ( ϵ,δ )-di ff erentially private learning of threshold functions over domain X , Bun et al. [2015] gave a lower bound of Ω (log ∗ | X | ). Recently, Cohen et al. [2022] gave a (nearly) matching upper bound of ˜ O (log ∗ | X | ).

3 Towards private everlasting prediction

In this work, we extend private prediction to answering any sequence of prediction queries - unlimited in length. Our main goal is to present a generic private everlasting predictor with low training sample complexity | S | .

Definition 3.1 (everlasting prediction) . Let A be an algorithm with the following properties:

Algorithm A receives as input n labeled examples S = { ( xi , y i ) } n i =1 ∈ ( X ×{ 0 , 1 } ) n and selects a hypothesis h 0 : X →{ 0 , 1 } .

For round r ∈ N , algorithm A gets a query, which is an unlabeled element xn + r ∈ X , outputs hr -1 ( xn + r ) and selects a hypothesis hr : X →{ 0 , 1 } .

We say that A is an ( α,β,n ) -everlasting predictor for a concept class C over a domain X if the following holds for every concept c ∈ C and for every distribution D over X . If x 1 , x 2 , . . . are sampled i.i.d. from D , and the labels of the n initial samples S are correct, i.e., yi = c ( xi ) for i ∈ [ n ], then Pr[ ∃ r ≥ 0 s.t. error D ( c,hr ) > α ] ≤ β, where the probability is over the sampling of x 1 , x 2 , . . . from D and the randomness of A .

Applying the Dwork-Feldman notion of a private prediction interface to everlasting predictors we get:

Definition 3.2. An algorithm A is an ( α,β,ϵ,δ,n )-everlasting di ff erentially private prediction interface if (i) A is a ( ϵ,δ )-di ff erentially private prediction interface M (as in Definition 1.1), and (ii) A is an ( α,β,n )-everlasting predictor.

As a warmup, consider an ( α,β,ϵ,δ,n )-everlasting di ff erentially private prediction interface A for concept class C over (finite) domain X (as in Definition 3.2 above). Assume that A does not vary its hypotheses, i.e. (in the language of Definition 3.1) hr = h 0 for all r > 0. 8 Note that a computationally unlimited adversarial querying algorithm can recover the hypothesis h 0 by issuing all queries x ∈ X . Hence, in using A indefinitely we lose any potential benefits to sample complexity of restricting access to h 0 to being black-box and getting to the point where the lower-bounds on n from private learning apply. A consequence of this simple observation is that a generic private everlasting predictor should answer all prediction queries with a single hypothesis - it should modify its hypothesis over time as it processes new queries.

We now take this observation a step further, showing that a private everlasting predictor that answers prediction queries solely based on its training sample S (and randomness, but not on the queries) is subject to the same sample complexity lowerbounds as private learners.

Consider an ( α,β < 1 / 8 , ϵ, δ, n )-everlasting di ff erentially private prediction interface A for concept class C over (finite) domain X that upon receiving the training set S ∈ ( X ×{ 0 , 1 } ) n selects an infinite sequence of hypotheses { hr } r ≥ 0 where hr : X → { 0 , 1 } . Formally, we can think of A as composed of three mechanisms A = ( M 0 , M 1 , M 2 ) where M 0 is ( ϵ,δ )-di ff erentially private:

On input a labeled training sample S ∈ ( X ×{ 0 , 1 } ) n mechanism M 0 computes an initial state and an initial hypothesis ( σ 0 , h 0 ) = M 0 ( S ).

On a query xn + r mechanism M 1 produces an answer M 1 ( xn + r ) = hi ( xn + r ) and mechanism M 2 updates the hypothesis-state pair ( hr +1 , σ r +1 ) = M 2 ( σr ).

Note that as M 0 and M 2 do not receive the sequence { xn + r } r ≥ 0 as input, the sequence { hr } r ≥ 0 depends solely on S . Furthermore as M 1 and M 2 post-process the outcome of M 0 , i.e., the sequence of queries and predictions { ( xr , h r ( xr )) } r ≥ 0 preserves ( ϵ,δ )-di ff erential privacy with respect to the training set S . In Appendix B we prove:

Theorem 3.3. A can be transformed into a ( O ( α ) , O ( β ) , ϵ, δ, O ( n log(1 /β )) -private PAC learner for C .

4 Private everlasting prediction - definition

Theorem 3.3 requires us to seek private predictors whose prediction relies on more information than what is provided by the initial labeled sample. Possibilities include requiring the input of additional labeled or unlabeled examples during the lifetime of the predictor, while protecting the privacy of these examples.

We choose to rely on the queries for updating the predictor's internal state. This introduces a potential privacy risk for these queries as sensitive information about a query may be leaked in the predictions following it. Furthermore, we need take into account that a privacy attacker may choose their queries adversarially and adaptively.

Definition 4.1 (private everlasting black-box prediction) . An algorithm A is an ( α,β,ε,δ,n )-private everlasting black-box predictor for a concept class C if

Prediction: A is an ( α,β,n )-everlasting predictor for C (as in Definition 3.1).

Privacy: For every adversary B and every t ≥ 1, the random variables View 0 B ,t and View 1 B ,t (defined in Figure 1) are ( ε,δ )-indistinguishable.

5 Tools from prior works

We briefly describe tools from prior works that we use in our construction. See Appendix C for a more detailed account.

Algorithm LabelBoost [Beimel et al., 2021]: Algorithm LabelBoost takes as input a partially labeled database S ◦ T ∈ ( X ×{ 0 , 1 , ⊥} ) ∗ (where the first portion of the database, S , contains examples by some concept c ∈ C ) and outputs a similar database where both S and T are (re)labeled by a concept h ∈ C such that error D ( c,h ) is bounded. We use the following lemmata from Beimel et al. [2021]:

Lemma 5.1 (privacy of Algorithm LabelBoost ) . Let A be an ( ϵ,δ ) -di ff erentially private algorithm operating on labeled databases. Construct an algorithm B that on input a partially labeled database S ◦ T ∈ ( X ×{ 0 , 1 , ⊥} ) ∗ applies A on the outcome of LabelBoost ( S ◦ T ) . Then, B is ( ϵ +3 , 4 eδ ) -di ff erentially private.

Lemma 5.2 (Utility of Algorithm LabelBoost ) . Fix α and β , and let S ◦ T be s.t. S is labeled by some target concept c ∈ C , and s.t. | T | ≤ β e VC( C ) exp( α | S | 2VC( C ) ) -| S | . Consider the execution of LabelBoost on S ◦ T , and let h denote the hypothesis chosen by LabelBoost to relabel S ◦ T . With probability at least (1 -β ) we have that error S ( h ) ≤ α .

Parameters: b ∈ { 0 , 1 } , t ∈ N .

Training Phase:

The adversary B chooses two sets of n labeled elements ( x 0 1 , y 0 1 ) , . . . , ( x 0 n , y 0 n ) and ( x 1 1 , y 1 1 ) , . . . , ( x 1 n , y 1 n ), subject to the restriction GLYPH<12> GLYPH<12> GLYPH<12> GLYPH<12> n i ∈ [ n ] : ( x 0 i , y 0 i ) , ( x 1 i , y 1 i ) o GLYPH<12> GLYPH<12> GLYPH<12> GLYPH<12> ∈ { 0 , 1 } .

If ∃ i s.t. ( x 0 i , y 0 i ) , ( x 1 i , y 1 i ) then set Flag = 1. Otherwise set Flag = 0.

Algorithm A gets ( x b 1 , y b 1 ) , . . . , ( x b n , y b n ) and selects a hypothesis h 0 : X →{ 0 , 1 } . \ * the adversary B does not get to see the hypothesis h 0 * \

Prediction phase:

For round r = 1 , 2 , . . . , t :

(a) If Flag = 1 then the adversary B chooses two elements x 0 n + r = x 1 n + r ∈ X . Otherwise, the adversary B chooses two elements x 0 n + r , x 1 n + r ∈ X .

(b) If x 0 n + r , x 1 n + r then Flag is set to 1.

(c) If x 0 n + r = x 1 n + r then the adversary B gets hr -1 ( x b n + r ). \ * the adversary B does not get to see the label if x 0 n + r , x 1 n + r * \

(d) Algorithm A gets x b n + r and selects a hypothesis hr : X →{ 0 , 1 } . \ * the adversary B does not get to see the hypothesis hr * \

Let View b B ,t be B 's view of the execution, i.e., its own randomness and the sequence of predictions in Step 4c.

6 AGeneric Construction

Our generic construction Algorithm GenericBBL transforms a (non-private) learner for a concept class C into a private everlasting predictor for C . The theorem below follows from Theorem 6.2 and Claim 6.3 which are proved in Appendix E.

Theorem 6.1. Given α,β,δ < 1 / 16 , ϵ < 1 , Algorithm GenericBBL is a (4 α, 4 β,ϵ,δ,n ) -private everlasting predictor, where n is set as in Algorithm GenericBBL .

Theorem 6.2 (accuracy of algorithm GenericBBL ) . Given α,β,δ < 1 / 16 , ε < 1 , for any concept c and any round r , algorithm GenericBBL can predict the label of xr as hr ( xr ) , such that Pr[ error D ( c ( xr ) , hr ( xr )) ≤ 4 α ] ≥ 1 -4 β .

Claim 6.3 (privacy of algorithm GenericBBL ) . GenericBBL is ( ε,δ ) -di ff erentially private.

Remark 6.4. For simplicity, we analyzed GenericBBL in the realizable setting, i.e., under the assumption that the training set S is consistent with the target class C . Our construction carries over to the agnostic setting via standard arguments (ignoring computational e ffi ciency). We refer the reader to [Beimel et al., 2021] and [Alon et al., 2020] for generic agnostic-to-realizable reductions in the context of private learning.

6.1 Improving the sample complexity dependency on accuracy

We briefly sketch how to improve the sample complexity of Algorithm GenericBBL from n = ˜ Θ GLYPH<18> VC 2 ( C ) α 2 ϵ 2 GLYPH<19> to n = ˜ Θ GLYPH<18> VC 2 ( C ) αϵ 2 GLYPH<19> by modifying steps 3(d)ii and 3(d)iii of Algorithm GenericBBL . To simplify the description, we consider a constant ϵ , we ignore the privacy amplification by subsampling occurring in steps 2 and 3f and illustrate how the modified algorithm would execute by considering round i = 1 of Algorithm GenericBBL .

Note that S 1 = S where n = | S 1 | = T 1 · λ 1 and λ 1 = ˜ Θ (VC( C ) /α 1 ) is the sample complexity needed such that each of the hypotheses computed in Step 3b has error α 1 except for

Algorithm GenericBBL

Initial input:

A

labeled database

S

∈

(

X

× {

0

,

1

}

)

n

where

n

=

8

α

τ

2

ε

Repeat for i = 1 , 2 , 3 , . . .

(a) Divide Si into Ti = τ · λi · log( 1 δ ) · log 2 ( λ i εα i β i δ ) ε disjoint databases Si, 1 , . . . , S i,T i of size λi .

(b) For t ∈ [ Ti ] let f t ∈ C be a hypothesis minimizing error Si,t ( · ). Define Fi = ( f 1 , . . . , f Ti ).

(c) Set Ri = 12800 | Si | ε . Set the privacy parameters ε ′ i = 1 3 q Ri ln( 2 δ ) and δ ′ i = δ 2 Ri . Instantiate

noises

η

1

, . . . , η

Ri

(d) For ℓ = 1 to Ri :

Receive as input a prediction query xi,ℓ ∈ X .

Compute qxi,ℓ ( Fi ) = 1 Ti P t ∈ [ Ti ] f t ( xi,ℓ ), and let yi,ℓ = qxi,ℓ ( Fi ) + ηℓ .

Respond with the label 0 if yi,ℓ < 1 / 2 and 1 if yi,ℓ ≥ 1 / 2.

(e) Denote Di = ( xi, 1 , . . . , x i,R i ).

(f) Let ˆ Si ⊆ Si and ˆ Di ⊆ Di be random subsets of size ε | Si | 3+exp( ε +4) and ε | Di | 3+exp( ε +4) respectively, and let ˆ S ′ i ◦ ˆ D ′ i ← LabelBoost ( ˆ Si ◦ ˆ Di ). Let Si +1 ⊆ ˆ D ′ i be a random subset of size λi +1 Ti +1 .

(g) Set αi +1 ← αi / 2 and βi +1 ← βi / 2.

probability βi . Applying advanced composition, we get that ˜ Θ ( T 2 1 ) noisy majority queries implemented in steps steps 3(d)ii and 3(d)iii can be performed, i.e., R 1 = ˜ Θ ( T 2 1 ). Finally, to feed the next phase with | S 2 | = Θ ( | S 1 | ) labeled samples, we need that R 1 = Θ ( T 1 · λ 1 ) = ˜ Θ ( T 1 · VC( C ) /α ) and hence T 1 = ˜ Θ ( VC ( C ) /α ) resulting in n = ˜ Θ (VC 2 ( C ) /α 2 ).

However, when queries are made from the same underlying distribution S was selected, we expect that most of them would exhibit a clear majority in steps 3(d)ii and 3(d)iii, except for a fraction of O ( α ). Hence a natural way to improve on the number of majority queries that can be performed is to replace steps 3(d)ii and 3(d)iii with a decision based on the sparse vector technique of Dwork et al. [2009]. In particular, we can use the BetweenThresholds mechanism of Bun et al. [2016] to get an improved R 1 = ˜ Θ ( T 2 1 /α ) and hence get T 1 = ˜ Θ ( VC ( C )) and n = ˜ Θ ( VC 2 /α ).

A final important wrinkle is that the above calculation is based on the queries coming from the same underlying distribution S was selected, while our worst-case privacy requirement allows for an adversarial choice of queries that may in turn cause the execution of BetweenThresholds to halt too often and hence exhaust the privacy budget for the phase. It is hence important to estimate the number of times BetweenThresholds halts within a phase and to stop the execution of Algorithm GenericBBL when the estimate crosses a threshold. This estimate needs to be done in an online manner and should preserve di ff erential privacy with respect to the queries. 9 This can be done, e.g., using the private counter algorithm of Dwork et al. [2010a], which preserves di ff erential privacy under continual observation.

∼

Lap(1

/Ti ε

′

i

).

2

·

References

Noga Alon, Roi Livni, Maryanthe Malliaris, and Shay Moran. Private PAC learning implies finite littlestone dimension. In Moses Charikar and Edith Cohen, editors, Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, STOC 2019, Phoenix, AZ, USA, June 23-26, 2019 , pages 852-860. ACM, 2019. doi: 10.1145/3313276.3316312. URL https://doi.org/10.1145/3313276.3316312 .

Noga Alon, Amos Beimel, Shay Moran, and Uri Stemmer. Closure properties for private classification and online prediction. In COLT , volume 125 of Proceedings of Machine Learning Research , pages 119-152. PMLR, 2020.

Raef Bassily, Abhradeep Guha Thakurta, and Om Dipakbhai Thakkar. Model-agnostic private learning. In NeurIPS , pages 7102-7112, 2018.

Amos Beimel, Kobbi Nissim, and Uri Stemmer. Characterizing the sample complexity of private learners. In ITCS , pages 97-110. ACM, 2013a.

Amos Beimel, Kobbi Nissim, and Uri Stemmer. Private learning and sanitization: Pure vs. approximate di ff erential privacy. In APPROX-RANDOM , pages 363-378, 2013b.

Amos Beimel, Kobbi Nissim, and Uri Stemmer. Learning privately with labeled and unlabeled examples. Algorithmica , 83(1):177-215, 2021.

Gavin Brown, Mark Bun, Vitaly Feldman, Adam D. Smith, and Kunal Talwar. When is memorization of irrelevant training data necessary for high-accuracy learning? In STOC , pages 123-132. ACM, 2021.

Mark Bun and Mark Zhandry. Order-revealing encryption and the hardness of private learning. In TCC (A1) , volume 9562 of Lecture Notes in Computer Science , pages 176-206. Springer, 2016.

Mark Bun, Kobbi Nissim, Uri Stemmer, and Salil P. Vadhan. Di ff erentially private release and learning of threshold functions. In FOCS , pages 634-649, 2015.

Mark Bun, Thomas Steinke, and Jonathan Ullman. Make up your mind: The price of online queries in di ff erential privacy. CoRR , abs/1604.04618, 2016. URL http://arxiv.org/ abs/1604.04618 .

Edith Cohen, Xin Lyu, Jelani Nelson, Tam´ as Sarl´ os, and Uri Stemmer. ˜ Optimal di ff erentially private learning of thresholds and quasi-concave optimization. CoRR , abs/2211.06387, 2022. doi: 10.48550/arXiv.2211.06387. URL https://doi.org/10.48550/arXiv.2211. 06387 .

Yuval Dagan and Vitaly Feldman. PAC learning with stable and private predictions. In COLT , volume 125 of Proceedings of Machine Learning Research , pages 1389-1410. PMLR, 2020.

Cynthia Dwork and Vitaly Feldman. Privacy-preserving prediction. In S´ ebastien Bubeck, Vianney Perchet, and Philippe Rigollet, editors, Conference On Learning Theory, COLT 2018, Stockholm, Sweden, 6-9 July 2018 , volume 75 of Proceedings of Machine Learning Research , pages 1693-1702. PMLR, 2018. URL http://proceedings.mlr.press/v75/ dwork18a.html .

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In TCC , pages 265-284, 2006.

Cynthia Dwork, Moni Naor, Omer Reingold, Guy N. Rothblum, and Salil P. Vadhan. On the complexity of di ff erentially private data release: e ffi cient algorithms and hardness results. In STOC , pages 381-390, 2009.

Cynthia Dwork, Moni Naor, Toniann Pitassi, and Guy N. Rothblum. Di ff erential privacy under continual observation. In Symposium on Theory of Computing (STOC) , pages 715724. ACM, 2010a.

Cynthia Dwork, Guy N. Rothblum, and Salil P. Vadhan. Boosting and di ff erential privacy. In FOCS , pages 51-60, 2010b.

Vitaly Feldman and David Xiao. Sample complexity bounds on di ff erentially private learning via communication complexity. SIAM J. Comput. , 44(6):1740-1764, 2015. doi: 10.1137/140991844. URL http://dx.doi.org/10.1137/140991844 .

Haim Kaplan, Katrina Ligett, Yishay Mansour, Moni Naor, and Uri Stemmer. Privately learning thresholds: Closing the exponential gap. In COLT , volume 125 of Proceedings of Machine Learning Research , pages 2263-2285. PMLR, 2020.

Shiva Prasad Kasiviswanathan, Homin K. Lee, Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. What can we learn privately? SIAM J. Comput. , 40(3):793-826, 2011.

Michael J. Kearns, Mallesh M. Pai, Ryan M. Rogers, Aaron Roth, and Jonathan R. Ullman. Robust mediators in large games. CoRR , abs/1512.02698, 2015.

Frank McSherry and Kunal Talwar. Mechanism design via di ff erential privacy. In FOCS , pages 94-103. IEEE, Oct 20-23 2007.

Anupama Nandi and Raef Bassily. Privately answering classification queries in the agnostic PAC model. In ALT , volume 117 of Proceedings of Machine Learning Research , pages 687-703. PMLR, 2020.

G. Valiant. A theory of the learnable. Commun. ACM , 27(11):1134-1142, November 1984. ISSN 0001-0782. doi: 10.1145/1968.1972. URL http://doi.acm.org/10.1145/1968. 1972 .

Vladimir N. Vapnik and Alexey Y. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and its Applications , 16 (2):264-280, 1971.

A Additional Preliminaries from PAC Learning

It is well know that that a sample of size Θ (VC( C )) is necessary and su ffi cient for the PAC learning of a concept class C , where the Vapnik-Chervonenkis (VC) dimension of a class C is defined as follows:

Definition A.1 (VC-Dimension [Vapnik and Chervonenkis, 1971]) . Let C be a concept class over a domain X . For a set B = { b 1 , . . . , b ℓ } ⊆ X of ℓ points, let Π C ( B ) = { ( c ( b 1 ) , . . . , c ( bℓ )) : c ∈ C } be the set of all dichotomies that are realized by C on B . Wesay that the set B ⊆ X is shattered by C if C realizes all possible dichotomies over B , in which case we have | Π C ( B ) | = 2 | B | .

The VC dimension of the class C , denoted VC( C ), is the cardinality of the largest set B ⊆ X shattered by C .

Theorem A.2 (VC bound) . Let C be a concept class over a domain X . For α,β < 1 / 2 , there exists an ( α,β,n ) -PAC learner for C , where n = 8VC( C ) log( 13 α )+4log( 2 β ) α .

B Proof of Theorem 3.3

The proof of Theorem 3.3 follows from algorithms HypothesisLearner , AccuracyBoost and claims B.1, B.2, all described below.

In Algorithm HypothesisLearner we assume that the everlasting di ff erentially private prediction interface A was fed with n i.i.d. samples taken from some (unknown) distribution D and labeled by an unknown concept c ∈ C . Assumning the sequence of hypotheses { hr } r ≥ 0 produced by A satisfies

we use it to construct - with constant probability - a hypothesis h with error bounded by O ( α ).

Algorithm HypothesisLearner

Parameters: 0 < β ≤ 1 / 8, R = | X | · log( | X | ) · log(1 /β ) Input: hypothesis sequence { } ≥

hr r 0 1. for all x ∈ X let Lx = ∅ 2. for r = 0 , 1 , 2 , . . . , R (a) select x uniformly at random from X and let Lx = Lx ∪{ hr ( x ) } 3. if Lx = ∅ for some x ∈ X then fail, output an arbitrary hypothesis, and halt /* Pr[ ∃ x such that Lx = ∅ ] ≤ | X | (1 -1 | X | ) R ≈ | X | e -R/ | X | = β */ 4. for all x ∈ X let r x be sampled uniformly at random from Lx 5. construct the hypothesis h , where h ( x ) = r x

Proof. Having D , c ∈ C fixed, and given a hypothesis h , we define e h ( x ) to be 1 if h ( x ) , c ( x ) and 0 otherwise. Thus, we can write error D ( c,h ) = E x ∼D [ e h ( x )].

Observe that when Algorithm HypothesisLearner does not fail, r x (and hence h ( x )) is chosen with equal probability among ( h 1 ( x ) , h 2 ( x ) , . . . , hR ( x )) and hence E θ [ e h ( x )] = E i ∈ R [ R ] [ e h i ( x )] where θ denotes the randomness of HypothesisLearner . We get:

By Markov inequality, we have Pr θ [error D ( c,h ) ≥ 8 α ] ≤ 1 / 8. The claim follows noting that Algorithm HypothesisLearner fails with probability at most β ≤ 1 / 8.

The second part of the transformation is Algorithm AccuracyBoost that applies Algorithm HypothesisLearner O (log(1 /β )) times to obtain with high probability a hypothesis with O ( α ) error.

Algorithm AccuracyBoost

Parameters: β , R = 104ln 1 β

Input: R labeled samples with n examples each ( S 1 , . . . , SR ) where Si ∈ ( X ×{ 0 , 1 } ) n

for i = 1 , 2 . . . R

(a) execute A ( Si ) to obtain a hypothesis sequence { h i r } r ≥ 0

(b) execute Algorithm WeakHypothesisLearner on { h i r } r ≥ 0 to obtain hypothesis h i

construct the hypothesis ˆ h , where ˆ h ( x ) = maj( h 1 ( x ) , . . . , h R ( x )).

Proof. Define Bi to be the event where the sequence of hypotheses { h i r } r ≥ 0 produced in Step 1a of AccuracyBoost does not satisfy Equation 1. We have,

Pr[error D ( c,hi ) > 8 α ] ≤ Pr[ B ] + (1 -Pr[ B ]) · Pr[error D ( c,h ) > 8 α ] ≤ β +1 / 4 < 3 / 8 .

Hence, by the Cherno ff bound, when R ≥ 104ln 1 β , we have at least 7 R/ 8 hypotheses are 8 α -good over distribution D . Consider the worst case, in which R/ 8 hypotheses always output wrong labels. To output a wrong label of x , we require at least 3 R/ 8 hypotheses to output wrong labels. Thus h is 24 α -good over distribution D .

C Tools from Prior Works

C.1 Algorithm LabelBoost [Beimel et al., 2021]

Algorithm LabelBoost [Beimel et al., 2021]

Parameters: A concept class C .

Input: A partially labeled database S ◦ T ∈ ( X ×{ 0 , 1 , ⊥} ) ∗ .

% We assume that the first portion of the database (denoted S ) contains labeled examples. The algorithm outputs a similar database where both S and T are (re)labeled.

Initialize H = ∅ .

Let P = { p 1 , . . . , p ℓ } be the set of all points p ∈ X appearing at least once in S ◦ T . Let Π C ( P ) = { ( c ( p 1 ) , . . . , c ( pℓ )) : c ∈ C } be the set of all dichotomies generated by C on P .

For every ( z 1 , . . . , z ℓ ) ∈ Π C ( P ), add to H an arbitrary concept c ∈ C s.t. c ( pi ) = zi for every 1 ≤ i ≤ ℓ .

Choose h ∈ H using the exponential mechanism with privacy parameter ϵ =1, solution set H , and the database S .

(Re)label S ◦ T using h , and denote the resulting database ( S ◦ T ) h , that is, if S ◦ T = ( xi , y i ) t i =1 then ( S ◦ T ) h = ( xi , y ′ i ) t i =1 where y ′ i = h ( xi ).

Output ( S ◦ T ) h .

Lemma C.1 (privacy of Algorithm LabelBoost [Beimel et al., 2021]) . Let A be an ( ϵ,δ ) -di ff erentially private algorithm operating on partially labeled databases. Construct an algorithm B that on input a partially labeled database S ◦ T ∈ ( X ×{ 0 , 1 , ⊥} ) ∗ applies A on the outcome of LabelBoos ( S ◦ T ) . Then, B is ( ϵ +3 , 4 eδ ) -di ff erentially private.

Consider an execution of LabelBoost on a database S ◦ T , and assume that the examples in S are labeled by some target concept c ∈ C . Recall that for every possible labeling ⃗ z of the elements in S and in T , algorithm LabelBoost adds to H a hypothesis from C that agrees with ⃗ z . In particular, H contains a hypothesis that agrees with the target concept c on S (and on T ). That is, ∃ f ∈ H s.t. error S ( f ) = 0. Hence, the exponential mechanism (on Step 4) chooses (w.h.p.) a hypothesis h ∈ H s.t. error S ( h ) is small, provided that | S | is roughly log | H | , which is roughly VC( C ) · log( | S | + | T | ) by Sauer's lemma. So, algorithm LabelBoost takes an input database where only a small portion of it is labeled, and returns a similar database in which the labeled portion grows exponentially.

Lemma C.2 (utility of Algorithm LabelBoost [Beimel et al., 2021]) . Fix α and β , and let S ◦ T be s.t. S is labeled by some target concept c ∈ C , and s.t.

Consider the execution of LabelBoost on S ◦ T , and let h denote the hypothesis chosen on Step 4. With probability at least (1 -β ) we have that error S ( h ) ≤ α .

D Some Technical Facts

We refer to the execution of steps 3a-3g of algorithm GenericBBL as a phase of the algorithm, indexed by i = 1 , 2 , 3 , . . . .

In GenericBBL , we require that the set of all predictions in the phase i is (1 , δ )-di ff erentially private.

Claim D.1. For δ < 1 , the composited Laplace mechanism used in step 3c in the i -th iteration, is (1 , δ ) -di ff erentially private.

Proof. Let ε ′ i , δ ′ i be as in Step 3c. Since e ε ′ i -1 < 2 ε ′ i for 0 < ε ′ i < 1, we have

The proof is concluded by using advanced composition (Theorem 2.6).

In step 3f, GenericBBL takes a random subset of size λi +1 Tt +1 from ˆ D ′ i . We show that the size of ˆ D ′ i is at least λi +1 Tt +1 .

Claim D.2. When ε ≤ 1 , for any i ≥ 1 , we always have | ˆ D ′ i | ≥ λi +1 Ti +1 .

Proof. Let m = 3+exp( ε +4) < 200. By the step 3c, step 3e and step 3f, | ˆ Dj | = ε | Dj | m = 12800 | Sj | m ≥ 64 | Sj | = 64 λj T j . Then it is su ffi cient to verify 128 λj T j ≥ λj +1 Tj +1

We can verify that

and

The last inequalitu holds because λj ≥ 4 and αj , β j ≤ 1 / 2.

To apply the privacy and accuracy of LabelBoost , the sizes of the databases need to satisfy the inequalities in lemma C.2. We verify that in each phase, the sizes of the databases always satisfy the requirement.

Claim D.3. When ε ≤ 1 , for any i ≥ 1 , we have | ˆ Di | ≤ βi e VC( C ) exp GLYPH<18> αi | ˆ Si | 2VC( C ) GLYPH<19> -| ˆ Si | .

Proof. By claim D.2, step 3c and step 3f,

and

Note that

for Ti = τ · λi · log( 1 δ ) · log 2 ( λ i εα i β i δ ) ε , the inequality holds when τ ≥ 50.

E Accuracy of Algorithm GenericBBL - proof of Theorem 6.2

We refer to the execution of steps 3a-3g of algorithm GenericBBL as a phase of the algorithm, indexed by i = 1 , 2 , 3 , . . . .

We give some technical facts in Appendix D. In Claim E.1, we show that in each phase, samples are labeled with high accuracy. In Claim E.3, we prove that algorithm GenericBBL fails with low probability. In Claim E.4, we prove that algorithm GenericBBL predict the labels with high accuracy.

Claim E.1. For each phase i we have

Proof. The proof is by induction on i . The base case for i = 1 is trivial, with g 1 = c . Assume the claim holds for all j ≤ i . By the properties of LabelBoost (Lemma C.2) and Claim D.3, with probability at least 1 -βi we have that Si is labeled by a hypothesis gi ∈ C s.t. error Si -1 ( gi -1 , g i ) ≤ αi . Observe that the points in Si (without their labels) are chosen i.i.d. from D , and hence, By Theorem A.2 (VC bounds) and | Si -1 | ≥ 128 λi -1 ≥ λi , with probability at least 1 -βi we have that error D ( gi , g i -1 ) ≤ αi . Hence, with probability 1 -2 βi , we have error D ( gi , g i -1 ) ≤ αi . Finally, by the triangle inequality, error D ( gi , c ) ≤ P i j =1 αj , except with probability 2 P i j =1 βj

Combining claims E.1 union bound, we get:

Claim E.2. Let D be an underlying distribution and let c ∈ C be a target concept. Then

For each phase i , if there exists a noise ηj ≥ 1 / 16 in step 3c, we say the phase i fails. Define the following good event.

Event E 1 : Phase i doesn't fail for all i ≥ 1.

Claim E.3. When λ ≥ 1 , ε ≤ 1 , α ≤ 1 / 2 , β ≤ 1 / 2 , δ ≤ 1 / 2 , event E 1 occurs with probability at least 1 -β .

Proof. In each phase i , For each

Pr[ | η | ≥ 1 / 16] = e -Ti ε ′ i / 16 = e -48 √ R i ln(2 /δ ) ≤ βi /R i . This inequality holds when τ ≥ bound, we have Pr[phase i fails] ≤ βi .

Using to union bound,

ηj , by the property of Laplace distribution, we have T i 1. By union

Notations. Consider the i th phase of Algorithm GenericBBL , and focus on the j -th iteration of Step 3. Fix all of the randomness of noises. Now observe that the output on step 3(d)iii is a deterministic function of the input xi,j . This defines a hypothesis which we denote as hi,j .

Si,1

f

Gx;(F)

Adding Laplace noise

Label

SiT;

f;

hi, (fix the randomness)

Claim E.4. For β < 1 / 16 , with probability at least 1 -4 β , all of the hypotheses defined above are 4 α -good w.r.t. D and c .

Proof. In the phase i , by Claim E.2, with probability at least 1 -2 β we have that Si is labeled by a hypothesis gi ∈ C satisfying error D ( gi , c ) ≤ α . We continue with the analysis assuming that this is the case.

On step 3a of the i th phase we divide Si into Ti subsamples of size λi each, identify a consistent hypothesis f t ∈ C for every subsample Si,t , and denote Fi = ( f 1 , . . . , f T ). By Theorem A.2 (VC bounds), every hypothesis in Fi satisfies error D ( f t , g i ) ≤ α with probability 1 -βi , in which case, by the triangle inequality we have that error D ( f t , c ) ≤ 2 α .

Set Ti ≥ 512(1 -4 βi ) ln( 1 β i ) (1 -64 βi ) 2 , using Cherno ff bound, it holds that for at least 15 Ti / 16 of the hypotheses in Fi have error error D ( f t , c ) ≤ 2 α with probability at least 1 -βi .

By Claim E.3, with probability 1 -β , all Laplace noises added in step 3d is less than Ti 16 . Let m : X →{ 0 , 1 } defined as m ( x ) = maj f t ∈ Fi ( f t ( x )). For m to err on a point x (w.r.t. the target concept c ), it must be that at least 3 / 8-fraction of the 2 α -good hypotheses in ˆ Fi err on x . Consider the worst case in Figure 3, based on Claim E.3, we have error D ( m,c ) ≤ 4 α with probability 1 -β .

High error hypotheses: 1/16

Laplacenoises:1/16

ErrorofPAC

learned

hypotheses:

3/8

Errorof PAC

learned

hypotheses:

3/8

ErrorofPAClearned

hypotheses:2/8

Conclude all above, with probability 1 -4 β , all the hypotheses are 4 α -good.

E.1 Privacy analysis - proof of Claim 6.3

Fix t ∈ N and the adversary B . We need to show that View 0 B ,t and View 1 B ,t (defined in Figure 1) are ( ε,δ ) -indistinguishable . We will consider separately the case where the executions di ff er in the training phase (Claim E.5) and the case where the di ff erence occurs during the prediction phase (Claim E.6).

Privacy of the initial training set S . Let S 0 , S 1 ∈ ( X ×{ 0 , 1 } ) n be neighboring datasets of labeled examples and let View 0 B ,t and View 1 B ,t be as in Figure 1 where GLYPH<16> ( x 0 1 , y 0 1 ) , . . . , ( x 0 n , y 0 n ) GLYPH<17> = S 0 and GLYPH<16> ( x 1 1 , y 1 1 ) , . . . , ( x 1 n , y 1 n ) GLYPH<17> = S 1 .

Claim E.5. For all adversaries B , for all t > 0 , and for any two neighbouring database S 0 and S 1 selected by B , View 0 B ,t and View 1 B ,t are ( ε,δ ) -indistinguishable.

Proof. Let R ′ 1 = min( t, R 1 ). Note that View b B ,R ′ 1 is a prefix of View b B ,t which includes the labels Algorithm GenericBBL produces in Step 3(d)iii for the R ′ 1 first unlabeled points selected by B . Let S b 2 be the result of the first application of algorithm LabelBoost in Step 3f of GenericBBL (if t < R 1 we set S b 2 as ⊥ ). The creation of these random variables is depicted in Figure 4, where D L 1 denotes the labels Algorithm GenericBBL produces for the unlabeled points D 1 .

Observe that View b B ,t results from a post-processing (jointly by the adversary B and Algorithm GenericBBL ) of the random variable GLYPH<18> View b B ,R ′ 1 , S b 2 GLYPH<19> , and hence it su ffi ces to show that GLYPH<18> View 0 B ,R ′ , S 0 2 GLYPH<19> and GLYPH<18> View 1 B ,R ′ , S 1 2 GLYPH<19> are ( ε,δ )-indistinguishable.

We follow the processes creating View b B ,t and S b 2 in Figure 4: (i) The mechanism M 1 corresponds to the loop in Step 3d of GenericBBL where labels are produced for the adver-

M2

S

S

S1

S2

D

M

M3

D1

D

sarially chosen points D b 1 . By application of claim D.1, M 1 is (1 , δ )-di ff erentially private. (ii) The mechanism M 2 , corresponds to the subsampling of ˆ S b 1 from S b 1 and the application of procedure LabelBoost on the subsample in Step 3f of GenericBBL resulting in S b 2 . By application of Claim 2.9 and Lemma C.1, M 2 is ( ε, 0)-di ff erentially private. Thus ( M 1 , M 2 ) is ( ε +1 , δ )-di ff erentially private. (iii) The mechanism M 3 with input of S b and output GLYPH<16> D b,L 1 , S b 2 GLYPH<17> = GLYPH<18> View b B ,R ′ 1 , S b 2 GLYPH<19> applies ( M 1 , M 2 ) on the sub-sample S b 1 obtained from S b in Step 2 of GenericBBL . By application of Claim 2.9 M 3 is ( ε, 4 εδ 3+exp( ε +1) )-di ff erentially private. Since 4 εδ 3+exp( ε +1) ≤ δ for any ε , hence GLYPH<18> View 0 B ,R ′ 1 , S 0 2 GLYPH<19> and GLYPH<18> View 1 B ,R ′ 1 , S 1 2 GLYPH<19> are ( ε,δ )-indistinguishable

Privacy of the unlabeled points D . Let D 0 , D 1 ∈ X t be neighboring datasets of unlabeled examples and let View 0 B ,t and View 1 B ,t be as in Figure 1 where GLYPH<16> x 0 1 , . . . , x 0 t GLYPH<17> = D 0 and GLYPH<16> x 1 1 , . . . , x 1 t GLYPH<17> = D 1 .

Claim E.6. For all adversaries B , for all t > 0 , and for any two neighbouring databases D 0 and D 1 selected by B , View 0 B ,t and View 1 B ,t are ( ε,δ ) -indistinguishable.

M2

's

S

Sit1

Si+2

D

M4

M

DL

i计1

D

D

Dit1

M3

Proof. Let D 0 1 , D 0 2 , . . . , D 0 k and D 1 1 , D 1 2 , . . . , D 1 k be the set of unlabeled databases in step 3e of GenericBBL. Without loss of generality, we assume D 0 i and D 1 i di ff er on one entry. When

i = k , View 0 B ,t = View 1 B ,t because all selected hypothesis are the same. When i < k , let R ′ = min GLYPH<16>P i +1 j =1 Rj , t GLYPH<17> .

Similar to the analysis if Claim E.5, View b B ,t results from a post-processing of the random variable (View b B ,R ′ , S b i +2 ) (if t < P i +1 j =1 Rj we set S b i +2 as ⊥ ). Note that View b B ,R ′ 1 = ( D b,L 1 , . . . , D b,L ∗ i , D b,L i +1 ), and ( D b,L 1 , . . . , D b,L i -1 , D b,L ∗ i ) follow the same distribution for b ∈ { 0 , 1 } , where D b,L ∗ i is the labels of points in D b i expect the di ff erent point. So that it su ffi ces to show that GLYPH<16> D 0 ,L i +1 , S 0 2 GLYPH<17> and GLYPH<16> D 1 ,L i +1 , S 1 2 GLYPH<17> are ( ε,δ )-indistinguishable.

We follow the processes creating D b,L i +1 and S b i +2 in Figure 5: (i) The mechanism M 1 corresponds to the loop in Step 3d of GenericBBL where labels are produced for the adversarially chosen points D b i +1 . By application of claim D.1, M 1 is (1 , δ )-di ff erentially private. (ii) The mechanism M 2 , corresponds to the subsampling of ˆ S b i +1 from S b i +1 and the application of procedure LabelBoost on the subsample in Step 3f of GenericBBL resulting in S b i +2 . By application of Claim 2.9 and Lemma C.1, M 2 is ( ε, 0)-di ff erentially private. Thus ( M 1 , M 2 ) is ( ε +1 , δ )-di ff erentially private. (iii) The mechanism M 3 with input of ˆ D b i and output GLYPH<16> D b,L i +1 , S b i +2 GLYPH<17> applies ( M 2 , M 3 ) on Si +1 , which is generated from ˆ D b i and in Step 3f of GenericBBL . By application of Claim C.1, M 3 is ( ε +4 , 4 εδ )-di ff erentially private. (iv) The mechanism M 4 , corresponds to the subsampling ˆ D b i from D b i and the application of M 4 on ˆ D b i . By application of Claim 2.9, M 4 is ( ε, 16 eεδ 3+exp( ε +4) )-di ff erentially private. Since 16 eε 3+exp( ε +4) ≤ 1 for any ε , GLYPH<16> D 0 ,L i +1 , S 0 2 GLYPH<17> and GLYPH<16> D 1 ,L i +1 , S 1 2 GLYPH<17> are ( ε,δ )-indistinguishable.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes the notion of private everlasting prediction. Given a training dataset, the predictor responds to a sequence of queries and privacy has to be preserved for both the training data and all queries. The authors explore the PAC learnability problem under this model and show that the sample complexity scales quadratically with the VC dimension through a generic construction from non-private learners.\n\nSTRENGTHS:\n1. The authors formulate a new notion of private everlasting prediction. This is an original theoretical concept which extends the single query prediction model and has practical relevance.\n2. The authors prove several interesting theoretical properties for private everlasting prediction. First it requires that the hypothesis needs to change over time. Second, the authors show that the sample complexity scales quadratically with the VC dimension of the concept class. This is a significant improvement compared to private learning which is impossible over infinite domains.\n3. The writing is clear and easy to follow.\n\nUpdate: increased my score to seven after seeing the authors' response and other reviewer comments.\n\nWEAKNESSES:\n1. In the algorithm GenericBBL, $\\tau>1.1\\times 10^10$ seems too large. Can the constant be made any smaller?\n2. The algorithm is not computationally efficient.\n3. In terms of writing, the authors could provide a brief overview of the proof ideas for Theorem 5.1.\n\nQUESTIONS:\nIn practice, some large models are pretrained on public data which can be considered non-private. Theoretically, is it relevant to consider the case when the training set $S$ is non-private and the stream of queries is private? How would the sample complexity change?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses private everlasting prediction, which extends private prediction to answer an unlimited sequence of prediction queries. The goal is to present a generic private everlasting predictor with low training sample complexity. The paper introduces definitions for everlasting prediction and everlasting differentially private prediction interfaces. It then presents a generic construction called GenericBBL for private everlasting prediction.\n\nSTRENGTHS:\n1. This paper introduces a formal framework for everlasting prediction.\n2. This paper provided a comprehensive privacy analysis to the proposed privacy-preserving everlasting prediction task.\n\nWEAKNESSES:\n1. Limited applicability: The paper focuses on the theoretical aspects of private everlasting prediction and does not provide concrete practical applications or empirical evaluations. It remains to be seen how well the proposed approach translates into real-world scenarios.\n2. Lack of comparative experimental analysis: The excerpt does not mention any comparison or benchmarking against existing methods or alternative approaches. Without such comparisons, it is difficult to assess the novelty or superiority of the proposed method.\n3. Lack of empirical validation: The excerpt does not mention any empirical experiments, simulations, or case studies to validate the effectiveness or robustness of the proposed private everlasting predictor. It is unclear how the proposed construction performs in terms of privacy preservation, utility, and sample complexity compared to other techniques.\n4. The contents shown in the supplementary materials contain a lot of redundant information compared with the paper.\n\nQUESTIONS:\nWhat is the application scenario of the everlasting prediction?\n\nLIMITATIONS:\nIt lacks experimental analysis of the proposed algorithm, which makes it hard to justify the utility of the proposed algorithm.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides an intriguing path to evading known lower bounds for differentially private PAC learning. Whereas nonprivately the sample complexity of learning is proportional to the VC dimension, the private sample complexity for (pure) DP is characterized by the representation dimension, which can be much larger. In particular, some natural classes such as threshold functions over an infinite domain (e.g. Z or R) have finite VC dimension but infinite representation dimension, so they can be learned nonprivately but not privately (even with approximate DP, by a separate result).\n\nThis paper shows that this impossibility only holds for privately releasing a hypothesis and not for privately classifying samples. In particular, it shows that in the online prediction setting, for any hypothesis class there is a generic stateful algorithm with sample complexity based on the VC dimension (to be precise, quadratic in the VC dimension) that can privately answer an unbounded sequence of iid queries with the same accuracy guarantee as PAC learning, but where the state of the algorithm remains hidden and only the query labels are revealed. The algorithm is allowed to remember previous queries but must be differentially private with respect to them as well as the points of the original (labeled) training set. It also shows that statefulness is necessary to achieve this result.\n\nFor general hypothesis classes the algorithm is inefficient, but it can be made efficient on important special cases, including threshold functions.\n\nSTRENGTHS:\nThis is a very interesting result that enhances our understanding of differentially private learning by bypassing known lower bounds. It essentially gives a separation between interactive and non-interactive private prediction, showing that interactivity reduces the private sample complexity to nearly that of nonprivate learning (as long as we only have to output predictions and not a hypothesis). The paper is quite well-written.\n\nWEAKNESSES:\nThere are two obvious limitations of the main result, that yield very interesting open problems posed in the paper: whether it's possible to come up with a similar algorithm that is computationally efficient for all hypothesis classes, and whether it's possible to improve the sample complexity from VC^2 to VC.\n\nMinor corrections:\n104: points --> point\n147: \"upto\" should be \"up to\"\n\nQUESTIONS:\n1) Is it strictly necessary for accuracy that the query points are iid? Or could we hope for the following stronger statement: \n\nGiven query points x_1, x_2, ... and a subset of indices I (not known to the algorithm) such that x_i is sampled from the distribution for i\\in I but is chosen by the adversary for i\\notin I, we require accuracy only for the points in I. \n\nThat is, as long as the challenge queries are sampled from the distribution, we want to be robust to data poisoning using the other queries. Does this seem like it could be possible, or is it too strong?\n\n2) Minor clarification: In the technical overview on page 3-4 it sounds like the parameters discussed are based on the standard composition bound, without using BetweenThresholds. It might be helpful to clarify whether this is the case and if we get a similar or only slightly weaker result without using BetweenThreshold, or if we truly need the tighter privacy analysis of BetweenThreshold.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies differentially private prediction. It has two major contributions: \n\na) Prediction corresponds to being given an initial labeled training set, and then subsequently making predictions on other data points based on it. This paper shows that differentially private prediction can be performed on an unbounded number of queries, with strong accuracy guarantees, with the initial training set sample complexity scaling only as a polynomial in the VC dimension of the hypothesis class. Prior work either studied prediction for a small number of queries or studied the more stringent task of private PAC learning where significantly stronger lower bounds are known (PAC learning asks for the release of an entire model as opposed to predictions alone).  \n\nb) Since queries correspond to data points, they are usually sensitive user information. The authors formalize a model of privacy (similar to joint differential privacy) that gives a meaningful notion of privacy for these data points, even when adversaries can choose the queries adaptively. Their algorithms satisfy this notion of privacy.\n\nSTRENGTHS:\nThis paper makes a valuable and somewhat surprising discovery by proving that differentially private prediction can be performed on an unbounded number of queries, with no training set sample complexity dependence on the number of queries! They leverage standard techniques such as sparse vector in clever ways to do so. They operate in rounds, and use queries themselves as data points for future rounds, calling upon techniques from the semi-supervised differentially private learning literature . Their results suggest new ways to get around lower bounds for differential privacy.\n\nTheir algorithms involve reducing private prediction to non-private PAC learning, and while their techniques are not yet practical in many cases (because of time complexity), their reduction can be readily extended to any non-private learning algorithm, and hence there is lots of potential for future work to come up with ways to make their techniques more practical.\n\nWEAKNESSES:\nOne (minor) drawback is that the notion of privacy defined for the adaptively chosen queries is (necessarily) not as strong as would be nice since the label for a data point needs to depend on the data point to achieve any reasonable notion of accuracy (since otherwise you can only do something like randomized response).\n\nQUESTIONS:\n1) The reasoning for the use of LabelBoost didnt entirely make sense to me- the stated reason was that the predictions provided during a round are not consistent with any concept in the hypothesis class. However, it’s not immediately clear to me that this is a problem- for example, using agnostic (non-private) learning algorithms may get around this issue (at the expense of worse sample complexity dependence on the accuracy parameter). More explanation of this would be useful.\n\n2) The comment about their model of prediction, in principle, allowing memorization was interesting. It’s not clear that this is true- it seems like memorizing something in the training set would affect future predictions which would affect the adversary’s views. Would love more clarification on this.\n\n3) Is there a simple (eps, 0)- version of this algorithm? It’s not clear to me that approx DP is necessary (sparse vector by itself is pure DP, though a variant is used in this paper).\n\n4) The diagrams in the privacy proofs are a little too convoluted to be useful to the reader. I don’t have great suggestions on how to simplify them, but the privacy proofs by themselves are relatively simple, and I’m not sure the diagrams make them simpler to understand.\n\n5) Some minor comments:\n\na) In Algorithm LabelBoost step 4, ‘choose’ based on exponential mechanism is not well defined since the score function is not specified (it can be inferred from context that it’s the negative of empirical risk) but specifying would be good.\n\nb) In Claim E.4 don’t you need to account for the fact that the function labeling the sample is not the real function, but rather the one chosen by LabelBoost? Specifically, Claim E.3 is about a function with 0 sample error, whereas LabelBoost will incur some error, and hence I think you want a version of claim E.3 where sample error is bounded by alpha.\n\nc) On page 8, where it says ‘using standard privacy amplification arguments, BetweenThresholds can be modified to allow for c times of outputting bot before halting)’, I think you mean privacy composition.\n\nLIMITATIONS:\nLimitations adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}