Title: Oracle-Efficient Differentially Private Learning with Public Data
Venue: NeurIPS 2024
Accepted: yes
Keywords: Oracle Efficiency

Differential Privacy

PAC learning
Abstract: Due to statistical lower bounds on the learnability of many function classes under privacy constraints, there has been recent interest in leveraging public data to improve the performance of private learning algorithms. In this model, algorithms must always guarantee differential privacy with respect to the private samples while also ensuring learning guarantees when the private data distribution is sufficiently close to that of the public data.  Previous work has demonstrated that when sufficient public, unlabelled data is available, private learning can be made statistically tractable, but the resulting algorithms have all been computationally inefficient.  In this work, we present the first computationally efficient, algorithms to provably leverage public data to learn privately whenever a function class is learnable non-privately, where our notion of computational efficiency is with respect to the number of calls to an optimization oracle for the function class.  In addition to this general result, we provide specialized algorithms with improved sample complexities in the special cases when the function class is convex or when the task is binary classification.

Oracle-Efficient Differentially Private Learning with Public Data

Abstract

Due to statistical lower bounds on the learnability of many function classes under privacy constraints, there has been recent interest in leveraging public data to improve the performance of private learning algorithms. In this model, algorithms must always guarantee differential privacy with respect to the private samples while also ensuring learning guarantees when the private data distribution is sufficiently close to that of the public data. Previous work has demonstrated that when sufficient public, unlabelled data is available, private learning can be made statistically tractable, but the resulting algorithms have all been computationally inefficient. In this work, we present the first computationally efficient, algorithms to provably leverage public data to learn privately whenever a function class is learnable non-privately, where our notion of computational efficiency is with respect to the number of calls to an optimization oracle for the function class. In addition to this general result, we provide specialized algorithms with improved sample complexities in the special cases when the function class is convex or when the task is binary classification.

1 Introduction

Differential privacy (DP) [Dwork et al., 2006] is a standard guarantee of individual-level privacy for statistical data analysis. Algorithmic research on differential privacy aims to understand what statistical tasks are compatible with the definition, and at what cost, e.g., in terms of sample complexity or computational efficiency. Unfortunately, it is known that some tasks may become more expensive or outright impossible to conduct with differential privacy. For example, in the setting of binary classification, there is no differentially private algorithm for solving the simple problem of learning a one-dimensional classifier over the real numbers [Bun et al., 2015, Alon et al., 2019].

Motivated in part by such barriers to full-fledged private learning, many papers have considered relaxing the model to allow the use of auxiliary 'public' data Balcan and Feldman [2013], Bassily et al. [2019, 2020b, 2022, 2023], Kairouz et al. [2021], Amid et al. [2022], Lowy et al. [2023]. Such data may be available if individuals can voluntarily opt-in to share or sell their information to enable a particular task. Alternatively, a data analyst might have background knowledge about the underlying data distribution from the results of previous analyses, or hold a plausible generative model for it. These situations are captured by semi-private learning, first discussed by Balcan and Feldman [2013], formally introduced by Beimel et al. [2014] and subsequently studied by Bassily et al. [2019], Hopkins et al. [2024]. In this model, a learning algorithm is given n 'private' samples from a joint distribution D over example-label pairs, as well as m unlabeled 'public' samples from the same marginal distribution over examples. The algorithm must be differentially private with respect to its private dataset, but can depend arbitrarily on its public samples. For learning a binary classifier over a class F with a VC-dimension vc ( F ) , these papers showed that in the presence of O ( vc ( F )) public unlabeled samples, every concept class F is agnostically learnable with O ( vc ( F )) private labeled samples, matching what is achievable without privacy guarantees.

While these results essentially resolve the statistical complexity of semi-private learning, they do not address the question of computational efficiency. These algorithms proceed by drawing enough public samples to construct a cover for the class F with respect to the target marginal distribution on examples, and then using the exponential mechanism [McSherry and Talwar, 2007] to select a hypothesis from this cover that fits the private dataset. As the size of this cover is exponential in vc ( F ) , constructing it explicitly is computationally expensive. This paper aims to address the following question: Is such computational overhead really necessary if F exhibits additional structure that make non-private learning tractable?

In this work, we give new semi-private learners that are efficient whenever fast non-private algorithms are available. More specifically, our main result is generic semi-private algorithms for regression and classification that are oracle-efficient in that they run in polynomial time given an oracle solving the non-private empirical risk minimization problem for F , and have sample complexity polynomial in the usual parameters such as Gaussian complexity and VC dimension.

Theorem 1 (Informal version of Theorem 2) . Fix a function class F : X → [ -1 , 1] . Then there is an oracle-efficient, ( ε, δ ) -differentially private algorithm (Algorithm 2) using poly (sup m G m ( F )) labeled private samples (where, G m ( F ) denotes the Gaussian complexity of function class F ), unlabeled public samples, and calls to an empirical risk minimization oracle for F that learns an approximately optimal predictor ˆ f ∈ F .

While Theorem 1 captures extremely broad learning settings, the polynomials governing its sample complexity are rather large. We identify several important cases in which the sample complexity can be improved and the number of oracle calls is only 2. In the case where the function class F is convex, we give a variant of Algorithm 2, inspired by follow-the-regularized-leader, with significantly improved sample complexity as a function of the desired error (Theorem 3, Algorithm 3). Finally, in the special case of binary classification (i.e., Boolean F under the 0-1 loss), we give a completely different oracle-efficient algorithm with improved sample complexity (Theorem 4, Algorithm 4), which requires the private sample size to grow at the rate of O (( vc ( F )) 2 ) . Prior work of Bassily et al. [2018] gave an oracle-efficient algorithm in this setting with somewhat better sample complexity than ours, based on a reduction to private classification. Meanwhile, our algorithm has the advantage of being able to guarantee pure (rather than only approximate) differential privacy algorithm, as well as making only two oracle calls as opposed to the polynomially many as a function of ε and the target accuracy. Our results in the binary classification setting can also be viewed as an extension of Neel et al. [2019], which gives oracle-efficient private learners for structured function classes F that have a small universal identification set [Goldman et al., 1993]. Our results relax this stringent combinatorial condition by leveraging a small public unlabelled dataset, which allows us to design an oracle-efficient private learner for any function class F with bounded VC-dimension.

In fact, our results also address a somewhat more general setting than the semi-private model described so far. Specifically, our results automatically handle the setting where there may be a bounded distribution shift between the public and private data. In particular, all of our results hold as long as the public unlabelled data distribution and the private marginal distribution over the fea-

ture space have a density ratio bounded by σ . 1 The standard semi-private setting corresponds to the special case where σ = 1 . Taking this view, we can interpret our results as oracle-efficient private learning in the smoothed learning setting. Our algorithms achieve accurate learning provided that the private marginal distribution does not deviate too much from a public reference distribution. However, our privacy guarantees hold even if the private data distribution has unbounded distribution shift from the public data.

1.1 Related Work

Our work brings together ideas and techniques from multiple literatures.

Oracle efficiency in private and online learning. Our notion of oracle-efficiency is standard in (theoretical) machine learning to model reductions in a world where worst-case hardness abounds, but optimization heuristics (e.g., integer programming solvers, non-convex optimization) often enjoy success. Within the differential privacy literature, oracle-efficient algorithms are known for binary classification with classes F that admits small universal identification sets [Neel et al., 2019], synthetic data generation [Gaboardi et al., 2014, Nikolov et al., 2013, Neel et al., 2019, Vietri et al., 2020], and certain types of non-convex optimization problems [Neel et al., 2020]. Oracle-efficiency is also well-established approach in online learning [Kalai and Vempala, 2005, Hazan and Koren, 2016, Kozachinskiy and Steifer, 2023, Haghtalab et al., 2022a, Block and Simchowitz, 2022, Block and Polyanskiy, 2023, Block et al., 2023a,b], an area with deep connections to differential privacy [Alon et al., 2019, Abernethy et al., 2019, Bun et al., 2020, Ghazi et al., 2021]. Indeed, our new semi-private learning algorithm Algorithm 2 adapts a follow-the-perturbed-leader inspired algorithm [Block et al., 2022] from the setting of smoothed online learning.

DP learning and release with public (unlabelled) data Our results contribute to a long line of theoretical work that leverages public data for private data analysis. In particular, our work provides general computationally efficient algorithms (in the oracle efficiency sense) for semi-private learning [Beimel et al., 2014]. In addition to the work in this direction we discussed above, several recent papers Bassily et al. [2022, 2023] developed efficient algorithms for private learning with domain adaptation from a public source. That work accommodates a more general notion of distribution shift than ours, but makes essential use of labeled public data, as well as handling only restricted concept classes or loss functions. There has also been work that leverages public data to remove statistical barriers in private query release [Bassily et al., 2020a] and density estimation Bie et al. [2022], Ben-David et al. [2023]. Papernot et al. [2018], Yu et al. [2022], Golatkar et al. [2022], Zhou et al. [2021] and Liu et al. [2021a,b] give empirical guarantees to the problem of private learning and private synthetic data from public samples respectively.

Smoothed Analysis in Online Learning. Smoothed analysis was pioneered in Spielman and Teng [2004] for the purpose of explaining the empirical success of algorithms whose worst-case behavior is provably intractable. More recently, the framework has come to online learning [Rakhlin et al., 2011, Haghtalab et al., 2020, 2022b, Block et al., 2022, Haghtalab et al., 2022a, Block and Simchowitz, 2022, Block et al., 2023a,b] in order to circumvent the strong statistical [Rakhlin et al., 2015] and computational [Hazan and Koren, 2016] lower bounds that worst-case data can induce. The assumption of smoothness has also been used in learning more broadly [Durvasula et al., 2023, Cesa-Bianchi et al., 2023] and its assumptions have been relaxed [Block and Polyanskiy, 2023].

2 Preliminaries

In this section, we formally introduce our setting. Let X denote the feature space and Y be the label space. In general, we consider Y = [ -1 , 1] , but in the special case of binary classification setting, we have Y = { 0 , 1 } . In general, we study learning algorithms A that map a dataset D with n examples from X × Y to a predictor in a function class F . We require A to satisfy differential privacy , defined below.

Definition 1 (Differential Privacy Dwork et al. [2006]) . Let A : ( X × Y ) n →F be a randomized algorithm and D , D ′ ∈ ( X × Y ) n be data sets. We say that D and D ′ are neighboring if |D \ D ′ | = |D ′ \ D| ≤ 1 , i.e. they differ in at most one datum. We say that A is ( ε, δ ) -differentially private if for all neighboring datasets D , D ′ , and for all measurable G ⊂ F , it holds that P ( A ( D ) ∈ G ) ≤ e ε · P ( A ( D ′ ) ∈ G ) + δ . If δ = 0 , we say that A is ε -(purely) differentially private.

As defined, it is trivial to construct algorithms that are differentially private by outputting functions independent of the data set; for an algorithm to be useful, however, we also require that it learns in a meaningful sense. Thus, in the context of learning, we consider the following accuracy desideratum.

Definition 2. Let A : ( X × Y ) n → F be a randomized algorithm. We say that A is an ( α, β ) -learner with respect to a measure ν on X × Y and loss function L : F → [ -1 , 1] if, for D sampled independently from ν , it holds that P ( L ( A ( D )) ≤ inf f ∈F L ( f ) + α ) ≥ 1 -β . For regression problems, we consider the loss function L to be induced by a function ℓ : Y × Y → [0 , 1] , convex and λ -Lipschitz in the first argument, such that L ( f ) = E ( X,Y ) ∼ ν [ ℓ ( f ( X ) , Y )] .

For simplicity, we will denote the empirical loss on a data set D as L D ( f ) = 1 n · ∑ ( X i ,Y i ) ∈D ℓ ( f ( X i ) , Y i ) . We emphasize that in contradistinction to the standard notion of PAClearnability [Valiant, 1984], our requirement is weaker in that we only require distribution-dependent learning, i.e., the algorithm A is allowed to depend on ν in some to-be-specified way. This is necessary in our setting as it is well known that distribution-independent differentially private PAC learning is possible only for very restricted classes of functions F with bounded Littlestone dimension [Alon et al., 2019, Bun et al., 2020]. To make private learning statistically tractable for broader classes of functions, we consider the following restriction on ν :

Definition 3. Given a measure µ ∈ ∆( X ) and a parameter σ ∈ (0 , 1] , we say that ν x is σ -smooth with respect to µ if ∥ ∥ ∥ dν x dµ ∥ ∥ ∥ ∞ ≤ 1 σ . We suppose that the learner has access to m samples Z 1 , . . . , Z m ∼ µ that are independent of each other and the training data D and thus A may depend on these samples.

We remark that Definition 3 can be significantly relaxed by assuming only that D f ( ν x || µ ) ≤ 1 σ as in Block and Polyanskiy [2023], where D f ( ·||· ) is a sufficiently strong f -divergence 2 . In this case, the statistical rates presented below will be worse and depend on f , but the algorithms and privacy guarantees will remain unchanged. Critically, we do not require that our algorithms are private with respect to Z 1 , . . . , Z m , which we treat as public, unlabelled data. The key reason that this public data helps us circumvent the lower bounds is that it gives us access (albeit indirectly) to a small subclass of the hypothesis set that still has approximately good hypotheses. Since our primary focus is to design computationally efficient private learners, we cannot directly handle either the original hypothesis class or the small proxy that the public data gives us access. Instead we suppose access to the following ERM oracle:

Definition 4. Given a function class F : X → R , a data set D = { x 1 , . . . , x m } ⊂ X and loss functions ℓ 1 , . . . , ℓ m : R → R , we define the empirical risk minimization oracle ERM : F → R such that ERM ( F , L D ) ∈ argmin f ∈F L D ( f ) , where L D ( f ) = ∑ x i ∈D ℓ i ( f ( x i )) .

ERM oracles are standard computational models in many learning domains such as online learning [Kalai and Vempala, 2005, Hazan and Koren, 2016, Block et al., 2022, Haghtalab et al., 2022a] and Reinforcement Learning [Foster and Rakhlin, 2020, Foster et al., 2021, Mhammedi et al., 2023b,a]. Assuming access to ERM allows us to disentangle the computational challenges of optimizing over specific function classes from the specific challenge of differentially private learning as well as to avoid the well-known intractability results for nonconvex optimization [Blum and Rivest, 1988] that do not accurately reflect the realities of modern optimization techniques (e.g., integer program solvers, SGD). We note that our algorithms also work in the case of ERM oracles with additive error by minor modification to the analysis similar to the one in [Block et al., 2022]. We remark that applying Neel et al. [2019, Theorem 8] gives a black-box robustification procedure for purely private, oracle-efficient algorithms, which ensures that the privacy guarantees continue to hold even when the oracle may fail to optimize the objective. In particular, Algorithms 3 and 4 below, when run in their pure DP forms can be made robust at a minimal cost on accuracy. We defer to Neel et al. [2019] for further discussion on this topic.

1: Input Function ¯ f ∈ F , distribution Q ∈ ∆( R ) , scale γ ≥ 0 , public data ˜ D x = { Z 1 , . . . , Z m } .

2: Sample ζ 1 , . . . , ζ m ∼ Q .

3: Define R ( f ) = ∥ ∥ f -¯ f -γ · ζ ∥ ∥ 2 m .

4: Output ̂ f = ERM ( R, F ) .

It is well-known that even absent differential privacy guarantees, learning arbitrary function classes is impossible; we now introduce the notions of complexity that are relevant to our results. We begin with the standard notion of VC dimension:

Definition 5. Let F : X → { 0 , 1 } be a function class. We say that a set of points x 1 , . . . , x d ∈ X shatters F if for all ε 1: d ∈ { 0 , 1 } d , there is some f ε such that f ε ( x i ) = ε i for all i . The VC dimension of F , denoted vc ( F ) , is the largest d such that there exists a set of d points shattering F .

In addition to VC dimension, we also use the Gaussian complexity of a function class:

Definition 6. Let F : X → [ -1 , 1] be a function class and x 1 , . . . , x m ∈ X be arbitrary points. We let ω m : F → R be the canonical Gaussian process on F , i.e.,

where ξ i are independent standard Gaussians. We define the (data-dependent) Gaussian complexity of F to be E [ sup f ∈F ω m ( f ) ] , the average Gaussian complexity as G m ( F ) = E Z E [sup f ∈F ω m ( f )] , and the worst-case Gaussian complexity of F to be G m ( F ) = sup x 1 ,...,x m ∈X E [ sup f ∈F ω m ( f ) ] .

Both vc ( F ) and G m ( F ) are well known measures of complexity from learning theory and their relationships to other notions of complexity like covering number are well-understood [Mendelson and Vershynin, 2003, Wainwright, 2019, Van Handel, 2014]. In particular, it is well-known that G m ( F ) = O ( √ vc ( F )) [Dudley, 1969, Mendelson and Vershynin, 2003] and that standard PAClearning is possible if and only if G m ( F ) = o ( √ m ) [Wainwright, 2019, Van Handel, 2014]. We remark that different texts use different scalings for G m ( F ) , with some replacing the m -1 / 2 factor in (1) with m -1 and others omitting it entirely; our choice of scaling is motivated by the fact that a natural complexity measure for many (Donsker [Wainwright, 2019]) function classes that our algorithms depend on is sup m G m ( F ) , which is most compactly represented with the present scaling.

Notation. We always reserve P and E for probability and expectation with respect to measures that are clear from the context. We denote by ∆( X ) the space of measures on some X and for any µ ∈ ∆( X ) we let ∥·∥ µ denote the L 2 ( µ ) norm, i.e., ∥ f ∥ 2 µ = E Z ∼ µ [ f ( Z ) 2 ] . Similarly, for m points Z 1 , . . . , Z m ∈ X , we let ∥·∥ m denote the empirical L 2 norm on these points so that ∥ f ∥ 2 m = m -1 · ∑ m i =1 f ( Z i ) 2 . We reserve ω m for the canonical empirical Gaussian process on F as in (1) and L for a functional on F .

3 Algorithms for Differentially Private Learning

In this section, we provide a general template for constructing differentially private learning algorithms with public data and instantiate this template with two oracle-efficient algorithms. Our first algorithm applies to arbitrary bounded function classes, whereas the second algorithm only applies to convex classes but has an improved sample complexity. Our general template is broken into the following two steps: (i) Use ERM (cf. Definition 4) and the public data to construct an initial estimate ¯ f that is a good learner and satisfies stability with respect to ∥·∥ m ; (ii) Output ̂ f as the function that minimizes ∥ ∥ f -¯ f -γ · ζ ∥ ∥ m , where γ ≥ 0 is a scale and ζ = ( ζ 1 , . . . , ζ m ) is a vector of independent random variables sampled according to some distribution Q . The second step is accomplished through Algorithm 1 and is the same across our algorithms. The first step, however,

Algorithm 2: Oracle Efficient Private Learner (Perturbation)

1: Input Oracle ERM , perturbation parameter η > 0 , public data set ˜ D x = { Z 1 , . . . , Z m } , private data set D = { ( X i , Y i ) | 1 ≤ i ≤ n } , function class F , loss function ℓ , noise level γ > 0 , number of iterations J ∈ N , noise distribution Q ∈ ∆( R ) .

2: for j = 1 , 2 , . . . , J do

3: Sample ξ ( j ) 1 , . . . , ξ ( j ) m ∼ N (0 , 1) .

4: Define ω ( j ) m : F → R such that

5: Define L ( j ) : F → R such that L ( j ) ( f ) = ∑ ( X i ,Y i ) ∈D ℓ ( f ( X i ) , Y i ) + η · ω ( j ) m ( f ) .

6: Define ¯ f j = ERM ( L ( j ) , F ) end

7: Define ¯ f = 1 J · ∑ J j =1 ¯ f j .

8: Output f = Perturb f, Q , γ, D x

̂ ( ¯ ˜ ) ▷ By running Algorithm 1

is algorithm-specific and is the primary factor affecting the sample complexity. The intuition for our template is as follows. We need to show that ̂ f is both a learner and is differentially private. To see why the template produces a good learner, note that if Z i ∼ µ are independent and m is sufficiently large, then ∥·∥ m ≈ ∥·∥ µ . Thus if γ is small, then ∥ ∥ ∥ ¯ f -̂ f ∥ ∥ ∥ µ ≪ 1 and ∣ ∣ ∣ E µ [ L ( ̂ f ) ] -E µ [ L ( ¯ f )] ∣ ∣ ∣ ≪ 1 whenever ℓ is Lipschitz. By smoothness, a similar guarantee holds for expectations with respect to ν and thus ̂ f is a good learner. To see why ̂ f is differentially private, note that by choosing Q to be a standard Gaussian, we can ensure that the likelihood ratios of choosing ̂ f given ¯ f versus ¯ f ′ are controlled by ∥ ∥ ¯ f -¯ f ′ ∥ ∥ m . Thus, if ¯ f is stable with respect to ∥·∥ m , then ̂ f will be private. The intuition of the stability of ¯ f is discussed in Section 4.

We now make the above intuition precise by instantiating this template in our most general setting in Algorithm 2. We construct ¯ f by running ERM on a perturbed version of the empirical risk minimization problem and then averaging. Specifically, for j ∈ [ J ] , we define L ( j ) : F → R as a sample path of a noncentred Gaussian process in (2) and let ¯ f j denote the minimizer of L ( j ) over F . We then output ¯ f as the average of ¯ f 1 , . . . , ¯ f J . We present motivation for the particular choice of ¯ f , as well as the analogue in Algorithm 3, in the subsequent section. The following theorem shows that if Q is chosen correctly, this algorithm is an oracle-efficient differentially private learner whenever ν x is σ -smooth with respect to µ .

Theorem 2. Suppose that F : X → [ -1 , 1] is a function class and µ ∈ ∆( X ) is a measure such that inf f ∈F ∥ f ∥ µ ≥ 2 3 . Let ℓ : [ -1 , 1] × [ -1 , 1] → [0 , 1] be a loss function that is convex and λ -Lipschitz in its first argument. If Q = N (0 , 1) in Algorithm 1, then for any ε, δ, α, β ∈ (0 , 1) , there are choices of η, γ > 0 and J, m ∈ N , all polynomial in problem parameters and given in Appendix C.6, such that if

then the ̂ f returned by Algorithm 2 is ( ε, δ ) -differentially private. If ν x is σ -smooth with respect to µ , then ̂ f is an ( α, β ) -learner with respect to ν x and ℓ .

We emphasize that Algorithm 2 is always differentially private , independent of ν ; however, our algoirthm is only a good learner if ν x is smooth with respect to µ . We remark that all of the conditions in Theorem 2 are standard with the exception of the assumption that ∥ f ∥ µ ≥ 2 / 3 for all f ∈ F . This condition is easy to ensure by setting ˜ µ = µ +2 · δ z ⋆ 3 , where z ⋆ is a distinguished point such that f ( z ⋆ ) = 1 for all f ∈ F ; note that this process deflates σ at most by a factor of 3 while ensuring the lower bound on the norm of f . Replacing µ by ˜ µ then suffices to ensure that Theorem 2 holds.

Algorithm 3: Oracle Efficient Private Learner (Regularization)

1: Input Oracle ERM , perturbation parameter η > 0 , public data set ˜ D x = { Z 1 , . . . , Z m } , private data set D = { ( X i , Y i ) | 1 ≤ i ≤ n } , function class F , loss function ℓ , noise level γ > 0 , number of iterations J ∈ N , noise distribution Q ∈ ∆( R ) .

2: Define L : F → R such that

3: Define ¯ f = ERM ( L , F )

4: Output f = Perturb ( ¯ f, Q , γ, D x

̂ ˜ ) ▷ By running Algorithm 1

We further remark that it is classical that the complexity notion sup m G m ( F ) is upper bounded by √ vc ( F ) for binary function classes and √ log( |F| ) for finite classes [Wainwright, 2019], ensuring that the proven sample complexity is polynomial in all standard notions of function class complexity. For even more complex function classes, where G m ( F ) = ω (1) , similar results hold, although with worse rates; further dicussion, as well as the precise polynomial dependence of hyperparameters and sample complexity, can be found in Appendix C.

While Algorithm 2 succeeds in our desiderata under general assumptions, the sample complexity is a large polynomial of the desired accuracy. Indeed, the sample complexity of Algorithm 2 scales like O ( vc ( F ) · ε -3 · α -14 ) , which is significantly worse than the O ( vc ( F ) · α -2 ) sample complexity that a non-private algorithm such as ERM can achieve [Wainwright, 2019] or even the O ( vc ( F ) · α -2 · ε -2 ) sample complexity achievable by private, inefficient algorithms with public data [Bassily et al., 2020b]. Furthermore, we are unable to achieve a pure differential privacy guarantee with this algorithm. We now address both issues by providing an improved algorithm in the special case that the function class F is convex . While we still use Algorithm 1 as a subroutine, in Algorithm 3, motivated by the difference between Follow the Perturbed Leader (FTPL) and Follow the Regularized Leader (FTRL) [Kalai and Vempala, 2005, Cesa-Bianchi and Lugosi, 2006] in online learning, we modify the way in which we choose our initial estimator ¯ f . In particular, we eliminate the averaging step and redefine ω to be a strongly convex regularizer instead of a Gaussian Process perturbation . More specifically, we define L in (3) as the empirical loss regularized by ∥·∥ 2 m and output ¯ f = ERM ( L , F ) . We have the following result:

Theorem 3. Suppose that F : X → [ -1 , 1] is a convex function class and ℓ : [ -1 , 1] × [ -1 , 1] → [0 , 1] is convex and λ -Lipschitz in its first argument. Suppose that Z 1 , . . . , Z m ∼ µ are independent and Q = N (0 , 1) . Then there are η, γ, m polynomial in problem parameters and given in Appendix C.6 such that, if

then the ̂ f returned by Algorithm 3 is ( ε, δ ) -differentially private. If ν x is σ -smooth with respect to µ , then ̂ f is an ( α, β ) -learner with respect to ν x and ℓ . Furthermore, if Q = Lap (1) , and

then Algorithm 3 is ε -purely differentially private and an ( α, β ) -PAC learner for any σ -smooth ν x .

As in the case of Theorem 2, we can easily generalize Theorem 3 to apply to function classes F where G m ( F ) = ω (1) at the cost of worse polynomial dependence in the sample complexity. We again omit this case for the sake of simplicity. While the sample complexity of Algorithm 3 is a marked improvement over that of Algorithm 2, it remains a far cry from the desired O ( α -2 ) rates of non-private learning that computationally inefficient private algorithms leveraging public data are able to achieve [Bassily et al., 2019]; we leave the interesting question of producing an oracle-efficient private algorithm with optimal sample complexity to future work.

Finally, we remark that even in the case where F is not convex, Algorithm 3 can be applied to conv( F ) , the convex hull of F , if we assume the learner has access to ERM ′ , a stronger ERM oracle

that can optimize over conv( F ) . In this case, Theorem 3 supercedes Theorem 2 as it is easy to see that G m ( F ) = G m (conv( F )) and thus the sample complexity of Algorithm 3 is strictly better than that of Algorithm 2 and the pure differential privacy result applies.

4 Analysis Techniques

In this section, we outline the proofs of our main results, with full details deferred to Appendix C. As is suggested by our template, the proof of the privacy part of Theorem 2 rests on two results: the first shows that if Q is a standard Gaussian (resp. exponential) then stability of ¯ f with respect to ∥·∥ m can be translated into differential privacy. The second shows that ¯ f will be stable with respect to ∥·∥ m . Similarly, the proof that ̂ f is a good learner first shows that ¯ f is a good learner and then that ̂ f and ¯ f are close. We begin with the more technically novel parts and show that, under standard assumptions, Algorithms 2 and 3 result in ¯ f that are stable in ∥·∥ m . In our proof of Theorem 2, we provide an improved analysis of the Gaussian anti-concentration result from Block et al. [2022], which may be of independent interest. We prove the stability of Algorithm 3 using a technique common in online learning. We then show that stability in ∥·∥ m can be boosted to a differential privacy guarantee using the Gaussian and Laplace Mechanisms [Dwork et al., 2006]. Finally, we apply standard learning theoretic techniques to show that ̂ f is a good learner.

4.1 Stability Analysis

In this section, we explain how to prove that Algorithms 2 and 3 are stable with respect to ∥·∥ m . Our stability results further cement the connections between differential privacy and online learning noted in Abernethy et al. [2019] as both algorithms are primarily motivated by online learning techniques. We begin by describing the stability analysis of Algorithm 2. The key lemma underlying the stability of Algorithm 2 is an improved version of a Gaussian anti-concentration result from Block et al. [2022], which may be of independent interest.

Proposition 1. Let F denote a subspace of the unit ball with respect to a norm ∥·∥ induced by an inner product ⟨· , ·⟩ and let m,m ′ : F → R be measurable functions such that ∥ m -m ′ ∥ ∞ ≤ τ . If ω is a centred Gaussian process on F with covariance kernel given by ⟨· , ·⟩ , Ω( f ) = m ( f ) + η · ω ( f ) , ¯ f = argmin f ∈F Ω( f ) , and Ω ′ and ¯ f ′ are defined similarly, then for any ρ, τ > 0 , it holds that P (∥ ∥ ¯ f -¯ f ′ ∥ ∥ > ρ ) ≤ 8 τ ρ 4 κ 2 η · E [ sup f ∈F ω ( f ) ] , where κ 2 = inf f ∈F E [ ω ( f ) 2 ] .

The proof proceeds in a similar way to that of Lemma 33 from Block et al. [2022], but involves a tighter analysis in several steps in order to improve the bound. The intuition for the result is straightforward: if ¯ f is the minimizer of the Gaussian process Ω , then with reasonable probability, almost minimizers of Ω (as measured by the tolerance τ ) are within a radius ρ of ¯ f as long as the Gaussian process is nontrivial in the sense that all indices f have sufficiently high variance. Moreover, the quantitative control on the probability of this event depends in a natural way both on τ and ρ as well as on the Gaussian process ω : more complex spaces F and lower variance processes lead to a worse anti-concentration guarantee. Finally, we note that Proposition 1 is an improvement of Lemma 33 from Block et al. [2022] in that the quantitative bound on the probability of anti-concentration is tighter by polynomial factors in ρ, η , and κ .

Like essentially all anti-concentration results [Chernozhukov et al., 2015], Proposition 1 holds only with moderate probability in the sense that the guarantee is polynomial in the scale ρ ; this fact is in contradistinction to concentration inequalities which tend to hold with high probability exponential in the scale. This discrepancy is precisely what motivates the averaging in Line 7 of Algorithm 2. Indeed, we can use Proposition 1 to show that if ¯ f j is as in Line 6 of Algorithm 2 and ¯ f ′ j is defined analogously with respect to D ′ , then with moderate probability ∥ ∥ ¯ f j -¯ f ′ j ∥ ∥ m is small. Using Jensen's inequality and a standard chernoff bound, we can then boost this moderate probability guarantee into a high probability guarantee to show that if J is sufficiently large, then ∥ ∥ ¯ f -¯ f ′ ∥ ∥ m is small with high probability. We formalize this argument in the following lemma:

Lemma 1 (Stability of Algorithm 2) . Suppose that F : X → [ -1 , 1] is a function class and ℓ : [ -1 , 1] × 2 → [0 , 1] is a bounded loss function. Suppose that D , D ′ are neighboring datasets and let ¯ f be as in Line 5 of Algorithm 2 and ¯ f ′ be defined analogously with respect to D ′ . Then

for any ρ, δ > 0 , with probability at least 1 -δ , over the Gaussian processes ω ( j ) , ∥ f -f ′ ∥

∥ ¯ ¯ ∥ m ≤ .

We note that the worse dependence on η in Lemma 1 as compared to Proposition 1 arises from integrating the tail bound to obtain the control on ∥ ∥ ¯ f j -¯ f ′ j ∥ ∥ m in expectation necessary to apply Jensen's inequality; details can be found in Appendix C.1.

Wenowturn to the stability of Algorithm 3. The proof is based on a technique borrowed from online learning and the analysis of the Follow the Regularized Leader (FTRL) algorithm [Gordon, 1999, Cesa-Bianchi and Lugosi, 2006].

Lemma 2 (Stability of Algorithm 3) . Suppose that ℓ is convex and λ -Lipschitz in its first argument. Let D , D ′ denote neighboring data sets and let ¯ f denote the output of Line 3 in Algorithm 3 and ¯ f ′ be the analogous output evaluated on D ′ . If F is convex, then, ∥ ∥ ¯ f -¯ f ′ ∥ ∥ m ≤ 2 √ η · n .

The proof of Lemma 2 can be found in Appendix C.2 and rests on elementary properties of strongly convex functions. We note that relative to Lemma 1, the dependence on η in Lemma 2 is improved, which in turn leads to the better sample complexity exhibited in Theorem 3. With stability of Algorithms 2 and 3 thus established, we proceed to analyze the effect of the output perturbation.

4.2 Output Perturbation Analysis

We now turn to the analysis of Algorithm 1. In order to boost a stability-in-norm guarantee into one for differential privacy while remaining a good learner, we require the output perturbation to be sufficiently small as to not not affect the learning guarantee of ¯ f while at the same time being sufficiently large as to ensure privacy. We balance these two competing objectives by tuning the variance of the added noise. This part of the analysis is relatively standard in the differential privacy literature [Chaudhuri et al., 2011, Neel et al., 2019], with the bound on the size of the output perturbation following from standard tail bounds on Gaussian and Laplace random vectors. The privacy guarantees are similarly standard and summarized in the following lemma:

Lemma 3. Suppose ¯ f ∈ F is the output of an algorithm A : D → F that is ρ -stable with respect to ∥·∥ m , i.e., for any neighboring data set D ′ , it holds that ∥A ( D ) -A ( D ′ ) ∥ m ≤ ρ . Then applying Algorithm 1 with Q = N (0 , 1) to ¯ f results in an ( ε, δ ) -private algorithm if m 2 γ 2 ( 1 + γ · √ log ( 1 δ ) ) ρ ≤ ε . Similarly, if Q = Lap ( γ ) , then the algorithm is ε -purely private if m 3 / 2 /γ · ρ ≤ ε .

This standard result is proved in Appendix C.3. Note that, perhaps counterintuitively, the privacy loss increases with the public data. This relationship occurs because the algorithm is implicitly discretizing the function class, where more public data leads to a finer discretization; though finer discretizations lead to higher accuracy, they also leads to more privacy loss. Furthermore, note that even were the whole marginal distribution known, the privacy-accuracy tradeoff is dictated by the number of labelled samples, not m .

The balance between privacy and learning is quantified in the choices of m and γ . If γ is too large, then ¯ f will be private but a poor learner, whereas the opposite occurs if γ is too small. Similarly, if m is too large then privacy is reduced whereas if m is too small then ∥·∥ m is a poor approximation for ∥·∥ µ .

4.3 Learning Guarantees and Concluding the Proof

By combining Lemma 1 (resp. Lemma 2) with Lemma 9, we can establish the privacy of Algorithm 2 (resp. Algorithm 3) as long as the tuning parameters m,γ,η , and J are chosen correctly. We now sketch the proof that these algorithms comprise good learners in the sense of Definition 2. We break our proof into three components, the first two of which are standard learning theoretic results. The first lemma says that if m ≫ 1 , then ∥·∥ m is a good approximation for ∥·∥ µ :

Lemma 4. Let F : X → [ -1 , 1] be a bounded function class and let Z 1 , . . . , Z m ∼ µ be independent samples. Then for any β > 0 it holds with probability at least 1 -β that for all f ∈ F ,

Lemma 4 is a standard bound from learning theory [Bousquet, 2002, Rakhlin et al., 2017] and is proved in Appendix C.7 for the sake of completeness. The second component is given by Lemma 13 in Appendix C.5, which amounts to a classical uniform deviations bound for the empirical process, ensuring that if n ≫ 1 , then L D ( f ) ≈ L ( f ) for all f ∈ F . The final step is the following simple lemma, which ensures that if η is not too large, then L D ( ¯ f ) ≈ L D ( f ERM ) :

Lemma 5. Let F : X → [ -1 , 1] be a bounded function class and let R : F → R be an arbitrary, possibly random, regularizer. Let f ERM ∈ argmin f ∈F L D ( f ) and ¯ f ∈ argmin f ∈F L D ( f ) + R ( f ) . Then, L D ( ¯ f ) ≤ L D ( f ERM ) + sup f,f ′ ∈F R ( f ) -R ( f ′ ) .

Lemma 5 is a simple computation proved in Appendix C.5. Letting R ( f ) be either η · ω ( j ) m ( f ) in Algorithm 2 or η · ∥ f ∥ 2 m in Algorithm 3 demonstrates that if η is not too large, then ¯ f performs similarly to f ERM . To prove that Algorithms 2 and 3 produce good learning algorithms then, it suffices to combine these three components, observing first that L ( ¯ f ) is close to optimal if n ≫ 1 and η is not too large, second that ∥ ¯ f -̂ f ∥ µ ≪ 1 if m ≫ 1 and γ is sufficently small, and third that | L ( ̂ f ) -L ( ¯ f ) | ≲ ∥ ¯ f -̂ f ∥ µ if ν x is σ -smooth with respect to µ and ℓ is λ -Lipschitz in its first argument. Combining these results concludes the proofs of Theorems 2 and 3. A detailed and rigorous argument for both proofs is presented in Appendix C. As a final remark, we note that in the case of Algorithm 2, convexity of ℓ in the first argument is irrelevant to the privacy guarantee despite being necessary for learning. Indeed, for ¯ f returned by Line 5 in Algorithm 2 to be proven a good learner, we apply Jensen's and the above argument that ensures that ¯ f j is a good learner. Interestingly, on the other hand, convexity in ℓ is irrelevant to the learning guarantee of Algorithm 3 while it is essential to the privacy guarantee. Further understanding the role that such structural assumptions play in allowing privacy is an interesting direction for future work.

Acknowledgments and Disclosure of Funding

AB acknowledges support from the National Science Foundation Graduate Research Fellowship under Grant No.1122374 as well as the Simons Foundation and the National Science Foundation through awards DMS-2031883 and DMS-1953181. MB acknowledges support from the National Science Foundation through award NSF CNS-2046425 and from a Sloan Research Fellowship. RD acknowledges support from the National Science Foundation through award NSF CNS-2046425. AS acknowledges support from the Apple AI+ML fellowship. AB also would like to thank Satyen Kale and Claudio Gentile for helpful discussions. Z.S.W. was in part supported by NSF Awards #1763786 and #2339775.

References

Jacob D Abernethy, Young Hun Jung, Chansoo Lee, Audra McMillan, and Ambuj Tewari. Online learning via the differential privacy lens. Advances in Neural Information Processing Systems , 32, 2019.

Noga Alon, Roi Livni, Maryanthe Malliaris, and Shay Moran. Private PAC learning implies finite littlestone dimension. In Moses Charikar and Edith Cohen, editors, Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, STOC 2019, Phoenix, AZ, USA, June 23-26, 2019 , pages 852-860. ACM, 2019. doi: 10.1145/3313276.3316312. URL https://doi.org/10.1145/3313276.3316312 .

Ehsan Amid, Arun Ganesh, Rajiv Mathews, Swaroop Ramaswamy, Shuang Song, Thomas Steinke, Vinith M Suriyakumar, Om Thakkar, and Abhradeep Thakurta. Public data-assisted mirror descent for private model training. In International Conference on Machine Learning , pages 517535. PMLR, 2022.

Maria-Florina F Balcan and Vitaly Feldman. Statistical active learning algorithms. Advances in neural information processing systems , 26, 2013.

Peter L Bartlett, Philip M Long, and Robert C Williamson. Fat-shattering and the learnability of realvalued functions. In Proceedings of the seventh annual conference on Computational learning theory , pages 299-310, 1994.

Raef Bassily, Om Thakkar, and Abhradeep Guha Thakurta. Model-agnostic private learning. Advances in Neural Information Processing Systems , 31, 2018.

Raef Bassily, Shay Moran, and Noga Alon. Limits of private learning with access to public data. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 814, 2019, Vancouver, BC, Canada , pages 10342-10352, 2019. URL https://proceedings. neurips.cc/paper/2019/hash/9a6a1aaafe73c572b7374828b03a1881-Abstract.html .

Raef Bassily, Albert Cheu, Shay Moran, Aleksandar Nikolov, Jonathan R. Ullman, and Zhiwei Steven Wu. Private query release assisted by public data. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 695-703. PMLR, 2020a. URL http://proceedings.mlr.press/v119/bassily20a.html .

Raef Bassily, Shay Moran, and Anupama Nandi. Learning from mixtures of private and public populations. Advances in Neural Information Processing Systems , 33:2947-2957, 2020b.

Raef Bassily, Mehryar Mohri, and Ananda Theertha Suresh. Private domain adaptation from a public source. arXiv preprint arXiv:2208.06135 , 2022.

Raef Bassily, Mehryar Mohri, and Ananda Theertha Suresh. Principled approaches for private adaptation from a public source. In International Conference on Artificial Intelligence and Statistics , pages 8405-8432. PMLR, 2023.

Amos Beimel, Kobbi Nissim, and Uri Stemmer. Learning privately with labeled and unlabeled examples. In Proceedings of the twenty-sixth annual ACM-SIAM symposium on Discrete algorithms , pages 461-477. SIAM, 2014.

Shai Ben-David, Alex Bie, Clément L Canonne, Gautam Kamath, and Vikrant Singhal. Private distribution learning with public data: The view from sample compression. arXiv preprint arXiv:2308.06239 , 2023.

Alex Bie, Gautam Kamath, and Vikrant Singhal. Private estimation with public data. Advances in Neural Information Processing Systems , 35:18653-18666, 2022.

Adam Block and Yury Polyanskiy. The sample complexity of approximate rejection sampling with applications to smoothed online learning. In Gergely Neu and Lorenzo Rosasco, editors, Proceedings of Thirty Sixth Conference on Learning Theory , volume 195 of Proceedings of Machine Learning Research , pages 228-273. PMLR, 12-15 Jul 2023. URL https: //proceedings.mlr.press/v195/block23a.html .

Adam Block and Max Simchowitz. Efficient and near-optimal smoothed online learning for generalized linear functions. Advances in Neural Information Processing Systems , 35:7477-7489, 2022.

Adam Block, Yuval Dagan, Noah Golowich, and Alexander Rakhlin. Smoothed online learning is as easy as statistical learning. In Conference on Learning Theory , pages 1716-1786. PMLR, 2022.

Adam Block, Max Simchowitz, and Alexander Rakhlin. Oracle-efficient smoothed online learning for piecewise continuous decision making. In Gergely Neu and Lorenzo Rosasco, editors, Proceedings of Thirty Sixth Conference on Learning Theory , volume 195 of Proceedings of Machine Learning Research , pages 1618-1665. PMLR, 12-15 Jul 2023a. URL https: //proceedings.mlr.press/v195/block23b.html .

Adam Block, Max Simchowitz, and Russ Tedrake. Smoothed online learning for prediction in piecewise affine systems. In Advances in Neural Information Processing Systems . Curran Associates, Inc., 2023b. URL https://openreview.net/pdf?id=Izt7rDD7jN .

Avrim Blum and Ronald Rivest. Training a 3-node neural network is np-complete. Advances in neural information processing systems , 1, 1988.

Olivier Bousquet. Concentration inequalities and empirical processes theory applied to the analysis of learning algorithms . PhD thesis, École Polytechnique: Department of Applied Mathematics Paris, France, 2002.

Mark Bun, Kobbi Nissim, Uri Stemmer, and Salil Vadhan. Differentially private release and learning of threshold functions. In 2015 IEEE 56th Annual Symposium on Foundations of Computer Science , pages 634-649. IEEE, 2015.

Mark Bun, Roi Livni, and Shay Moran. An equivalence between private classification and online prediction. In 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS) , pages 389-402. IEEE, 2020.

Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games . Cambridge university press, 2006.

Nicolò Cesa-Bianchi, Tommaso R Cesari, Roberto Colomboni, Federico Fusco, and Stefano Leonardi. Repeated bilateral trade against a smoothed adversary. In The Thirty Sixth Annual Conference on Learning Theory , pages 1095-1130. PMLR, 2023.

Kamalika Chaudhuri, Claire Monteleoni, and Anand D Sarwate. Differentially private empirical risk minimization. Journal of Machine Learning Research , 12(3), 2011.

Victor Chernozhukov, Denis Chetverikov, and Kengo Kato. Comparison and anti-concentration bounds for maxima of gaussian random vectors. Probability Theory and Related Fields , 162: 47-70, 2015.

Miroslav Dudík, Nika Haghtalab, Haipeng Luo, Robert E Schapire, Vasilis Syrgkanis, and Jennifer Wortman Vaughan. Oracle-efficient online learning and auction design. Journal of the ACM (JACM) , 67(5):1-57, 2020.

Richard Mansfield Dudley. The speed of mean glivenko-cantelli convergence. The Annals of Mathematical Statistics , 40(1):40-50, 1969.

Naveen Durvasula, Nika Haghtalab, and Manolis Zampetakis. Smoothed analysis of online nonparametric auctions. In Proceedings of the 24th ACM Conference on Economics and Computation , pages 540-560, 2023.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam D. Smith. Calibrating noise to sensitivity in private data analysis. In Shai Halevi and Tal Rabin, editors, Theory of Cryptography, Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006, Proceedings , volume 3876 of Lecture Notes in Computer Science , pages 265-284. Springer, 2006. doi: 10.1007/11681878\_14. URL https://doi.org/10.1007/11681878_14 .

Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science , 9(3-4):211-407, 2014.

Dylan Foster and Alexander Rakhlin. Beyond ucb: Optimal and efficient contextual bandits with regression oracles. In International Conference on Machine Learning , pages 3199-3210. PMLR, 2020.

Dylan J Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 , 2021.

Marco Gaboardi, Emilio Jesús Gallego Arias, Justin Hsu, Aaron Roth, and Zhiwei Steven Wu. Dual query: Practical private query release for high dimensional data. In International Conference on Machine Learning , pages 1170-1178. PMLR, 2014.

Badih Ghazi, Noah Golowich, Ravi Kumar, and Pasin Manurangsi. Sample-efficient proper pac learning with approximate differential privacy. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 183-196, 2021.

Aditya Golatkar, Alessandro Achille, Yu-Xiang Wang, Aaron Roth, Michael Kearns, and Stefano Soatto. Mixed differential privacy in computer vision. In CVPR 2022 , 2022. URL https://www.amazon.science/publications/ mixed-differential-privacy-in-computer-vision .

Sally A Goldman, Michael J Kearns, and Robert E Schapire. Exact identification of read-once formulas using fixed points of amplification functions. SIAM Journal on Computing , 22(4):705726, 1993.

Geoffrey J Gordon. Regret bounds for prediction problems. In Proceedings of the twelfth annual conference on Computational learning theory , pages 29-40, 1999.

Nika Haghtalab, Tim Roughgarden, and Abhishek Shetty. Smoothed analysis of online and differentially private learning. Advances in Neural Information Processing Systems , 33:9203-9215, 2020.

Nika Haghtalab, Yanjun Han, Abhishek Shetty, and Kunhe Yang. Oracle-efficient online learning for smoothed adversaries. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 4072-4084. Curran Associates, Inc., 2022a. URL https://proceedings.neurips.cc/paper_files/ paper/2022/file/1a04df6a405210aab4986994b873db9b-Paper-Conference.pdf .

Nika Haghtalab, Tim Roughgarden, and Abhishek Shetty. Smoothed analysis with adaptive adversaries. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS) , pages 942-953. IEEE, 2022b.

Elad Hazan and Tomer Koren. The computational power of optimization in online learning. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing , pages 128141, 2016.

Max Hopkins, Daniel M. Kane, Shachar Lovett, and Gaurav Mahajan. Realizable learning is all you need. TheoretiCS , 3, 2024.

Peter Kairouz, Monica Ribero Diaz, Keith Rush, and Abhradeep Thakurta. (Nearly) dimension independent private erm with adagrad rates via publicly estimated subspaces. In Conference on Learning Theory , pages 2717-2746. PMLR, 2021.

Adam Kalai and Santosh Vempala. Efficient algorithms for online decision problems. Journal of Computer and System Sciences , 71(3):291-307, 2005.

Jeankyung Kim and David Pollard. Cube root asymptotics. The Annals of Statistics , pages 191-219, 1990.

Alexander Kozachinskiy and Tomasz Steifer. Simple online learning with consistency oracle. arXiv preprint arXiv:2308.08055 , 2023.

Jean-François Le Gall. Brownian motion, martingales, and stochastic calculus . Springer, 2016.

Terrance Liu, Giuseppe Vietri, Thomas Steinke, Jonathan Ullman, and Steven Wu. Leveraging public data for practical private query release. In International Conference on Machine Learning , pages 6968-6977. PMLR, 2021a.

Terrance Liu, Giuseppe Vietri, and Steven Wu. Iterative methods for private synthetic data: Unifying framework and new methods. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 690-702, 2021b. URL https://proceedings.neurips. cc/paper/2021/hash/0678c572b0d5597d2d4a6b5bd135754c-Abstract.html .

Andrew Lowy, Zeman Li, Tianjian Huang, and Meisam Razaviyayn. Optimal differentially private learning with public data. arXiv preprint arXiv:2306.15056 , 2023.

Frank McSherry and Kunal Talwar. Mechanism design via differential privacy. In 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS 2007), October 20-23, 2007, Providence, RI, USA, Proceedings , pages 94-103. IEEE Computer Society, 2007. doi: 10.1109/FOCS.2007. 41. URL https://doi.org/10.1109/FOCS.2007.41 .

Shahar Mendelson and Roman Vershynin. Entropy and the combinatorial dimension. Inventiones mathematicae , 152(1):37-55, 2003.

Zakaria Mhammedi, Adam Block, Dylan J Foster, and Alexander Rakhlin. Efficient model-free exploration in low-rank mdps. arXiv preprint arXiv:2307.03997 , 2023a.

Zakaria Mhammedi, Dylan J Foster, and Alexander Rakhlin. Representation learning with multistep inverse kinematics: An efficient and optimal approach to rich-observation rl. arXiv preprint arXiv:2304.05889 , 2023b.

Seth Neel, Aaron Roth, Giuseppe Vietri, and Zhiwei Steven Wu. Oracle efficient private nonconvex optimization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 7243-7252. PMLR, 2020. URL http://proceedings.mlr.press/v119/ neel20a.html .

Seth V Neel, Aaron L Roth, and Zhiwei Steven Wu. How to use heuristics for differential privacy. In 2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS) , pages 72-93. IEEE, 2019.

Aleksandar Nikolov, Kunal Talwar, and Li Zhang. The geometry of differential privacy: the sparse and approximate cases. In Proceedings of the forty-fifth annual ACM symposium on Theory of computing , pages 351-360, 2013.

Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Úlfar Erlingsson. Scalable private learning with PATE. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018. URL https://openreview.net/forum?id=rkZB1XbRZ .

Yury Polyanskiy and Yihong Wu. Information Theory: From Coding to Learning . Cambridge University Press, 2022+.

Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Online learning: Stochastic, constrained, and smoothed adversaries. Advances in neural information processing systems , 24, 2011.

Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Online learning via sequential complexities. J. Mach. Learn. Res. , 16(1):155-186, 2015.

Alexander Rakhlin, Karthik Sridharan, and Alexandre B Tsybakov. Empirical entropy, minimax regret and minimax risk. Bernoulli , pages 789-824, 2017.

Ralph Tyrell Rockafellar. Convex Analysis:(PMS-28) . Princeton university press, 2015.

Mark Rudelson and Roman Vershynin. Combinatorics of random processes and sections of convex bodies. Annals of Mathematics , pages 603-648, 2006.

Daniel A Spielman and Shang-Hua Teng. Smoothed analysis of algorithms: Why the simplex algorithm usually takes polynomial time. Journal of the ACM (JACM) , 51(3):385-463, 2004.

Nathan Srebro, Karthik Sridharan, and Ambuj Tewari. Smoothness, low noise and fast rates. Advances in neural information processing systems , 23, 2010.

Vasilis Syrgkanis, Akshay Krishnamurthy, and Robert Schapire. Efficient algorithms for adversarial contextual learning. In International Conference on Machine Learning , pages 2159-2168. PMLR, 2016.

Leslie G Valiant. A theory of the learnable. Communications of the ACM , 27(11):1134-1142, 1984.

Ramon Van Handel. Probability in high dimension. Lecture Notes (Princeton University) , 2014.

Giuseppe Vietri, Grace Tian, Mark Bun, Thomas Steinke, and Steven Wu. New oracle-efficient algorithms for private synthetic data release. In International Conference on Machine Learning , pages 9765-9774. PMLR, 2020.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge university press, 2019.

Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language models. In International Conference on Learning Representations (ICLR) , 2022.

Yingxue Zhou, Steven Wu, and Arindam Banerjee. Bypassing the ambient dimension: Private SGD with gradient subspace identification. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. URL https://openreview.net/forum?id=7dpmlkBuJFC .

Contents

A Differentially Private Classification

In the previous section, we presented a private algorithm for general, real-valued loss functions. Here, we turn to the special case of classification, where we provide an algorithm with improved

rates. Formally, binary classification is a special case of Definition 2, where F : X → { 0 , 1 } and ℓ is the indicator loss.

Much like Algorithms 2 and 3, our approach to classification in Algorithm 4 relies on minimizing a perturbed empirical loss over F and projecting the output ˜ f onto the public data. Unlike in these earlier algorithms, which require a further perturbation of the output in order to boost stability into differential privacy, in the special case of classification we are able to circumvent this second perturbation and return any ̂ f that agrees with ˜ f on the public data. This is accomplished by carefully choosing the initial perturbation to the ERM objective (see (4)) so that the predictions of ˜ f on the public data satisfy differential privacy without ensuring some form of stability in norm. As a result, our improved rates then follow from lack of a second perturbation. We present the following guarantee for our classification algorithm, whose pseudo-code can be found in Algorithm 4.

Algorithm 4: Rounded Report Separator Perturbed Minimum Algorithm (RRSPM)

Input ERM oracle ERM , dataset D = { ( X i , Y i ) | 1 ≤ i ≤ n } , hypothesis class F , smoothness parameter σ , loss function ℓ : Y × Y → { 0 , 1 } , arbitrary Q ∈ ∆( R ) .

Draw ˜ D = ( ˜ D x , ˜ D y ) where ˜ D x = { Z 1 , . . . , Z m } and ˜ D y = { ˜ Y 1 , . . . , ˜ Y m } such that Z i ∼ µ and ˜ Y i ∼ Uni( { 0 , 1 } ) , for all i ∈ [ m ] .

Draw weights ξ = { ξ 1 , . . . , ξ m } such that ξ i ∼ Lap(2 m/ε ) .

Define L ξ , D , ˜ D : F → R such that

Get ˜ f = ERM ( F , L ξ , D , ˜ D ) .

Output ˆ f = Perturb ( ˜ f, Q , γ = 0 , ˜ D x )

▷ By running Algorithm 1

Theorem 4. Suppose that F : X → Y is a function class of VC dimension d and ℓ : Y×Y → { 0 , 1 } is the indicator loss. Suppose that Z 1 , . . . , Z m ∼ µ and ξ 1 , . . . , ξ m ∼ Lap(2 m/ε ) are independent. Then there is a choice of m polynomial in the problem parameters such that if

then the ˆ f returned by Algorithm 4 is ε -pure differentially private. If ν x is σ -smooth with respect to µ , then ˆ f is an ( α, β ) -learner with respect to ν x and ℓ . Furthermore, for some C > 0 , if ξ 1 , . . . , ξ m ∼ N (0 , C √ m log(1 /δ ) /ε ) then there is a choice of m polynomial in the problem parameters such that if

then the ˆ f returned by Algorithm 4 with Gaussian perturbations is ( ε, δ ) -differentially private and is an ( α, β ) -PAC learner with respect to any σ -smooth ν x .

Remark 1. Note that the sample complexity we get in the above theorems is in the general, agnostic setting. In the realizable setting, where some f ⋆ ∈ F perfectly predicts the Y from the X , we get a sample complexity of n = ˜ Ω( d 2 ε -1 α -3 log( β -1 )) for ε -pure differential privacy and n = ˜ Ω( d 2 ε -1 α -2 . 5 log 1 / 2 (1 /δ ) log( β -1 )) for ( ε, δ ) -differential privacy.

We emphasize that Theorem 4 attains the improved O ( α -5 ) sample complexity (even O ( α -4 for approximate differential privacy)), which is significantly better than the O ( α -14 ) from Theorem 2. While this is a major improvement, it still falls short of the desired O ( α -2 ) statistical rates achievable by inefficient algorithms from [Bassily et al., 2019]. We leave the interesting question of whether improved sample complexity is possible to future work. We now briefly sketch the proof of Theorem 4.

A.1 Privacy Analysis of Algorithm 4

While the privacy of Algorithms 2 and 3 is proven in two steps, by first demonstrating stability and then leveraging the output perturbation to ensure privacy, the privacy of Algorithm 4 is proven

directly. Our approach is motivated by techniques from Neel et al. [2019], which adapt the earlier notion of separator sets from Goldman et al. [1993], Syrgkanis et al. [2016], Dudík et al. [2020] to the setting of differential privacy. Unlike those works, however, we do not require the strong assumption that F has a small separator set and our results hold for general VC function classes. The main technical result that ensures privacy of Algorithm 4 demonstrates that the projection F to the public data set ˜ D x is private with respect to D , where we let F| ˜ D x = { ( f ( Z i )) 1 ≤ i ≤ m | f ∈ F} . We have the following privacy guarantee for f ( ˜ D x ) ∈ F| ˜ :

Lemma 6. (Privacy over Projection) Let D , D ′ be arbitrary datasets containing n points each. Let ˜ D x be a set of m points Z 1 , . . . , Z m ∈ X . Let ˜ Y 1 , . . . , ˜ Y m ∈ { 0 , 1 } be the set of corresponding labels. Then for all measurable H ⊆ F| ˜ D x

where L ξ , D , ˜ D is defined as in (4) .

With the above lemma in hand, the privacy of Algorithm 4 follows immediately from the postprocessing property of differential privacy. We provide a full proof of Lemma 6 in Appendix D and now turn to the accuracy guarantee.

A.2 Concluding the Proof of Theorem 4

Proving that Algorithm 4 is an ( α, β ) -learner whenever ν x is σ -smooth with respect to µ is similar to the approach taken in Section 4.3, with the critical difference that in the absence of the second perturbation, we are able to achieve a stronger guarantee on the difference between ˜ f and ̂ f . Indeed, much as in the previous analysis, we observe that as m increases, the suboptimality of the intermediate ˜ f is driven up, while the difference between ˜ f and ̂ f is driven down, thereby requiring a careful balance; here, however, we do not also need to account for the balancing of the variance γ . We provide a full proof of the accuracy guarantee in Appendix D.

B Gaussian Anti-Concentration and Proof of Lemma 1

In this section we present and prove a more general version of Proposition 1. We begin by defining a Gaussian process and then state and prove the result. We then show how Proposition 1 follows as an immediate corollary. To begin, we recall the formal definition of a Gaussian process.

Definition 7. Let T be an index set and m : T → R be a function. Let K : T × T → R be a covariance kernel in the sense that for any t 1 , . . . , t n ∈ T , the matrix ( K ( t i , t j )) i,j ∈ [ n ] is positive semi-definite. We say that ω : T → R is a Gaussian process with mean function m and covariance kernel K if for any t 1 , . . . , t n ∈ T , the random vector ( ω ( t i )) i ∈ [ n ] is Gaussian with mean ( m ( t i )) i ∈ [ n ] and covariance matrix ( K ( t i , t j )) i,j ∈ [ n ] . We say that ω is a centered Gaussian process if m is identically zero.

Note that by Le Gall [2016, Theorem 1.11] such a process always exists given m,K . Furthermore, we note that K induces a semi-metric d on T by letting d ( t, t ′ ) 2 = E [ ( ω ( t ) -ω ( t ′ )) 2 ] . We now prove the following result, which is a tighter version of Block et al. [2022, Lemma 33].

Theorem 5 (Gaussian Anti-concentration) . Let T be a set, m : T → R be a mean function and K : T × T → R be a covariance kernel (in the sense of being positive definite). Let d denote the metric induced by K and suppose that m is continuous with respect to d , and the metric space ( T, d ) is separable and compact. Let ω denote a Gaussian process on T with covariance K and for η > 0 , let

be an offset Gaussian process. We further suppose that ω is taken to be a version with almost surely continuous paths t ↦→ ω ( t ) and that 0 < κ ≤ K ( t, t ) ≤ 1 for all t ∈ T . Let

and,

for ρ, τ > 0 . The following holds:

Note that K ( s,t ) K ( t,t ) is a measure of how close s and t are to each other; indeed, in the special case where κ = 1 this is precisely the correlation and thus s and t are more closely related the closer this quantity is to 1. Thus the event E ( ρ, τ ) can be interpreted to mean that there exists some point s far from t ⋆ (as governed by ρ ) such that Ω( s ) is almost minimal (as governed by τ ); in other words, Theorem 5 puts an upper bound on the probability that almost-minimizers of a Gaussian process lie far from the true minimizer. We now prove Theorem 5.

Proof of Theorem 5. Note that by compactness of T and almost sure continuity of Ω , a minimizer of Ω exists almost surely; furthermore, by [Kim and Pollard, 1990, Lemma 2.6], t ⋆ is almost surely unique. As T is separable and Ω has almost surely continuous sample paths, it suffices to replace T with a countable dense subset. We will hereafter suppose without loss of generality that T is countable. For each t ∈ T , define the set

It then suffices to lower bound the probability that A ( t ⋆ ) = ∅ . We compute

̸

where the expectation is taken over the distribution of Ω( t ) . Now, fix t and let Ω t,y denote the Gaussian process Ω conditioned on the event that Ω( t ) = y . Let m t,y and η 2 · K t denote the mean and covariance processes of Ω t,y . Critically, note that K t is independent of y and for all s = t , we have

Define the functions

Now, note that if K ( s, t ) ≤ (1 -ρ 2 ) · K ( t, t ) , then

We also have that b ( s ) ≥ 0 for all s by the fact that K ( s, t ) ≤ √ K ( s, s ) · K ( t, t ) and K ( s, s ) ∨ K ( t, t ) ≤ 1 . Furthermore, for all s , it holds that

Thus, for fixed t ∈ T and y ∈ R , we have

where the first inequality follows from (5), the second equality follows from the construction, and the last equality follows from (6) and the fact that K t is independent of y . Now, denote

the density of Ω( t ) and note that we have

We then compute

where we added and subtracted the first term and then made the variable substitution y + τ ρ 2 ↦→ y for the latter integral. Note that

as Ω( s ) ≥ Ω( t ⋆ ) for all s ∈ T by definition. Combining this observation with (7) and (8) yields

For the first term, we have

For the second term, using the fact that 1 -e x ≤ x for all x , we have

Thus we have

We now prove a corollary of Theorem 5 that will be useful in the proof of Proposition 1 and which makes the relationship between E ( ρ, τ ) and the intuition of distance between t ⋆ and s more explicit. Corollary 1. Suppose that we are in the situation of Theorem 5 with the additional conditions that T is a subset of a real vector space and that d ( s, t ) = √ K ( s -t, s -t ) . Let m ′ : T → R denote a mean function such that sup t ∈ T | m ( t ) -m ′ ( t ) | ≤ τ and let Ω ′ denote the corresponding shifted Gaussian process. If t ⋆ ′ = argmin t ∈ T Ω ′ ( t ) , then

Proof. Note that

Let M = max( K ( t ⋆ , t ⋆ ) , K ( t ⋆ ′ , t ⋆ ′ )) ≤ 1 and note that the above implies:

Thus,

where the last inequality follows by applying a union bound and Theorem 5 to the Gaussian processes Ω and Ω ′ after observing that

and similarly for Ω( t ⋆ ′ ) .

We are now ready to prove Proposition 1 using Corollary 1.

Proof of Proposition 1. Let T = F , and observe that

The result then follows immediately by applying Corollary 1 to the Gaussian process Ω( f ) = m ( f ) + η · ω ( f ) .

C Analysis of Algorithms 2 and 3

In this section we provide the full proofs for Theorems 2 and 3, as well as more general statements under different measures of the complexity of the function class F . We begin the section by stating formal bounds on the learning and differential privacy guarantees for each algorithm. In Appendix C.1, we prove Lemma 1, which is a key technical lemma in the differential privacy guarantee of Algorithm 2; we then continue in Appendix C.2 by proving Lemma 2, which plays an analogous role except in the analysis of Algorithm 3. In Appendix C.3, we prove that algorithmic stability in ∥·∥ m can be boosted to differential privacy through Algorithm 1. In Appendix C.4 we combine the previous results to give guarantees on the differential privacy of Algorithm 2 and Algorithm 3. We continue in Appendix C.5 by applying more standard learning theoretic techniques to demonstrate that both algorithms are PAC learners before concluding the proofs of the main theorems in Appendix C.6. Finally, for the sake of completeness, we prove a norm comparison lemma in Appendix C.7 that was deferred from Appendix C.5 for the purpose of continuity.

C.1 Stability of Algorithm 2 and Proof of Lemma 1

We break the proof into two parts. First, we integrate the tail bound from Proposition 1 to get control on E [ ∥ ∥ ¯ f j -¯ f ′ j ∥ ∥ m ] for any j ∈ [ J ] . We then apply a Jensen's inequality and a Chernoff bound to get high probability control on ∥ ∥ ¯ f -¯ f ′ ∥ ∥ . We begin with the following lemma:

Lemma 7. Let ¯ f j be as in in Line 6 of Algorithm 2 and ¯ f ′ j is defined analogously with respect to D ′ , a neighboring dataset to D . Then

where the expectation is with respect to ξ ( j ) .

Proof. By boundedness of ℓ , it holds that | L D ( f ) -L D ′ ( f ) | ≤ 1 n for all f ∈ F . Thus, if ¯ f (1) and ¯ f (1) ′ are as in the statement of the corollary, then L D ( ¯ f (1) ′ ) + η · ω (1) m ( ¯ f (1) ′ ) ≤ L D ( ¯ f (1) ) + η · ω (1) m ( ¯ f (1) ) + 1 n . Plugging in τ = 1 n and applying Proposition 1 yields

Now, we can integrate the tail bound to get for any ζ > 0 ,

Minimizing over ζ yields the result.

We now apply a Jensen's inequality and a Chernoff bound to get high probability control on ∥ ∥ ¯ f -¯ f ′ ∥ ∥ .

Proof of Lemma 1. By Jensen's inequality and the definition of ¯ f , it holds that

Finally, by Hoeffding's inequality [Wainwright, 2019, Van Handel, 2014] and Lemma 7, we have that

The result follows.

C.2 Stability of Algorithm 3 and Proof of Lemma 2

In this section we prove Lemma 2 based on a technique borrowed from online learning and the analysis of Follow the Regularized Leader (FTRL) [Gordon, 1999, Cesa-Bianchi and Lugosi, 2006].

Proof of Lemma 2. Let L be as in (3) and L ′ be defined similarly but with D replaced by D ′ . Note that ∥·∥ 2 m is strongly convex with respect to ∥·∥ m [Rockafellar, 2015]. Thus, by convexity of F , it holds that

On the other hand,

Combining this with the previous display and rearranging yields:

Rearranging again proves the result.

C.3 Boosting Stability to Differential Privacy: Proofs from Section 4.2

In this section, we analyze Algorithm 1 and prove that if ¯ f is stable in ∥·∥ m then applying the output perturbation yields a differentially private algorithm for standard choices of perturbation distribution Q . We also show that Algorithm 1 returns ̂ f close to the ¯ f with high probability. This first claim is implied by the following standard concentration bound.

Lemma 8. Suppose that Q = N (0 , 1) and let ̂ f = Perturb ( ¯ f, Q , γ, ˜ D x ) be as in Algorithm 1. Then with probability at least 1 -β ,

If Q = Lap (1) , then with probability at least 1 -β ,

Proof. By construction, it holds that

By the triangle inequality, it holds that

Thus it suffices to bound ∥ ζ ∥ m . Bounds on this quantity when ζ ∼ N (0 , 1) or ζ ∼ Lap (1) are standard and can be found in, for example Wainwright [2019]. The result follows.

We now prove the main property of Algorithm 1, namely that it boosts stability to differential privacy.

Lemma 9. Let Perturb be as in Algorithm 1 and suppose that ¯ f, ¯ f ′ ∈ F . If ˜ D x = { Z 1 , . . . , Z m } ⊂ X is arbitrary and Q = N (0 , 1) , then for any δ > 0 and any measurable G ⊂ F , it holds that

P ( Perturb ( ¯ f, Q , γ, ˜ D x ) ∈ G ) ≤ e m 2 γ 2 ( 1+ γ · √ log ( 1 δ ) ) · ∥ ¯ f -¯ f ′ ∥ m · P ( Perturb ( ¯ f ′ , Q , γ, ˜ D x ) ∈ G ) + δ. On the other hand, if Q = Lap (1) , then

Proof. To prove the first statement, let B δ denote the event that ∥ ζ ∥ m ≤ γ · √ log ( 1 δ ) and let

be the density of γ · ζ . Note that P ( B c δ ) ≤ δ by Lemma 8. We compute:

For the second term, we compute:

The first claim follows. To prove the second claim, we may repeat the same argument with Q = Lap (1) and δ = 0 . Indeed, observe that if

where the second inequality follows from Cauchy-Schwarz. Plugging this ratio into the above argument yields the second claim.

C.4 Concluding the Proofs of Differential Privacy

In this section, we combine the results from Appendix C.3 with the stability results from Appendix C.6 to prove the differential privacy guarantees of Algorithm 2 and Algorithm 3. We separate this section into two lemmas, each corresponding to one of the algorithms. We begin with the more general result. Note that this lemma does not quite follow immediately from combining the stability guarantee with the results of Appendix C.3 as we wish to assume a uniform lower bound on ∥ f ∥ µ whereas Lemma 1 requires a uniform lower bound on ∥·∥ m . We apply a result from Appendix C.5 below to reconcile this discrepancy.

Lemma 10 (Differential Privacy Guarantee for Algorithm 2) . Suppose that F : X → [ -1 , 1] is a function class and let µ ∈ ∆( X ) such that ∥ f ∥ 2 ≥ 2 3 for all f ∈ F . Suppose further that ℓ is bounded in [0 , 1] . Let δ > 0 and suppose that m,γ,η , and J are such that and

Then Algorithm 2 is ( ε, δ ) -differentially private for Q = N (0 , 1) .

Proof. The result follows immediately by combining Lemmas 1 and 9 assuming we have a lower bound on κ . Indeed, by Lemma 12, it holds with probability at least 1 -δ that inf f ∈F ∥ f ∥ m ≥ 1 4 . The result follows.

We also have a guarantee for the more specialized algorithm.

then

Lemma 11 (Differential Privacy Guarantee for Algorithm 3) . Suppose that F : X → [ -1 , 1] is a convex function class and suppose that ℓ is convex and λ -Lipschitz in its first argument. If δ > 0 and m,γ,η are such that

then Algorithm 3 run with Q = N (0 , 1) is ( ε, δ ) -differentially private. On the other hand, if δ = 0 and

then Algorithm 3 run with Q = Lap (1) is ε -purely differentially private.

Proof. This follows immediately by combining Lemmas 2 and 9.

These results show that for any choice of ν , Algorithms 2 and 3 are differentially private. In the next section we show that if ν is σ -smooth with respect to µ , then the algorithms are also PAC learners with respect to ν .

C.5 PAC guarantees for Algorithms 2 and 3

The previous sections have shown that Algorithms 2 and 3 are differentially private, which comprises the main difficulty of our analysis. Here we apply standard learning theoretic techniques to show that if ν is σ -smooth with respect to µ , then the algorithms are also PAC learners with respect to ν . This proof rests on three main results: first, we recall a norm comparison guarantee in high probability that allows us to relate ∥·∥ µ to ∥·∥ m ; second, we recall a classical uniform deviations bound for empirical processes; and third, we show that the perturbed empirical minimizer ¯ f has similar loss to the empirical minimizer f ERM of a loss function as long as the perturbation is not too large. Combining all three results will result in a PAC learning guarantee for Algorithms 2 and 3.

We begin with the following lemma, which is a fairly standard result in learning theory. To state the lemma, we recall from Definition 6 that the worst-case Gaussian complexity is defined as

We then have the following control on ∥·∥ µ in terms of ∥·∥ m :

Lemma 12. Suppose that F : X → [ -1 , 1] is a function class and let µ ∈ ∆( X ) with Z 1 , . . . , Z m ∼ µ independent. Then for any β > 0 , it holds with probability at least 1 -β that for all f, f ′ ∈ F ,

Because the proof is relatively standard [Bousquet, 2002, Rakhlin et al., 2017], but also a technical digression, we defer it to Appendix C.7 and continue with our arguments.

Our second lemma is a standard uniform deviation bound for empirical processes:

Lemma 13. Let F : X → [ -1 , 1] be a bounded function class and let D denote a data set of ( X i , Y i ) ∼ ν be independent. Then for any β > 0 , with probability at least 1 -β , it holds that

Proof. This follows immediately from combining Wainwright [2019, Theorem 4.10] with Van Handel [2014, Lemma 7.4].

Finally, we show that the perturbed empirical minimizer ¯ f has similar loss to the empirical minimizer f ERM of a loss function as long as the perturbation is not too large.

Lemma 14. Let F : X → [ -1 , 1] denote a function class and let ℓ : [ -1 , 1] × 2 → [0 , 1] denote a bounded loss function convex in the first argument and let D denote a dataset of size n . For η > 0 , let ¯ f be as in Line 5 of Algorithm 2. Then for any β > 0 , with probability at least 1 -β ,

If instead we let ¯ f be as in Line 3 of Algorithm 3, then almost surely,

Proof. Applying Lemma 5 and noting that ¯ f, f ERM ∈ F and applying Lemma 13 yields

The second statement follows immediately by noting that 0 ≤ ∥ f ∥ m ≤ 1 for all f ∈ F and letting R ( f ) = η · ∥ f ∥ m .

For the first statement, we note that by convexity of ℓ , it holds that

To prove the second statement, we observe that by the Borell-Tsirelson-Ibragimov-Sudakov inequality (see, e.g., Wainwright [2019, Example 2.30]) and the fact that E [ ω ( j ) m ( f ) 2 ] ≤ 1 for all f ∈ F , that for all j ∈ [ J ] , with probability at least 1 -β , it holds that

Applying symmetry and a Chernoff bound tells us that with probability at least 1 -β it holds that

The result follows.

Before continuing, we prove Lemma 5 from Section 4:

Proof of Lemma 5. For an arbitrary regularizer R : F → R , if we let ¯ f ∈ argmin f ∈F L D ( f ) + R ( f ) , then by definition L D ( ¯ f ) + R ( ¯ f ) ≤ L D ( f ERM ) + R ( f ERM ) and so

where f ERM ∈ argmin f ∈F L D ( f ) is the ERM.

Combining these three lemmas yields the following PAC learning guarantee for Algorithm 2.

Lemma15 (PAC Learning Guarantee for Algorithm 2) . Suppose that F : X → [ -1 , 1] is a function class and ℓ : [ -1 , 1] × 2 → [0 , 1] is a bounded loss function λ -Lipschitz and convex in the first argument. Let µ ∈ ∆( X ) and suppose that ν is σ -smooth with respect to µ . For any β > 0 it holds with probability at least 1 -β that

for ̂ f returned by Algorithm 2 with Q = N (0 , 1) and D a dataset of size n .

Proof. We compute:

where the first inequality uses Jensen's and the second usesthe fact that ν is σ -smooth. We now observe that by Lemma 8, with probability at least 1 -β ,

Combining this with Lemma 14 yields

Applying Lemma 12 concludes the result.

Similarly, we have a result for Algorithm 3; note that while convexity of ℓ is required to demonstrate that Algorithm 2 is a PAC learner, although is irrelevant to the privacy guarantee in Lemma 10, the situation for Algorithm 3 is reversed in that convexity is not required to demonstrate that Algorithm 3 is a PAC learner while it is necessary for the privacy guarantee in Lemma 11.

Lemma 16 (PAC Guarantees for Algorithm 3) . Suppose that F : X → [ -1 , 1] is a convex function class and ℓ : [ -1 , 1] × 2 → [0 , 1] is a bounded loss function λ -Lipschitz in the first argument. Let µ ∈ ∆( X ) and suppose that ν is σ -smooth with respect to µ . For any β > 0 it holds with probability at least 1 -β that

for ̂ f returned by Algorithm 3 with Q = N (0 , 1) and D a dataset of size n . Similarly, if we replace Q = Lap (1) , then with probability at least 1 -β ,

Proof. The proof of the first statement is identical to that of Lemma 15 with the exception of replacing (11) by (12) in the invocation of Lemma 14. The second statement is also identical but now replacing (9) by (10) when applying Lemma 8.

With these results in hand, along with those from Appendix C.4, all that remains to conclude the proofs of the main theorems is to tune the hyperparameters and control the complexity terms. We do this in the next section.

C.6 Concluding the Proofs of Theorems 2 and 3

In this section, we combine the results from Appendices C.4 and C.5 to prove the main theorems. The main theorems in the text, Theorems 2 and 3, follow immediately from the following two results. We begin by stating a more detailed version of Theorem 2:

Theorem 6. Suppose that F : X → [ -1 , 1] is a function class such that G m ( F ) = O ( √ d ) for some d ∈ N and ℓ : [ -1 , 1] × 2 → [0 , 1] is convex and λ -Lipschitz in the first argument. If we let Q = N (0 , 1) in Algorithm 1 and set and

then Algorithm 2 is ( ε, δ ) -differentially private and an ( α, β ) -PAC learner with respect to any ν that is σ -smooth with respect to µ .

Proof. This follows by combining Lemma 10 with Lemma 15 and plugging in the parameter choices.

We also have a result for Algorithm 3:

Theorem 7. Suppose that F : X → [ -1 , 1] is a convex function class such that G m ( F ) = O ( √ d ) for some d ∈ N and ℓ : [ -1 , 1] × 2 → [0 , 1] is convex and λ -Lipschitz in the first argument. If we let Q = N (0 , 1) in Algorithm 1 and set

and

then Algorithm 3 is ( ε, δ ) -differentially private and an ( α, β ) -PAC learner with respect to any ν that is σ -smooth with respect to µ .

On the other hand, if we set Q = Lap (1) in Algorithm 1 and set

then Algorithm 3 is ε -purely differentially private and an ( α, β ) -PAC learner with respect to any σ -smooth ν .

Proof. This follows immediately by combining Lemma 11 with Lemma 16 and plugging in the parameter choices, with the pure differential privacy guarantees coming from the second halves of each lemma.

As a final remark, we note that Lemmas 15 and 16 are both phrased entirely in terms of G m ( F ) and thus apply to function classes F such that G m ( F ) = ω (1) . Such non-donsker [Wainwright, 2019, Van Handel, 2014] classes can still be learned in the PAC framework, albeit with slower rates. Indeed, it is immediate from the above results that as long as G m ( F ) = o ( √ m ) , then appropriately tuning the hyperparamters results Algorithms 2 and 3 being differentially private PAC learners with respect to ν . It is well-known that (with our scaling) G m ( F ) = o ( √ m ) is a necessary condition for PAC learnability even absent a privacy condition [Wainwright, 2019, Van Handel, 2014] and thus our results qualitatively demonstrate that private learnability with public data is possible whenever non-private learning is possible.

C.7 Proof of Lemma 12

Replacing F by F - F = { f -f ′ | f, f ′ ∈ F} and noting that the uniform bound only increases by a factor of 2 and the Rademacher complexity increases at most by a factor of 2, we observe that it suffices to prove the result for f ′ = 0 . We thus instead prove the notationally simpler claim that with probability at least 1 -β , for all f ∈ F ,

We first note that by Bousquet [2002, Theorem 6.1], with probability at least 1 -β , it holds that for all f ∈ F ,

for some universal constant C , with

Taking square roots on both sides of (13) shows that it suffices to upper bound r . For the remainder of the proof, we do this.

In order to proceed, we recall the following standard definition of covering numbers.

Definition 8. Let F be a function class and ∥·∥ m, ∞ the the L ∞ norm on the empirical measure on Z 1 , . . . , Z m ∈ X , i.e., ∥ f ∥ m, ∞ = max i ∈ [ m ] | f ( Z i ) | . We say that f 1 , . . . , f N is an ε -cover with respect to ∥·∥ m, ∞ if for all f ∈ F there is some f j such that ∥ f -f j ∥ m, ∞ ≤ ε . We then let N m, ∞ ( F , ε ) denote the size of the smallest ε -cover of F with respect to ∥·∥ m, ∞ .

The notion of a cover is standard throughout learning theory and can be used to control the Rademacher and Gaussian complexities [Dudley, 1969, Van Handel, 2014, Wainwright, 2019]. We will use it to control r .

Proceeding with the proof, let N m, ∞ ( F , u ) denote the covering number of the function class F with respect to ∥·∥ m, ∞ at scale u > 0 . We then claim that r can be upper bounded by any r satisfying

We also claim that for any r > G m ( F ) , the following holds:

for some universal constant C . We now suppose that (15) and (16) hold and set r = Cm -1 / 2 · log 3 ( m ) · G m ( F ) . Then it is immediate that r is a member of the set in (14).

We now prove the two claims.

Proof that a solution to (15) is an upper bound on r . By a standard Dudley Chaining argument [Van Handel, 2014, Wainwright, 2019], it holds that

Letting f 1 , . . . , f M ∈ F ∩ { ∥ f ∥ 2 m ≤ r 2 } be a proper u -cover of F ∩ { ∥ f ∥ 2 m ≤ r 2 } with respect to ∥·∥ m, ∞ at scale s ≤ r and π : F → { f i } be projection to the cover, we have

where we used the fact that a proper covering at scale ε has size bounded by a covering at scale ε/ 2 by the triangle inequaltiy. Substituting into (17) and rescaling yields the claim.

Proof that (16) holds. This proof goes through fat shattering numbers, a complexity measure taking a function class and a scale u > 0 and returns fat ( F , u ) ∈ N [Bartlett et al., 1994]. We do not need the full definition of fat shattering numbers and defer to [Bartlett et al., 1994, Srebro et al., 2010, Rudelson and Vershynin, 2006] for details. We only need the following two properties. First, for any u > 0 , it holds by Rudelson and Vershynin [2006] that

Plugging (18) into the left hand side of (16) and then applying (19) concludes the proof of the claim.

D Classification and Analysis of RRSPM

In this section, we provide full proofs for the guarantees of Algorithm 4. In Section D.1, we describe the concept of universal identification sets and the result of Neel et al. [2019] which plays a crucial role in the privacy analysis of our algorithm. In Section D.2, we formally prove Lemma 6 which is the technical lemma used for the proof of differential privacy based on Neel et al. [2019]. We then continue in Section D.3 by applying standard learning theoretic techniques to demonstrate that our algorithm is an accurate classifier.

D.1 Universal Identification Set based Algorithm

In this section, we formally define universal identification sets and informally describe the algorithm used in Neel et al. [2019]. Intuitively, a universal identification set captures the combinatorial property of a function class that all distinct functions in the function class disagree on at least one point from the data universe. For many natural classes, the size of the universal identification set is proportional to the VC dimension of the function class.

We now describe the algorithm from Neel et al. [2019] and explain the usefulness of universal identification sets. First, we formally define the notion of universal identification set.

Definition 9. (Universal Identification Set) A set U ⊆ X is a universal identification set for a hypothesis class F if for all pairs of functions f, f ′ in the hypothesis class F , there is a x ∈ U such that:

̸

Additionally, if |U| = m , we say that F has a universal identification set of size m .

Assuming the existence of a universal identification set for the function class F of size m denoted by U = { U 1 , . . . , U m } , Neel et al. [2019] showed that the following algorithm(called RSPM ) is an ε - pure differentially private and ( α, β ) -accurate algorithm.

Algorithm 5: RSPM

1: Input ERM oracle ERM , dataset D = { ( X i , Y i ) | 1 ≤ i ≤ n } , hypothesis class F , universal identification set U = { U 1 , . . . , U m } , loss function ℓ : Y × Y → { 0 , 1 } .

2: Draw weights ξ = { ξ 1 , . . . , ξ m } such that ξ i ∼ Lap(2 m/ε ) .

3: Draw labels ˜ Y = { ˜ Y 1 , . . . , ˜ Y m } such that ˜ Y i ∼ Uni( { 0 , 1 } ) .

4: Define L ξ , D , U : F → R such that

5: Get ˆ f = ERM ( F , L ξ , D , U ) .

RSPM roughly simulates 'Report-Noisy-Min'(Dwork et al. [2014]) attempting to output a function that minimizes a perturbed estimate, where the perturbation is sampled from a Laplace distribution. Astraight forward implementation of 'Report-Noisy-Min" to minimize over all perturbed estimates of functions, it'd have to check for all functions in F and thus the computational complexity would depend on the size of F . RSPM avoids this problem by implicitly perturbing the function evaluations via an augmented dataset. The proof of privacy thus exploits the structure of the universal identification set.

Although many natural function classes have bounded universal identification sets, their existence is not as general as having bounded VC dimension. In our work, we only assume finite VC dimension of the function class.

D.2 Privacy Analysis

In this section, we prove that Algorithm 4 is differentially private. A notion that we will need is that of a projection of a hypothesis class onto a set of points from the domain.

Definition 10. (Projection) Given a hypothesis class F ⊆ Y |X| and a subset Z = { z 1 , . . . z m } of the feature space X , we define the projection of F onto Z to be F| Z = { ( f ( z 1 ) , . . . , f ( z m )) : f ∈ F} .

In the following sections, we will interchangeably think of the projection of a hypothesis class onto a set of points Z as a set of functions on Z or as a set of vectors in Y | Z | .

̸

By construction, F| Z has the property that any two distinct functions f = f ′ ∈ F| Z must disagree on at least one point z ∈ Z . We encapsulate this as the following lemma.

̸

Lemma 17. Let F be a hypothesis class and let Z = { z 1 , . . . , z m } be a set of points in the instance space X . Then, for all f = f ′ ∈ F| Z , there exists z ∈ Z such that f ( z ) = f ′ ( z ) .

̸

Remark 2. This property is analogous to the notion of a universal identification set considered in Neel et al. [2019]. In particular, the above lemma can be seen as the statement that the set Z is a universal identification set for the class F| Z .

We first provide an informal sketch of the proof of privacy. In the later sections, we formalize these ideas. Let ˆ f ∈ F be any arbitrary function and let D , D ′ be any pair of neighbouring datasets. We show that P ( RRSPM ( D ) = ˆ f ) ≤ e ε P ( RRSPM ( D ′ ) = ˆ f ) . By the definition of the projection, there is some ˜ f ∈ F| ˜ D x that is consistent with the labelling of the selected function ˆ f . P ( ERM ( F , L ξ , D , ˜ D ) = ˜ f ) ≤ e ε P ( ERM ( F , L ξ , D ′ , ˜ D ) = ˜ f ) since privacy for ˆ f follows from the post-processing property of differential privacy.

We now provide with a proof sketch for the main technical lemma showing that the privacy is preserved over the projected function class. Optimizing a loss function perturbed by Laplace-weighted examples implicitly tries to implement 'Report-Noisy-Min' algorithm outputting a function that minimizes a perturbed estimate. For any neighbouring datasets D and D ′ , the evaluation of any function ˜ f can differ by at most 1 . We show that the set of public points ˜ D is a universal identification set for the set of functions projected onto ˜ D and leverage this to prove that whenever the shift in the noise vectors is bounded by 2 in every coordinate, then ˜ f is the minimizing function when switching from D to D ′ . This intuition is made precise in Lemma 18.

Lemma 18. Let D , D ′ be two neighbouring data sets, and let ˜ D = ( ˜ D x , ˜ D y ) ∈ ( X × { 0 , 1 } ) m where ˜ D x = { Z 1 , . . . , Z m } and ˜ D y = { ˜ Y 1 , . . . , ˜ Y m } . Define E ( f ˜ D , D , ˜ D ) = { ξ : ERM ( F| ˜ D x , L ξ , D , ˜ D ) = f ˜ D } , where L ξ , D , ˜ D is a functional as defined below:

Let ξ = { ξ 1 , . . . , ξ m } such that ξ i ∼ Lap(2 m/ε ) . Given a fixed f ˜ D ∈ F| ˜ D x , define a mapping Ψ f ˜ D ( ξ ) : R m → R m on noise vectors as follows:

Equivalently, Ψ f ˜ D ( ξ ) i = ξ i + 2(1 -2 ℓ ( f ˜ D ( Z i ) , ˜ Y i )) . Let ξ ∈ E ( f ˜ D , D , ˜ D ) where f ˜ D ∈ ERM ( F| ˜ D x , L ξ , D , ˜ D ) . Then Ψ f ˜ D ( ξ ) ∈ E ( f ˜ D , D ′ , ˜ D ) .

̸

Proof. Let Ψ f ˜ D ( ξ ) = ξ ′ = ( ξ ′ 1 , . . . ξ ′ m ) . Our goal is to show that for every f ∈ F| ˜ D x such that f = f ˜ D , we have L ξ ′ , D ′ , ˜ D ( f ) > L ξ ′ , D ′ , ˜ D ( f ˜ D ) . First, recall that by our assumption for all f ∈ F| ˜ D x , we have

This proves that Ψ f ˜ D ( ξ ) ∈ E ( f ˜ D , D ′ , ˜ D ) .

Lemma 19 (Laplace shift) . Let ξ = { ξ 1 , . . . , ξ m } such that ξ i ∼ Lap(2 m/ε ) . Fix some noise realization r ∈ R m and fix a hypothesis in the projection set f ∈ F| ˜ D x . Then,

where Ψ f ( ξ ) i = ξ i +2(1 -2 ℓ ( f ( Z i ) , ˜ Y i )) as defined in Lemma 18.

Proof. Let i ∈ [ m ] be any index and let r ∈ R m . Since ξ i ∼ Lap(2 m/ε ) , we know

For any t, t ′ ∈ R such that | t -t ′ | ≤ 2 , we get

Since for all i ∈ [ d ] , | Ψ f ( r ) i -r i | ≤ 2 , we get

̸

Wenowargue that L ξ ′ , D ′ , ˜ D ( f ) -L ξ ′ , D ′ , ˜ D ( f ˜ D ) is strictly positive for all f ∈ F| ˜ D x such that f ˜ D = f . To see this we calculate,

where the second inequality follows from the fact that D and D ′ differ in only one entry and ℓ is 1 -sensitive. The last equation follows from statement 20.

̸

We know from Lemma 17 that there exists a Z ∈ ˜ D such that f ( Z ) = f ˜ D ( Z ) . Recall that ξ ′ i = ξ i +2(1 -2 ℓ ( f ˜ D ( Z i ) , ˜ Y i )) . By construction, each term is non-negative. Therefore,

To wrap up, we can bound

Our proof of privacy also makes use of the following lemma, which says that minimizers are unique with probability 1 [Neel et al., 2019, Lemma 4].

Lemma 20. Let ˜ D = ( ˜ D x , ˜ D y ) ∈ ( X × { 0 , 1 } ) m . Consider F| ˜ D x , where F| ˜ D x is the projection of F on ˜ D x . For every dataset D , there is a subset B ⊆ R m such that:

P ( ξ ∈ B ) = 0 and

On the restricted domain R m \ B , there is a unique minimizer ˜ f ∈ arg min f ∈F| ˜ D x L ξ , D , ˜ D ( f ) .

Using standard results about Laplace perturbations stated in Lemma 19 and Lemma 20 and using the perturbation coupling bound in Lemma 18, we get the our formal statement of privacy for optimizing over the projected function class as follows:

Lemma 21. (Privacy over Projection) Let D , D ′ be arbitrary datasets containing n points each. Let ˜ D = ( ˜ D x , ˜ D y ) ∈ ( X × { 0 , 1 } ) m . Then,

Proof. We calculate,

Theorem 8. Algorithm 4 is ε -pure differentially private.

Proof. Let D and D ′ be any neighbouring datasets. Fix any function ˆ f ∈ F . We now show that

where the probability is taken over the randomness of the algorithm. From the definition of the projection, we know that there exists a unique function in the projection say, ˜ f ∈ F| ˜ D x such that ˆ f ( Z i ) = ˜ f ( Z i ) for all i ∈ [ m ] .

Using Lemma 21, we know that

As defined in the algorithm, let ˜ L ( f ) = ∑ m i =1 ℓ ( f ( Z i ) , ˜ f ( Z i )) . From the definition of ˜ L , it follows that ˆ f ∈ arg min f ∈F ˜ L ( f ) . Following the post-processing guarantee of differential privacy, it is easy to see that

with probability 1 -β.

Proof. Let α ′ = O (√ d +log(1 /β ) m ) . First we show that | L ( ˆ f ) X ∼ µ Y = ˜ f ( X ) -L ( ˜ f ) X ∼ µ Y = ˜ f ( X ) | ≤ 2 α ′ with probability at least 1 -β . Consider the dataset ˆ D = { ( X 1 , ˜ f ( X 1 ) , . . . , ( X m , ˜ f ( X m )) } we minimize over and output ˆ f . Using Theorem 9 we know that,

Since L ( ˜ f ) X ∼ µ Y = ˜ f ( X ) = 0 and | L ˜ D ( ˆ f ) -L ˜ D ( ˜ f ) | = 0 and we get that with probability at least 1 -β ,

D.3 Accuracy Analysis

In this section we analyze the accuracy of our algorithm. Let f ∗ denote the function in the hypothesis class that minimizes the loss with respect to the distribution ν i.e. f ∗ ∈ arg min f ∈F L ν ( f ) . Let ˆ f denote the output hypothesis of our algorithm. We show that the loss of ˆ f is close to f ∗ with respect to the data generating distribution ν .

As in the algorithm description, let ˜ f be the function that minimizes the perturbed loss. Our algorithm outputs ˆ f whose labelling is consistent with ˜ f on the public dataset ˜ D x . Since ˜ D x is sampled from the base distribution, it follows from VC theorem that ˆ f and ˜ f are close under the base distribution µ . We show in Lemma 22 that ˆ f and ˜ f are close under ν by leveraging that ν x is a σ -smooth distribution. Using standard results about Laplace perturbations, we show that f ′ is close to ˜ f in Lemma 23, where f ′ is the empirical risk minimizer over D . Using the VC theorem, it is easy to see that f ′ is close to f ∗ and consequently using the triangle inequality we finish the proof by showing that f ∗ and ˆ f are close under ν .

We now state the VC theorem below which we use in our analysis.

̸

Theorem 9. Let D = { ( X 1 , Y 1 ) , . . . , ( X n , Y n ) } where for all i ∈ [ n ] , ( X i , Y i ) ∈ X × { 0 , 1 } are sampled from a fixed distribution ν . Let L D ( f ) = 1 n |{ i : f ( X i ) = Y i }| and let L ν ( f ) = E ( X,Y ) ∼ ν [ ✶ ( f ( X ) = Y ] . If the function class F has V C dimension d then,

In particular if f ∗ ∈ arg min f ∈F L D ( f ) then,

Lemma 22. Let ν x be a σ -smooth distribution that is ∥ ∥ ∥ dν x dµ ∥ ∥ ∥ ≤ 1 σ . Let L ν be the loss function as defined in Defintion 2. Let ˆ f be the hypothesis returned by our algorithm and let ˜ f ∈ ERM ( F , L ξ , D , ˜ D ) . Then,

̸

with probability 1 -β .

Proof. Following the algorithm we know that for all i ∈ [ m ] , ξ i ∼ Lap(2 m/ε ) . Using Chernoff's bound and a union bound we get that with probability 1 -β ,

̸

̸

We now show that P X ∼ ν x ( ˆ f ( x ) = ˜ f ( x )) ≤ 2 α ′ /σ with probability at least 1 -β .

̸

̸

̸

Having shown that with probability at least 1 -β ,

̸

we now prove that with probability at least 1 -β ,

which is equivalent to the theorem statement. Using the triangle inequality we get

̸

̸

̸

̸

where the second equation follows from the observation that for any value of X , if ˆ f ( X ) = ˜ f ( X ) then the difference of the probabilities equate to 0 and if ˆ f ( X ) = ˜ f ( X ) then the difference of the probabilities equate to 1 and the last equation follows from Equation 22.

Substituting the value of α ′ proves that with probability 1 -β ,

Lemma 23. Let F be a function class with V C dimension d . Let D = { ( X 1 , Y 1 ) , . . . , ( X n , Y n ) } and ˜ D x = { Z 1 , . . . , Z m } where Z i ∼ µ . Let L D be the loss function as defined in Definion 2. Let f ′ ∈ ERM ( F , L D ) and let ˜ f ∈ ERM ( F , L ξ , D , ˜ D ) then,

̸

̸

̸

Since ˜ f ( Z i ) ∈ { 0 , 1 } for all Z i ∈ ˜ D , with probability 1 -β , L ξ , D , ˜ D ( ˜ f ) ≥ L D ( ˜ f ) -m · 2 m log( m/β ) ε . Similarly, L ξ , D , ˜ D ( f ′ ) ≤ L D ( f ′ ) + m · 2 m log( m/β ) ε . Dividing by n and combining the bounds we get,

with probability 1 -β.

Theorem 10. Let ˆ f be as defined in Algorithm with D sampled from some distribution ν such that ∥ ∥ ∥ dν x dµ ∥ ∥ ∥ ≤ 1 σ . Suppose the function class F has VC-dimension d then setting

yields an ε - pure differentially private ( α, β ) -learner as long as

where ˜ Ω hides log factors.

Proof. Let ν be the distribution from which the given dataset D is sampled where ∥ ∥ ∥ dν x dµ ∥ ∥ ∥ ≤ 1 σ . Let ˆ f be the output of our algorithm, f ′ ∈ arg min f ∈F L D ( f ) , f ∗ be the target function from our function class F and ˜ f ∈ arg min f ∈F L ξ , D , ˜ D ( f ) . We wish to compare the guarantee of the function ˆ f with respect to the function f ∗ with respect to the distribution ν i.e. | L ν ( ˆ h ) -L ν ( f ∗ ) | .

In our analysis below, we break the loss function several times using triangle inequality and bound each term using previously stated results via a union bound.

Setting

and combining the guarantee from Theorem 8 we get that RRSPM (Algorithm 4) is a ε - pure differentially private ( α, β ) -learner.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately captures all the contributions of the paper. All the claims are formally stated in the main body and the appendix of the paper. Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are that the work requires public, unlabelled data and that the statistical rates of our efficient algorithms do not match those achievable by inefficient methods. Both limitations are stated clearly and repeatedly throughout the text.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Proof sketches are provided in the main body and detailed arguments are found in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: Did not run experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: Did not run experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: Did not run experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Did not run experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: Did not run experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Evident.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Evident.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Does not apply.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: Does not apply.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Does not apply.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Does not apply.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Does not apply.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores methods for private learning by leveraging public unlabeled data. Previous work in this area utilized public data to construct an $\\alpha$-covering and then employed an exponential mechanism to output a hypothesis privately. However, the primary drawback of this approach was its exponential running time in both building the covering and sampling from the exponential mechanism. In contrast, this work assumes the availability of an Empirical Risk Minimization (ERM) oracle, which can find the minimizer of a function class given a loss function. The authors propose new private and oracle-efficient algorithms with polynomial complexity in terms of Gaussian complexity, objective error $\\alpha$, and other parameters.\n\nSTRENGTHS:\nThe inefficiency of previous methods highlights the importance of this problem. This paper makes significant progress by designing private algorithms that achieve oracle efficiency, addressing a crucial gap in the literature.\n\nWEAKNESSES:\nI find it hard to understand the intuitions behind the algorithm design and, hence, hard to appreciate the results, possibly because I am not an expert in learning theory. Several areas remain unclear to me:\n(1). If we already have the strong convexity and assume the L is differentiable, can we compute the gradient, run the gradient descent, and reduce the problem to the convex optimization?\n(2) What is the main motivation for considering Gaussian complexity rather than the VC dimension? Can the results from previous studies be extended to Gaussian complexity?\n(3) There is a lack of discussion on sample complexity. For instance, Theorem 2 requires $n\\ge \\Omega(1/\\alpha^{14})$. It is helpful to discuss the origin of this term.\n(4) What is the state-of-the-art (SOTA) performance without public data? Does the proposed approach with public data outperform current methods in terms of sample complexity?\n\nQUESTIONS:\nPlease refer to the Weaknesses above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the setting of differentially private learning when there is some amount of public data available. A downside of existing algorithms is that they generally use the public data to build a cover which ends up being inefficient. A natural question is to design more efficient algorithms that do make use of public data.\n\nThe main result of this paper is to show that when the Gaussian complexity of the hypothesis class is small then there is a poly-time algorithm for the problem. Furthermore, if the hypothesis class is well-structured (here they use convexity as the structure) then there is an FTRL-type algorithm that is significantly more efficient and also works under pure DP. The authors also mention that their results hold in the setting where the public data and private data may be different and they quantify this via the ratio between the cdf's.\n\nThe actual algorithms themselves actually appear quite simply and practical. In particular, one defines either a noisy or regularized ERM problem and then solve another noisy ERM problem.\n\nSTRENGTHS:\nOverall, I think this a solid contribution to the DP literature. It considers the reasonably well-studied problem of private learning with public data but provides new and simple algorithms for the setting. While I have only looked briefly at the proof techniques, they seem interesting and I think researchers working in this area will be interested to learn about them (e.g. the use of FTRL and anti-concentration type of arguments).\n\nWEAKNESSES:\nNo weaknesses to discuss from my side.\n\nQUESTIONS:\n- In Theorem 1, I am not sure what $d$ refers to. Is it a dimension or it refers to something else? In particular, is there a need for the square root since you just write poly after? Also, why not use $\\mathcal{G}$ for the Gaussian complexity here as you do later in the paper?\n- I wonder if the authors have a sense what about what happens if their CDF ratio assumption in Line 73 fails to hold everywhere but does hold in all except maybe $\\delta$ fraction of the time. I'm thinking in particular if maybe the public data has an exponential tail whereas the private data has a Gaussian tail (or vice-versa). So in particular, the distributions might be fairly similar in most places except in the tail where the ratio can be wildly different. Is this captured in the remark after Definition 3 in line 149?\n\nLIMITATIONS:\nNo concerns from me.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of semi-private learning. In this setting, both public and private data are given while the learning algorithm only has to satisfy differential privacy with respect to the private part. Previous work has showed that leveraging the sample complexity of private learning can be improved by leveraging public data, though the methods are computationally inefficient. \n\nIn this work, the authors propose the first oracle-efficient algorithm for this task, which requires only a polynomial number of calls of  an ERM oracle. Furthermore, they consider the special cases where the functions are convex or the problem is binary classification and obtain algorithms with improved sample complexity and number of oracle calls, and can satisfy strong privacy property (pure DP).\n\nSTRENGTHS:\n1. The paper is the first to give an oracle-efficient semi-private learning algorithm. Oracle efficiency could be useful in practice due to the fact that many algorithms have large theoretical bounds but work quite well in practice. An oracle-efficient algorithm can directly use them as black-boxes. Previous work, though attains better sample complexity, fails to exploit such a fact.\n2. The algorithms work as long as the distribution of private data is smooth w.r.t. that of public data, i.e., the two distributions are not required to be exactly the same.\n3. The algorithms are simple and easy to implement.\n\nWEAKNESSES:\n1. The resulting sample complexity, though being polynomial, is much higher than previous work. \n2. The algorithm requires calls to an ERM oracle. For many tasks, one may only be able to find an approximate minimizer with a small excess risk. It is not clear whether the conclusion still holds if we only have an approximate ERM oracle but not an exact one.\n\nQUESTIONS:\n1. The algorithms construct semi-private learner using ERM oracles. Is it possible to construct it directly using non-private learners (which may not minimize the empirical risk)?\n2. On line 64, page 2, it is said that \"Prior work of Bassily et al.[2018] gave an oracle-efficient algorithm in this setting with somewhat better sample complexity\". Can you elaborate more on this? It looks like their work only guarantees an error of $c\\cdot OPT + \\alpha$ in the agnostic setting, which is weaker than the result obtained in this paper.\n\nLIMITATIONS:\nDiscussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

comments: SUMMARY:
The paper investigates oracle-efficient semi-private learning. It provides a general framework for transforming an efficient non-private learner into an oracle-efficient semi-private learner for smooth data distributions. For convex and Lipschitz loss functions, as well as binary classification loss, they instantiate the general framework and provide an implementable algorithm with polynomial oracle calls. The output of the proposed algorithm achieves high accuracy with high probability for any hypothesis class with a bounded VC dimension under smooth data distributions. Although the initial algorithm does not achieve optimal sample complexity, the authors introduce a second algorithm inspired by the Follow-the-Regularized-Leader approach from online learning. This improved algorithm enhances the sample complexity bound and requires only two oracle calls.

STRENGTHS:
- The paper is clearly written. 
- The proposed methods not only have theoretical guarantees, but are of practical relevance due to its efficiency. For example, algorithm 3 requires only 2 calls of the ERM oracle. 
- The results hold under slight distribution shift between the public and private data.

WEAKNESSES:
The sample complexity bounds rely on the assumption of a $\sigma$-smooth data distribution. For instance, in Theorem 6, the public sample complexity is given by $m = \tilde{\Theta}(1/\sigma)$ and $n = \tilde{\Omega}(1/\sigma^{12})$. Could you discuss whether it's possible to improve this dependence on $\sigma$? Additionally, could you elaborate on the necessity of assuming a smooth data distribution? Understanding the implications and possible relaxations of this assumption would be beneficial.

QUESTIONS:
See weaknesses.

LIMITATIONS:
I don't identify significant limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of identifying a particular class of functions which can be learnt efficiently with unlabelled public and labeled private samples maintaining privacy with respect to the private samples only. The authors propose an algorithmic framework for privately learning such function classes with public unlabelled data. For the general category of functions, if the expectation of the function with respect to the measure is greater than a certain value, the learning is done in two stages. First they learn the unlabelled data by minimizing the loss function with respect to the labeled private data and then add a noisy gaussian process using the function output of the unlabelled data. Then they simply privatize it by using the private output perturbation method. The algorithm can be further simplified for convex functions by simply using L2 regularizer with respect to the function outputs of the unlabelled data instead of using a Gaussian process in case of the general category. They mainly provide the theoretical guarantees by using tools from existing work on Gaussian anti concentration to bound the stability of the learning algorithm on perturbed datasets in terms of the worst case Gaussian complexity of the function class. Then they use the bound on the stability to get a differential privacy guarantee of the output perturbation method. Finally, they show the fact that the algorithm can learn using standard learning theory arguments. These proposed algorithms are oracle efficient in the sense that they only require a polynomial number of calls to the ERM oracle.\n\nSTRENGTHS:\n1. This work contributes to the area of learning theory and differential privacy in two ways. First it identifies certain classes of functions which can be learnt privately with public data in polynomial calls to the ERM oracle. Secondly, it provides the first algorithm with conditional polynomial time learning and privacy guarantee for these classes of functions.\n2. The main novelty in this paper is tightening the result by Block et. Al. [1] to get the stability guarantees for the gaussian process which this algorithm follows paving way for making analysis easier for works which use a similar kind of a framework to learn using auxiliary data.\n3. This paper is very self-contained in terms of the content and all the proofs have been reiterated, with the necessary changes, to ensure that a beginner in the area of learning theory and differential privacy can develop a good understanding of the paper.\n4. The arguments in this paper are mostly direct and easy to follow, while some more complex arguments like the anti concentration expressions are self contained and do not require any additional references.\n\n[1] Adam Block, Yuval Dagan, Noah Golowich, and Alexander Rakhlin. Smoothed online learning is as easy as statistical learning. In Conference on Learning Theory, pages 1716–1786. PMLR, 2022.\n\nWEAKNESSES:\nThe main weakness in the paper is not discussing a clear intuition of the role of public data in the algorithm and how it helps to learn when private learning is not possible. The questions regarding the paper mentioned below also follow the same theme. It would be very helpful if the authors can add a specific and detailed discussion on the role of public data answering the questions below.\n\nQUESTIONS:\nThe paper is theoretically sound and makes sense from the perspective of getting an oracle efficient algorithm for private learning with public data but the intuition behind algorithm 1 is not completely clear. There is part of the algorithm in which the authors add \\omega which is the weighted gaussian sum of the function values of the auxiliary data. I have some questions regarding that\n1. Why was there a need to create a gaussian process with respect to the auxiliary data in the first place? How would the analysis have gone differently or learnability would have been hampered simply by running ERM on the standard loss function with respect to the private data?\n2. Following up with the first question, can the authors emphasize on the point in the analysis where we got an advantage due to using the public data in the mentioned algorithm? Where did we consider the additional information from the public data in the analysis?\n3. Can the authors also give an example or maybe give a specific reference to a paper in which they could mention a particular learning problem which would not be privately learnable but can be learnt with public data? Does this algorithm ensure that the particular example problem can be learnt in polynomial time?\n\nLIMITATIONS:\nThe authors have mentioned the limitations of their work multiple times within their text and have provided some open problems to further the direction of research in this topic.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}