Title: Public-data Assisted Private Stochastic Optimization: Power and Limitations
Venue: NeurIPS 2024
Accepted: yes
Keywords: Differential Privacy

Public Data

Stochastic Optimization

Generalized Linear Model
Abstract: We study the limits and capability of public-data  assisted differentially private (PA-DP) algorithms. Specifically, we focus on the problem of stochastic convex optimization (SCO) with either labeled or unlabeled public data. For complete/labeled public data, we show that any $(\epsilon,\delta)$-PA-DP has excess risk $\tilde{\Omega}\big(\min(\frac{1}{\sqrt{n_{\text{pub}}}},\frac{1}{\sqrt{n}}+\frac{\sqrt{d}}{n\epsilon} ) \big)$, where $d$ is the dimension, ${n_{\text{pub}}}$ is the number of public samples, ${n_{\text{priv}}}$ is the number of private samples, and $n={n_{\text{pub}}}+{n_{\text{priv}}}$. These lower bounds are established via our new lower bounds for PA-DP mean estimation, which are of a similar form. Up to constant factors, these lower bounds show that the simple strategy of either treating all data as private or discarding the private data, is optimal. We also study PA-DP supervised learning with \textit{unlabeled} public samples. In contrast to our previous result, we here show novel methods for leveraging public data in private supervised learning. For generalized linear models (GLM) with unlabeled public data, we show an efficient algorithm which, given $\tilde{O}({n_{\text{priv}}}\epsilon)$ unlabeled public samples, achieves the dimension independent rate $\tilde{O}\big(\frac{1}{\sqrt{{n_{\text{priv}}}}} + \frac{1}{\sqrt{{n_{\text{priv}}}\epsilon}}\big)$. We develop new lower bounds for this setting which shows that this rate cannot be improved with more public samples, and any fewer public samples leads to a worse rate. Finally, we provide extensions of this result to general hypothesis classes with finite \textit{fat-shattering dimension} with applications to neural networks and non-Euclidean geometries.

Public-data Assisted Private Stochastic Optimization: Power and Limitations

Abstract

We study the limits and capability of public-data assisted differentially private (PA-DP) algorithms. Specifically, we focus on the problem of stochastic convex optimization (SCO) with either labeled or unlabeled public data. For complete/labeled public data, we show that any ( glyph[epsilon1], δ ) -PA-DP has excess risk ˜ Ω ( min { 1 √ n pub , 1 √ n + √ d nglyph[epsilon1] }) , where d is the dimension, n pub is the number of public samples, n priv is the number of private samples, and n = n pub + n priv . These lower bounds are established via our new lower bounds for PA-DP mean estimation, which are of a similar form. Up to constant factors, these lower bounds show that the simple strategy of either treating all data as private or discarding the private data, is optimal. We also study PA-DP supervised learning with unlabeled public samples. In contrast to our previous result, we here show novel methods for leveraging public data in private supervised learning. For generalized linear models (GLM) with unlabeled public data, we show an efficient algorithm which, given ˜ O ( n priv glyph[epsilon1] ) unlabeled public samples, achieves the dimension independent rate ˜ O ( 1 √ n priv + 1 √ n priv glyph[epsilon1] ) . We develop new lower bounds for this setting which shows that this rate cannot be improved with more public samples, and any fewer public samples leads to a worse rate. Finally, we provide extensions of this result to general hypothesis classes with finite fat-shattering dimension with applications to neural networks and non-Euclidean geometries.

1 Introduction

The framework of differential privacy has become the primary standard for protecting individual privacy in data analysis and machine learning. Unfortunately, this rigorous framework has also been shown to lead to worse performance on such tasks both empirically and in theory [BST14, PVX + 23]. However, it is often the case that, in addition to a collection of privacy-sensitive data points, analysts have access to a pool of public data, for which guaranteeing privacy protections is not required. This can happen, for example, when consumers deem their own data non-sensitive and opt-in to sell this data to a company. This has motivated a long line of work analyzing how public data can be leveraged in tandem with private data to provide better utility [BNS13, ABM19, BCM + 20, ZWB21, BKS22, AGM + 22, NMT + 23]. In machine learning. for example, two commonly proposed strategies are public pretraining and using public data to identify gradient subspaces [ZWB21, KDRT21]. Public pretraining, in particular, has proven effective in practice [YNB + 22a, BWZK22], and prior work has even identified a specific problem instance where public and private data used in tandem leads to better rates than is possible using only the public or private datasets in isolation [GHN + 23]. Despite this surge of work, theory has struggled to show that public data leads to fundamental rate improvements more generally. Recent work has even shown that, for the problem of pure PA-DP stochastic convex optimization, a small amount of public data, n pub ≤ nglyph[epsilon1]/d , leads to no rate improvement, where n = n pub + n priv and n priv is the number of private samples [LLHR23].

One particularly important version of this problem is in supervised learning when the public data is unlabeled. This setting has found importance in medical domains and deep learning more generally [LW19, SCZ + 20, PAE + 17]. Notably, unlabeled data is much less time intensive to collect than labeled data. Due to this fact, and the fact that the unlabeled public data does not contain the same kind of information contained in the private data, the regime n pub = Ω( n priv ) is meaningful both in theory and in practice. We also note this setting is a stronger (in terms of privacy) version of the label-private setting, where only the labels of the dataset are considered private [CH11, BNS13].

Motivated by the importance of these settings and the lack of existing theory for them in stochastic optimization, we study fundamental limitations and applications of public data in ( glyph[epsilon1], δ ) -PA-DP stochastic optimization. In the case where the public data is complete/labeled, we show that the application of public data is fundamentally limited. We then contrast this result with new results in the unlabeled public data setting. In this setting, we provide new results for GLMs, and extend these results to more general hypothesis classes, with finite fat-shattering dimension, and non-Euclidean geometries.

1.1 Our Contributions

We outline our primary contributions in the following.

Limits of Private Stochastic Convex Optimization with Public Data. First, we show a tight lower bound for the problem of differentially-private stochastic convex optimization (DP-SCO) assisted with complete public data, that is, the public data and private data have the same number of features (and labels when applicable). Specifically, we show a lower bound of Ω ( min { 1 √ n pub , 1 √ n + √ d nglyph[epsilon1] } ) on the excess population risk for this problem. When d ≥ nglyph[epsilon1] and n pub ≤ n log(1 /δ ) , we further √

improve this lower bound to Ω ( min { 1 √ n pub , 1 √ n + d log(1 /δ ) nglyph[epsilon1] } ) . This lower bound is matched by the simple upper bound strategy which either discards the private data entirely and outputs the public mean or simply treats all data-points as private. Barring constant factors, this shows more sophisticated attempts at leveraging public data will yield no benefit. These results also hold even for generalized linear models. Our results are based on new results we establish for DP mean estimation with public data, and a reduction of mean estimation to SCO. We note that previous work [LLHR23], on this problem either focused on the pure PA-DP case when n pub ≤ nglyph[epsilon1]/d , or, in the approximate PA-DP case, did not obtain the dimension dependence. Our mean estimation lower bound uses a novel analysis of fingerprinting codes [BUV14], and our SCO reduction further builds on ideas from [BST14, CWZ21]. We also show that, when d ≥ nglyph[epsilon1] , our lower bounds for approximate PA-DP SCO directly imply a tight lower bound for pure PA-DP.

Private Supervised Learning with Unlabeled Public Data. While the previously discussed results show there is no hope for leveraging public data in 'interesting' ways, even for GLMs, they do not preclude settings where the public data is less informative. In particular, in the setting where the

public data is unlabeled , it makes sense to even consider n pub ≥ n priv. In this setting, we provide the following results.

For (Euclidean) GLMs we develop an efficient algorithm which, given ˜ O ( n priv glyph[epsilon1] ) unlabeled public data points, achieves the dimension independent rate ˜ O ( 1 √ n priv + 1 √ n priv glyph[epsilon1] ) . We obtain this result via a dimensionality reduction procedure of the private feature vectors using the public data, and then running an efficient private algorithm in the lower dimensional space. The key idea is that public data can be used to identify a low dimensional subspace, which under the appropriate metric acts as a cover for the higher dimensional space. We elucidate the tightness of our upper bound by proving two new lower bounds which show that access to a greater number of unlabeled public samples cannot improve this rate, and that any fewer public samples lead to a worse rate. While dimension independent rates for the GLMs have previously been developed in the unconstrained setting [SSTT21, ABG + 22], in the constrained setting which we study, dependence on dimension is known to be unavoidable even for GLMs if no public data is available [BST14]. Our result thus allows us to bypass these limitations.

By observing that the key requirement in our GLM result is the construction of an appropriate cover, we extend this result to general hypothesis classes with bounded fat-shattering dimension . In the non-private setting, it is known that finiteness of fat-shattering dimension characterizes learnability of real-valued predictors with scale-sensitive losses [BLW94, ABDCBH97]. In the private setting, such a result is not known, and is in fact impossible in the proper learning setting. This follows from the fact that norm bounded linear predictors, regardless of their (ambient) dimension d , have the same fat-shattering dimension [SST10]. However, it is known that they are not learnable privately in high dimensions d ≥ ( nglyph[epsilon1] ) 2 [BST14]. In contrast, in the PA-DP setting, we show that it is possible to properly learn such classes with a rate of roughly O ( R n priv ( H ) + inf α> 0 ( fat α ( H ) n priv glyph[epsilon1] + α )) , where R n priv ( H ) denotes the Rademacher complexity of H and fat α ( H ) denotes its fat-shattering dimension at scale α (see Section 2 for preliminaries).

As applications of our result for hypothesis classes with bounded fat-shattering dimension, we obtain guarantees for learning feed-forward neural networks and non-Euclidean GLMs. In particular, for depth M feed-forward neural networks with weights bounded as ‖ W j ‖ F ≤ R j and 1 -Lipschitz positive homogeneous activation, we achieve an excess risk bound of essen-

(

√

M

∏

√

j M

n

tially

O

=1

priv

+

∏

j M

n

=1

priv

R

glyph[epsilon1]

. For non-Euclidean GLMs, our guarantees are dimension-independent which is not known to be achievable, as of yet, even in the unconstrained setting with no public data (unlike Euclidean GLMs).

1.2 Related Work

With regards to labeled public data, the most directly related work to ours is the recent work of [LLHR23]. This work proves a lower bound of Ω ( min { 1 √ n pub , 1 √ n + 1 nglyph[epsilon1] }) for approximate PA-DP mean-estimation/SCO. We note that our results for approximate PA-DP crucially obtain a dependence on d that is the key 'price' paid for privacy in this setting. [LLHR23] also show a lower bound of Ω ( min { 1 √ n pub , 1 √ n + d nglyph[epsilon1] }) on a pure PA-DP mean estimation/SCO, but this result only holds when d ≤ nglyph[epsilon1] n pub . As such, their result is orthogonal to our result in the pure PA-DP setting, which operates in the regime d ≥ nglyph[epsilon1] . In both cases, our proof technique is fundamentally different than theirs. 3 Tangentially, [BKS22] showed a small amount of public data is useful in pure-DP mean estimation when the range parameters on the data are unknown.

An important setting where public data is shown to be useful is PAC learning . Non-privately, it is known that the finiteness of VC dimension characterizes learnability [VC71, BEHW89]. However, under DP, it is impossible to PAC learn even the class of thresholds , which has VC dimension of one [BNS13]. The works of [BNS13, BTGT18, ABM19] showed that given access to a small unlabelled public data, it is possible to go beyond this limitation and privately learn VC classes, essentially by reducing a hypothesis class with finite VC dimension to a finite hypothesis class

R

(

M

(

)

2

)

1

/

3

)

˜

j

j

A number of works have studied the impact of public data in applied settings as well. A common technique is to use public data to reduce the problem dimension in some way [ZWB21, YZCL21, PHYS24]. The work of [GHN + 23] identified a specific problem instance which supports the method of public pretraining commonly used in practice.

With regards to unlabelled public data, there are several existing works. Transfer learning is a common approach in this setting. Besides the benefits in PAC learning, this setting also has applications in deep learning, where (empirically) unlabeled public data has been used to obtain performance improvements [PAE + 17, PSM + 18]. Unlabeled public data has also yielded impressive results used for pre-training large language models [LTLH22, YNB + 22b]. We also remark that, in practice, it is reasonable to expect the private and public datasets to come from slightly different distributions. Accounting for this distribution shift has also been the study of several recent works [BKS22, BDBC + 23]. However, in this work we focus on first characterizing the more fundamental problem where the public and private datasets are drawn i.i.d. from the same distribution.

2 Preliminaries

Here, we describe the concepts and assumptions used in the rest of this paper. In this work, ‖ · ‖ always denotes the glyph[lscript] 2 norm unless stated otherwise.

Public-Data Assisted Differential Privacy. We first present the traditional notion of differential privacy (DP). Let n, d ∈ N and X be some data domain. When no public data is present, we say that an algorithm A satisfies ( glyph[epsilon1], δ ) -differential privacy (DP) if for all datasets S and S ′ differing in one data point and all events E in the range of A , P [ A ( S ) ∈ E ] ≤ e glyph[epsilon1] P [ A ( S ′ ) ∈ E ] + δ [DMNS06].

In our work, we denote the number of public samples in the dataset, S = ( S pub , S priv ) ∈ X n , as n pub and the number of private samples as n priv, such that n = n pub + n priv. In keeping with previous work [BNS13, BCM + 20], we define public data assisted differentially private algorithms in the following way 4 .

Definition 1 (PA-DP) . An algorithm A is ( glyph[epsilon1], δ ) public-data assisted differentially private (PADP) algorithm with public sample size n pub and private sample size n priv if for any public dataset S pub ∈ X n pub , and any pair of private datasets S priv , S ′ priv ∈ X n priv differing in at most one entry, it holds for any event E that P [ A ( S pub , S priv ) ∈ E ] ≤ e glyph[epsilon1] P [ A ( S pub , S ′ priv ) ∈ E ] + δ . When δ = 0 , we refer to this notion as pure PA-DP , denoted as glyph[epsilon1] -PA-DP.

Stochastic Convex Optimization Let D be a distribution supported on X . Given some constraint set W ⊆ R d of diameter at most D , and a G Lipschitz convex loss glyph[lscript] : W × X → R , we are interested in minimizing the population loss, L ( w ; D ) = E x ∼D [ glyph[lscript] ( w ; x )] . Denote the minimizer as w ∗ = min w ∈W { L ( w ; D ) } . We evaluate the quality of the approximate solution, w , via the excess risk, L ( w ; D ) -L ( w ∗ ; D ) . Specifically, we are interested in PA-DP algorithms which minimizes this quantity when given S pub , S priv i.i.d. ∼ D . For a datset S we also define the empirical loss L ( w ; S ) = 1 ∑ glyph[lscript] ( w ; x ) .

̂ | S | x ∈ S

Supervised Learning and Generalized Linear Models (GLMs) In the supervised learning setting, in addition to the feature space X , we define the label space Y . We here let D be a joint probability distribution over X × Y and D X and D Y denote the respective marginal distributions. Let H ⊆ R X be a hypothesis class of real-valued predictors, and let fat α ( H ) denote its fat shattering dimension at scale α . Consider the loss function glyph[lscript] : H×X ×Y → R , such that glyph[lscript] ( h ; x, y ) = φ y ( h ( x )) for some function φ y . We assume that the map φ y : R → R is G -Lipschitz for all y ∈ Y and is B -bounded. Further, we assume that sup x ∈X | h ( x ) | ≤ R and define sup x ∈X ‖ x ‖ = ‖X‖ .

GLMs are a special case of supervised learning setting where the hypothesis class is that of linear predictors, H = W ⊆ R d , over X ⊆ R d , and h ( x ) = w glyph[latticetop] x . We refer to the public dataset of unlabeled feature vectors as X pub.

Covering numbers, fat-shattering and Rademacher Complexity Given X = ( x 1 , x 2 , · · · , x m ) the glyph[lscript] p distance between two hypothesis h 1 , h 2 ∈ H with respect to the empirical measure over X , is defined as, ‖ h 1 -h 2 ‖ p,X = ( 1 m ∑ x ∈ X | h 1 ( x ) -h 2 ( x ) | p ) 1 /p . Similarly, the distance with respect

to the population, is given by ‖ h 1 -h 2 ‖ p, D X = ( E x ∼D X | h 1 ( x ) -h 2 ( x ) | p ) 1 /p . The covering number of H at scale α > 0 and given dataset X , denoted as N p ( H , α, X ) is the size of the minimal set of hypothesis, ˜ H , such that for any h ∈ H there exists ˜ h with ‖ h -˜ h ‖ p,X ≤ α . We define N p ( H , α, m ) = sup X : | X | = m N p ( H , α, X ) , the covering number with respect to all datasets of size m . We define fat-shattering dimension below.

Definition 2. [BLW94] Let H ⊆ R X and α > 0 . We say that H α -shatters X = { x 1 , x 2 , . . . , x m } if sup r ∈ R m min y ∈{-1 , 1 } m sup h ∈H min i ∈ [ m ] y i ( h ( x i ) -r i ) ≥ α . The fat-shattering dimension, fat α ( H ) , is the size of the largest α -shattered set.

We define R m ( H ) , the worst-case Rademacher complexity of H with respect to m data points, as R m ( H ) = sup X : | X | = m E σ i sup h ∈H 1 m ∑ m i =1 σ i h ( x i ) . An important example is that of normbounded linear predictors H = { w : x ↦→〈 w,x 〉 : ‖ w ‖ ≤ D } over X = { x : ‖ x ‖ ≤ ‖X‖} . Herein, fat α ( H ) = Θ ( D 2 ‖X‖ 2 α 2 ) and R m ( H ) = Θ ( D ‖X‖ √ m ) [KST08, SST10].

3 Private Stochastic Convex Optimization with Labeled Public Data

In this section, we present our lower bounds for private stochastic convex optimization with public data. When interpreting the following results, it is helpful to note that in the nontrivial regime, n pub = Θ( n ) and n pub = o ( n ) , although our results hold regardless. Further, recall that an upper ( √ } )

bound for this problem of O R min { 1 √ n pub , 1 √ n + d log(1 /δ ) nglyph[epsilon1] can be obtained by simply either applying an optimal SCO algorithm to only the public data (and discarding the private data) or applying an optimal DP-SCO algorithm and treating the entire dataset as private [BFTGT19]. As we will see, this strategy is essentially optimal.

3.1 Lower Bound for Stochastic Convex Optimization

We start by stating our lower bound for public-data assisted differentially private SCO.

Theorem 1. Let δ ≤ 1 16 nd , glyph[epsilon1] ≤ 1 , and d be larger than some universal constant. For any ( glyph[epsilon1], δ ) -PA-DP algorithm, there exists a distribution D , and a G -Lipschitz loss such that E [ L ( A ( S pub , S priv ); D ) -min w : ‖ w ‖≤ D { L ( w ; D ) } ] = Ω( GD · Ψ( n pub , n, d, glyph[epsilon1], δ )) , where for some universal constant c ,

The function Ψ is defined to avoid repetitive notation in the rest of this section. Barring the mild restriction on n pub, even though the √ log (1 /δ ) term is only obtained when d ≥ nglyph[epsilon1] , the 'aggregate' lower bound is tight for all d / ∈ [ nglyph[epsilon1] 2 log(1 /δ ) , nglyph[epsilon1] ] since when d ≤ nglyph[epsilon1] 2 log(1 /δ ) the non-private 1 √ n lower bound dominates. It is also pertinent to our results in Section 4 that the problem construction used to achieve this lower bound is a convex GLM, and as a result this lower bound holds even for GLMs.

Finally, similar statements can be made about strongly convex optimization. We again provide just one such statement here.

Theorem 2. Let δ ≤ 1 16 nd , glyph[epsilon1] ≤ 1 . For any ( glyph[epsilon1], δ ) -PA-DP algorithm there exists a distribution D , λ -strongly convex and G -Lipschitz loss such that

The crux of the proofs for both the above results lies in establishing new mean estimation lower bounds for PA-DP mean estimation, which we give in Appendix B.1. These mean estimation lower bound use a novel application of a construction known as fingerprinting codes. In particular, the introduction of public data introduces significant challenges in the traditional analysis of fingerprinting codes. As these challenges are more technical in nature, we defer their discussion to Appendix B.2.

After establishing the mean estimation lower bounds, we can adapt the reductions first used in [BST14] that show mean estimation lower bounds can be used to provide lower bounds for risk minimization without public data. Full proofs for the above claims, and in particular details for the above reductions, are found in Appendix B.3.

Lower Bound for Pure DP Case. While not the primary focus of this work, the previous lower bound directly leads to a lower bound for pure PA-DP SCO. Since any glyph[epsilon1] -DP algorithm is ( glyph[epsilon1], δ ) -DP for any δ > 0 , we can use the above theorem to obtain a non-trivial lower bound for the pure DP case by setting δ small. Specifically, by setting δ such that log (1 /δ ) = nglyph[epsilon1] 120 2 n pub , one immediately obtains a lower bound of Ω ( min { 1 √ n pub , √ d √ n pub · nglyph[epsilon1] } ) for d large enough. Simplifying this expression yields the following.

Corollary 1. Let d ≥ cnglyph[epsilon1] for a constant c , and A be an glyph[epsilon1] -PA-DP algorithm. There exist a distribution D and a G -Lipschitz loss such that E [ L ( A ( S pub , S priv ); D ) -min w : ‖ w ‖≤ D { L ( w ; D ) } ] = Ω ( GD √ n pub ) .

The known O ( GD min { 1 √ n pub , 1 √ n + d nglyph[epsilon1] }) upper bound for this problem shows that this bound is tight (in the regime in which it holds). Essentially, this bound states that when d ≥ nglyph[epsilon1] , the public dataset is not useful (at least asymptotically). Previously [LLHR23, Theorem 31] established that when d ≤ nglyph[epsilon1]/n pub, a tight lower bound of Ω ( GD ( d nglyph[epsilon1] + 1 √ n ) ) holds, effectively showing that in this regime the public dataset is not useful 5 . We leave the remaining regime where d ∈ ( nglyph[epsilon1] n pub , nglyph[epsilon1] ) as an interesting open problem for future work. Finally, we note that similar statements can be made about strongly convex losses using Theorem 2.

4 Private Supervised Learning with Unlabeled Public Data

In this section, we consider supervised learning with real-valued predictors given labeled private data and unlabeled public data. Our results show that, in this setting, it is possible to go beyond the limitations established in the prior section.

4.1 Efficient PA-DP learning of Convex Generalized Linear Models

We start with learning linear predictors with convex loses a.k.a. convex generalized learning models. We propose Algorithm 1, which uses the public unlabeled data to perform dimensionality reduction of the private labeled feature vectors. In the following, we use span to denote the span of a set of vectors and dim to denote the dimension of a subspace. The dataset of public unlabeled feature vectors is denote as X pub. Our algorithm projects the private feature vectors onto the subspace spanning W ∩ span ( X pub ) to get dim(span( X pub) ∩W )-dimensional representation of the private feature vectors. It then reparametrizes the loss function so that its domain is dim(span( S pub) ∩W )-dimensional and applies a private subroutine in the lower dimensional space. The output of the subroutine is then embedded back in R d . Algorithms similar to Algorithm 1 have appeared in the literature (e.g. [PHYS24]). We emphasize that our key contribution is the formal analysis of this technique and the fact that we provide tight upper and lower bounds while simultaneously avoiding many of the strong assumptions seen in previous work, such as large margin assumptions.

Algorithm 1 Efficient PA-DP learning of GLMs with unlabeled public data

Input: Private labeled dataset S priv, public unlabeled dataset X pub, privacy parameters glyph[epsilon1],δ > 0 .

1: Let U ∈ R d × dim ( W∩ span ( X pub )) denote the orthogonal projection onto span ( X pub ) ∩ W .

3: Apply ( glyph[epsilon1], δ ) -DP subroutine, ˜ A , on loss function w ↦→ φ y ( 〈 w,x 〉 ) with dataset ˜ S priv over the constraint set ˜ W , to get ˜ w ∈ R dim ( W∩ span( S pub )) . w = U ˜ w .

2: Define ˜ S priv = { ( U glyph[latticetop] x i , y i ) } n priv i =1 and let ˜ W = { U glyph[latticetop] w : w ∈ W } .

Output: ̂

Our main result for convex Lipschitz losses is the following.

Theorem 3. Let glyph[epsilon1] > 0 , δ > 0 and glyph[epsilon1] ≤ log (1 /δ ) . For a G -Lipschitz, B -bounded convex loss function, Algorithm 1 satisfies ( glyph[epsilon1], δ ) -PA-DP. If the private subroutine ˜ A guarantees the following, with probability at least 1 -β ,

then with n pub = ˜ O ( n priv glyph[epsilon1] (log(2 /β )+log(1 /δ )) 1 / 2 ) , with probability at least 1 -β , L ( ̂ w ; D ) -L ( w ∗ ; D ) is

We note that DP algorithms such as projected noisy SGD [BST14] and the regularized exponential mechanism [GLL22], both of which can be implemented efficiently, are can be used to achieve (1), since the projected problem is at most n pub dimensional.

The above result shows that in the usual regime of glyph[epsilon1] = Θ(1) , there is no price of privacy, thereby obtaining the non-private rate of O ( 1 √ n priv ) . We contrast this with the rate of O ( 1 √ n priv + √ d n priv glyph[epsilon1] ) , achievable without public data. Our result is better when d ≥ n priv glyph[epsilon1] , which is the interesting regime since herein the private error dominates the non-private error. Further, our lower bound (Theorem 4 below) shows that this is the non-trivial regime (for any glyph[epsilon1] = O (1)) , since otherwise, even with unlimited public data, the optimal rate is achieved without using any of it. We also note that the above rate is achievable without public data, but in the unconstrained setting where the output ̂ w can have very large norm and so may lie outside W [ABG + 22].

The proof of the result primarily follows from the more general result with fat-shattering hypothesis classes (Theorem 7). We provide the key ideas as well as some details pertaining to linear predictors in Section 4.2 after Theorem 7. The full proof of this result is deferred to Appendix C.

Lower Bounds. The above rate as well as the number of public samples used are nearly-optimal. The first claim is due to the following result, which gives a lower bound on excess risk of DP algorithms under full knowledge of the marginal distribution, for Lipschitz GLMs. As unlabeled public data can only reveal information about the marginal distribution, this shows that further unlabeled public samples cannot hope to improve the rate we give in Theorem 3.

Theorem 4. Let glyph[epsilon1] ≤ 1 , δ ≤ glyph[epsilon1] and A be an ( glyph[epsilon1], δ ) -DP algorithm. There exists a G -Lipschitz convex GLM loss function, and joint distribution D such that given a dataset S comprising n i.i.d. samples from D and full knowledge of the marginal distribution D X , we have the following: E A ,S [ L ( A ( S ); D ) -min w : ‖ w ‖≤ D L ( w ; D ) ] = Ω ( GD ‖X‖ ( 1 √ n +min { 1 √ nglyph[epsilon1] , √ d nglyph[epsilon1] })) .

We note that the bound with d n priv glyph[epsilon1] can be achieved without using any public data via standard results [BFTGT19, ABG + 22]. This result is largely a corollary of [ABG + 22, Theorem 6]. We provide full details in Appendix C.3.1.

To establish optimality of public sample complexity, we give the following lower bound which shows that ˜ Ω( n priv glyph[epsilon1] ) samples are necessary to achieve the above rate. See Appendix C.3.2 for proof. Theorem 5. Let n priv , n pub , d ∈ N , glyph[epsilon1] ≤ 1 , δ < 1 16 dn and d = ω ( n priv glyph[epsilon1] ) . If there exists an ( glyph[epsilon1], δ ) -PA-DP algorithm A , which, for any G -Lispschitz convex GLM, achieves excess risk E [ L ( A ( X pub , S priv ); D ) -min w : ‖ w ‖≤ D L ( w ; D ) ] = O ( GD ‖X‖ ( 1 √ n priv + √ log(1 /δ ) √ n priv glyph[epsilon1] )) , for S priv ∼ D n priv and X pub ∼ D n pub X , then n pub = Ω( n priv glyph[epsilon1] log(1 /δ ) ) .

Optimistic rates. We now consider additional assumptions that the loss function is non-negative and H -smooth, such as in the case of linear regression where φ y ( a ) = ( a -y ) 2 . This is a well-studied setting [SST10] especially since it allows for obtaining optimistic rates: those that interpolate between a slow worst-case rate and a faster rate under (near) realizability or interpolation conditions. The main result is the following.

Theorem 6. Let glyph[epsilon1] > 0 , δ > 0 and glyph[epsilon1] ≤ log (1 /δ ) . For a G -Lipschitz, B -bounded non-negative H -smooth loss function, Algorithm 1 satisfies ( glyph[epsilon1], δ ) -PA-DP. If the private subroutine ˜ A guarantees Equation (1) with probability at least 1 -β , then with n pub = ˜ O ( ( HD ‖X‖ ) 2 / 3 ( n priv glyph[epsilon1] ) 2 / 3 G 2 / 3 (log(1 /δ )) 1 / 3 + √ Hn priv glyph[epsilon1] √ ̂ L ( ̂ w ∗ ; S priv ) G √ log(1 /δ )) ) , with probability at least 1 -β ,

where ̂ w ∗ is the minimizer of ̂ L w.r.t S priv and ˜ O hides poly (log (1 /δ ) , log (1 /β )) terms.

A similar result as above can be obtained with ̂ L ( ̂ w ∗ ; S priv ) replaced by L ( w ∗ ; D ) above - see Theorem 14 for the full theorem statement. This rate, in the worst-case, is essentially the same as that of Theorem 3, which is ˜ O ( 1 √ n priv + 1 √ n priv glyph[epsilon1] ) . However, optimistically, when L ( w ∗ ; D ) or ̂ L ( ̂ w ∗ ; S priv ) is small, we get a faster rate of ˜ O ( 1 n priv + 1 ( n priv glyph[epsilon1] ) 2 / 3 ) . We note that this is seemingly weaker than what is known in the unconstrained setting, where [ABG + 22] obtained a worst-case rate of ˜ O ( 1 √ n priv + 1 ( n priv glyph[epsilon1] ) 2 / 3 ) . We show that we can recover this faster rate under an extra assumption that the global minimizer of the risk, lies in the constraint set W - note that this is trivially true in the unconstrained setup; see Theorem 15 for the statement.

We note that projected noisy SGD [BST14] and the regularized exponential mechanism [GLL22], both of which can be implemented efficiently, are possible choices for the private sub-routine ˜ A that realize the above theorem statements.

4.2 PA-DP Supervised learning of Fat-Shattering Classes

In this section, we consider a general supervised learning setting with fat-shattering hypothesis classes and potentially non-convex losses, with unlabeled public data. Our proposed algorithm is similar to that of [ABM19], which uses the pubic unlabeled data to construct a small finite, yet representative, subset of the hypothesis class. Our construction uses a cover of the hypothesis class with respect to the glyph[lscript] 2 distance of predictions on the public data points. We then use the exponential mechanism to privately select a hypothesis using the empirical loss on private data as the score function.

We note that we operate under the pure DP setting (as opposed to approximate DP). Our techniques are based on selection which do not exhibit improved guarantees under approximate DP. Further, we note that, without public data, with non-convex losses, there is no separation of optimal rates between pure and approximate DP [GTU23].

Algorithm 2 Supervised private learning with public unlabeled data

Input: Datasets X pub and S priv, privacy parameter glyph[epsilon1] > 0 , scale of cover α > 0 , γ > 0 .

1: Construct ˜ H , a minimal α -cover of H , with respect to the following metric

2: Return ̂ h sampled with probability p ( h ) ∝ exp ( -γ ̂ L ( h ; S priv ) ) over h ∈ ˜ H

Our main result for the Lispchitz setting is the following.

Theorem 7. Algorithm 2 with γ = 2 min( B,GR ) n priv glyph[epsilon1] satisfies glyph[epsilon1] -PA-DP. For any α > 0 and n pub = O ( max ( R 2 log(2 /β ) α 2 , min { m : log 3 ( m ) R 2 m ( H ) ≤ α 2 } )) < ∞ , with probability at least 1 -β , we have L ( ̂ h ; D ) -min h ∈H L ( h ; D ) is at most

where c is an absolute constant.

Our result shows that the model of PA-DP with unlabeled public data allows for obtaining nontrivial rates for supervised learning with any fat-shattering class, as is the case in the non-private setting. Further, in many standard settings, such as that of (Euclidean) GLMs, the Rademacher complexity is R m ( H ) = O ( 1 √ m ) which implies that fat α ( H ) = O ( 1 α 2 ) (see Theorem 9). In those cases, our guarantee simplifies to essentially yield a rate of O ( R n priv ( H ) + 1 ( n priv glyph[epsilon1] ) 1 / 3 + 1 √ n priv ) - see Corollary 4 for the exact statement for GLMs.

Proof Idea. We briefly discuss some main ideas in the proof. The key is to show that if ˜ H is a cover of H with respect to the empirical distance on public feature vectors, ‖·‖ 2 ,X pub , then with enough public feature vectors, it is also a cover with respect to the population distance ‖·‖ 2 , D X . This is captured in the following result.

Lemma 1. Let ˜ H be a τ -cover of H with respect to ‖·‖ 2 ,X pub . For n pub = O ( max ( R 2 log(1 /β ) α 2 , min { m : log 3 ( m ) R 2 m ( H ) ≤ α 2 } ) ) < ∞ , for every h ∈ H , with probability at least 1 -β , there exists ˜ h ∈ ˜ H such that ‖ h -˜ h ‖ 2 , D X ≤ α + τ .

This result allows us to appropriately approximate a hypothesis class with enough public unlabeled points. This approximation roughly translates to the same additive error in the final bound while concurrently allowing for the use of the smaller finite hypothesis class ˜ H of size | ˜ H| = ˜ O ( fat τ ( H )) .

For linear predictors with convex losses, as in Theorem 3, we show that the span ( X pub ) ∩ W is a valid 0 -cover w.r.t. ‖ · ‖ 2 ,X pub . However, the cover being continuous and convex allows application of convex optimization techniques (as opposed to selection, as above), thereby obtaining stronger results with efficient procedures. The above procedure yields optimistic rates for non-negative and smooth losses; see Theorem 17 for details.

4.2.1 Application: Neural Networks

In this section, we instantiate our general result to give a guarantee for learning feed-forward neural networks in the PA-DP setting. We use the result of [GRS18] but note that other results which give bounds on the Rademacher complexity of neural networks, such as [BFT17, Sel23] can also be used.

We consider a depth M feed-forward neural network which implements the function x ↦→ W M ( σ ( W M -1 . . . σ ( W 1 x )) . . . ) . Here, W 1 , W 2 , . . . , W M are the weight matrices and σ is a (nonlinear) activation function. We consider 1 -Lipschtiz positive-homogeneous activation such as the ReLU function, σ ( z ) = max(0 , z ) , applied coordinate-wise. Our main result is the following.

Corollary 2. Let ( R j ) M j =1 be a sequence of scalars and M ∈ N . In the setting of Theorem 7 with X = { x ∈ R d : ‖ x ‖ ≤ ‖X‖ } and H being the class of depth M feed-forward neural networks, with 1 -Lipschitz positive-homogenous activation, and weight matrices, bounded as ‖ W j ‖ F ≤ R j , with n pub = ˜ O ( ( ‖X‖ ( ∏ M j =1 R j )) 2 / 3 ( n priv glyph[epsilon1] ) 2 / 3 M 1 / 3 log (2 /β ) ) , with probability at least 1 -β , L ( ̂ h ; D ) -min h ∈H L ( h ; D ) is at most

We note that the above result has a polynomial dependence on the depth M , which is a consequence of the (non-private) Rademacher complexity of [GRS18]. It is also possible to get fully size-independent bounds by utilizing such existing results, however they require more stringent norm bounds on the weight matrices [Sel23]. Further, a similar result follows for non-negative smooth losses from [SST10], but we omit this extension for brevity.

4.2.2 Application: Non-Euclidean GLMs

In the non-Euclidean GLM setting, we consider ( X , ‖·‖ ) as a d dimensional (where d ∈ N ∪{∞} ) Banach space, and ( W , ‖·‖ ∗ ) is its dual space. The feature vectors x are bounded as X = { x ∈ X : ‖ x ‖ ≤ ‖X‖} and W ⊆ { w ∈ W : ∆( w ) ≤ D r } where ∆ is a r -uniformly convex function 6 with respect to ‖·‖ ∗ . A canonical example is the ( glyph[lscript] p , glyph[lscript] q ) -setup [KST08, FGV17], wherein the functions ∆( w ) = log( d ) 2 ‖ w ‖ 2 1+(1 / log( d )) , ∆( w ) = 1 2( p -1) ‖ w ‖ 2 p and ∆( w ) = 2 p -2 p ‖ w ‖ p p are 2 , 2 and p -uniformly convex with respect to ‖·‖ p for p = 1 , 1 < p ≤ 2 and p ≥ 2 respectively.

The GLM loss function glyph[lscript] ( w ; x, y ) = φ y ( 〈 w,x 〉 ) where 〈· , ·〉 : X × W → R is a duality pairing. In this case, the Rademacher complexity of linear functions, is bounded as O ( D ‖X‖ m 1 /r ) , where s is the conjugate of r i.e. 1 r + 1 s = 1 (see, e.g. [FGV17]). We obtain the following result by instantiating Theorem 7 with the Rademacher complexity and fat-shattering dimension of non-Euclidean GLMs. Corollary 3. In the setting of Theorem 7, together with X = { x ∈ R d : ‖ x ‖ ≤ ‖X‖ } and H = { x ↦→〈 w,x 〉 , x ∈ X , ∆( w ) ≤ D r } . Given n pub = ˜ O ( ( n priv glyph[epsilon1] ) r/ ( r +1) log (2 /β ) ) , with probability at least 1 -β , L ( ̂ w ; D ) -min w ∈W L ( w ; D ) is at most

The above yields guarantees for the special case of ( glyph[lscript] p , glyph[lscript] q ) -setup with r = max { 2 , p } . We remind that in the (constrained) convex Euclidean GLM setting, our dimension-independent rate in Theorem 3, with public unlabeled data recover the rates which were known to be achievable in the unconstrained setting. Further the above rate for p = 1 case can be used to obtain guarantees for the polyhedral setting with ‖ w ‖ 1 ≤ D constraint, resulting in a O ( √ log( d ) n priv + ( log( d ) n priv glyph[epsilon1] ) 1 / 3 ) rate. We note that √

glyph[negationslash]

[BGM21] showed a rate of ˜ O (√ log( d ) n + log( d ) √ nglyph[epsilon1] ) for this setting, with convex losses without public data. Importantly, for the other cases, i.e. p > 1 , p = 2 , there are no such (nearly) dimensionindependent analogs of our result without public data, as of yet.

Acknowledgments and Disclosure of Funding

R. Arora's and E. Ullah's research was supported, in part, by NSF BIGDATA award IIS-1838139 and NSF CAREER award IIS-1943251. R. Bassily's and M. Menart's research was supported by NSF CAREER Award 2144532 and, in part, by NSF Award 2112471. C. Guzmán's research was partially supported by INRIA Associate Teams project, ANID FONDECYT 1210362 grant, ANID Anillo ACT210005 grant, and National Center for Artificial Intelligence CENIA FB210017, Basal ANID.

References

[ABDCBH97] Noga Alon, Shai Ben-David, Nicolo Cesa-Bianchi, and David Haussler. Scalesensitive dimensions, uniform convergence, and learnability. Journal of the ACM (JACM) , 44(4):615-631, 1997.

[ABG + 22] Raman Arora, Raef Bassily, Cristóbal Guzmán, Michael Menart, and Enayat Ullah. Differentially private generalized linear models revisited. Advances in Neural Information Processing Systems , 35:22505-22517, 2022.

[ABM19] Noga Alon, Raef Bassily, and Shay Moran. Limits of private learning with access to public data. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

[AGM + 22] Ehsan Amid, Arun Ganesh, Rajiv Mathews, Swaroop Ramaswamy, Shuang Song, Thomas Steinke, Thomas Steinke, Vinith M Suriyakumar, Om Thakkar, and Abhradeep Thakurta. Public data-assisted mirror descent for private model training. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 517-535. PMLR, 17-23 Jul 2022.

[BCM + 20] Raef Bassily, Albert Cheu, Shay Moran, Aleksandar Nikolov, Jonathan Ullman, and Steven Wu. Private query release assisted by public data. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 695-703. PMLR, 13-18 Jul 2020.

[BDBC + 23] Shai Ben-David, Alex Bie, Clement Louis Canonne, Gautam Kamath, and Vikrant Singhal. Private distribution learning with public data: The view from sample compression. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

[BEHW89] Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K Warmuth. Learnability and the vapnik-chervonenkis dimension. Journal of the ACM (JACM) , 36(4):929-965, 1989.

[BFT17] Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems , 30, 2017.

[BFTGT19] Raef Bassily, Vitaly Feldman, Kunal Talwar, and Abhradeep Guha Thakurta. Private stochastic convex optimization with optimal rates. Advances in Neural Information Processing Systems , 32, 2019.

[BGM21] Raef Bassily, Cristóbal A Guzmán, and Michael Menart. Differentially private stochastic optimization: New results in convex and non-convex settings. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

[BKS22] Alex Bie, Gautam Kamath, and Vikrant Singhal. Private estimation with public data. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

[BLW94] Peter L Bartlett, Philip M Long, and Robert C Williamson. Fat-shattering and the learnability of real-valued functions. In Proceedings of the seventh annual conference on Computational learning theory , pages 299-310, 1994.

[BNS13] Amos Beimel, Kobbi Nissim, and Uri Stemmer. Private learning and sanitization: Pure vs. approximate differential privacy. In Prasad Raghavendra, Sofya Raskhodnikova, Klaus Jansen, and José D. P. Rolim, editors, Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques , pages 363-378, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg.

[BST14] Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th Annual Symposium on Foundations of Computer Science , pages 464-473. IEEE, 2014.

[BTGT18] Raef Bassily, Om Thakkar, and Abhradeep Guha Thakurta. Model-agnostic private learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

[BUV14] Mark Bun, Jonathan Ullman, and Salil Vadhan. Fingerprinting codes and the price of approximate differential privacy. In Proceedings of the Forty-Sixth Annual ACM Symposium on Theory of Computing , STOC '14, page 1-10, New York, NY, USA, 2014. Association for Computing Machinery.

[BWZK22] Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis. Differentially private bias-term only fine-tuning of foundation models. In NeurIPS 2022 Workshop on Trustworthy and Socially Responsible Machine Learning (TSRML) , 2022.

[CH11] Kamalika Chaudhuri and Daniel Hsu. Sample complexity bounds for differentially private learning. In Sham M. Kakade and Ulrike von Luxburg, editors, Proceedings of the 24th Annual Conference on Learning Theory , volume 19 of Proceedings of Machine Learning Research , pages 155-186, Budapest, Hungary, 09-11 Jun 2011. PMLR.

[CWZ21] T. Tony Cai, Yichen Wang, and Linjun Zhang. The cost of privacy: Optimal rates of convergence for parameter estimation with differential privacy. The Annals of Statistics , 49(5):2825 - 2850, 2021.

[DMNS06] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference , pages 265-284. Springer, 2006.

[DSS + 15] Cynthia Dwork, Adam Smith, Thomas Steinke, Jonathan Ullman, and Salil Vadhan. Robust traceability from trace amounts. In 2015 IEEE 56th Annual Symposium on Foundations of Computer Science , pages 650-669, 2015.

[Duc23] John Duchi. Lecture notes on statistics and information theory, May 2023.

[FGV17] Vitaly Feldman, Cristobal Guzman, and Santosh Vempala. Statistical query algorithms for mean vector estimation and stochastic convex optimization. In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 1265-1277. SIAM, 2017.

[FS17] Vitaly Feldman and Thomas Steinke. Generalization for adaptively-chosen estimators via stable median. In Satyen Kale and Ohad Shamir, editors, Proceedings of the 2017 Conference on Learning Theory , volume 65 of Proceedings of Machine Learning Research , pages 728-757. PMLR, 07-10 Jul 2017.

[GHN + 23] Arun Ganesh, Mahdi Haghifam, Milad Nasr, Sewoong Oh, Thomas Steinke, Om Thakkar, Abhradeep Thakurta, and Lun Wang. Why is public pretraining necessary for private model training? In International Conference on Machine Learning , 2023.

[GLL22] Sivakanth Gopi, Yin Tat Lee, and Daogao Liu. Private convex optimization via exponential mechanism. In Conference on Learning Theory , pages 1948-1989. PMLR, 2022.

[GRS18] Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In Conference On Learning Theory , pages 297-299. PMLR, 2018.

[GTU23] Arun Ganesh, Abhradeep Thakurta, and Jalaj Upadhyay. Universality of langevin diffusion for private optimization, with applications to sampling from rashomon sets. In The Thirty Sixth Annual Conference on Learning Theory , pages 1730-1773. PMLR, 2023.

[KDRT21] Peter Kairouz, Monica Ribero Diaz, Keith Rush, and Abhradeep Thakurta. (nearly) dimension independent private erm with adagrad rates via publicly estimated subspaces. In Mikhail Belkin and Samory Kpotufe, editors, Proceedings of Thirty Fourth Conference on Learning Theory , volume 134 of Proceedings of Machine Learning Research , pages 2717-2746. PMLR, 15-19 Aug 2021.

[KST08] Sham M Kakade, Karthik Sridharan, and Ambuj Tewari. On the complexity of linear prediction: Risk bounds, margin bounds, and regularization. Advances in neural information processing systems , 21, 2008.

[KU20] Gautam Kamath and Jonathan Ullman. A primer on private statistics. arXiv preprint arXiv:2005.00010 , 2020.

[LLHR23] Andrew Lowy, Zeman Li, Tianjian Huang, and Meisam Razaviyayn. Optimal differentially private learning with public data. arXiv preprint arXiv:2306.15056v1 , 2023.

[LLHR24] Andrew Lowy, Zeman Li, Tianjian Huang, and Meisam Razaviyayn. Optimal differentially private model training with public data. arXiv preprint arXiv:2306.15056v2 , 2024.

[LTLH22] Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In International Conference on Learning Representations , 2022.

[LW19] Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation, 2019.

[MT07] Frank McSherry and Kunal Talwar. Mechanism design via differential privacy. In 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07) , pages 94-103. IEEE, 2007.

[NMT + 23] Milad Nasr, Saeed Mahloujifar, Xinyu Tang, Prateek Mittal, and Amir Houmansadr. Effectively using public data in privacy preserving machine learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 25718-25732. PMLR, 23-29 Jul 2023.

[PAE + 17] Nicolas Papernot, Martin Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi-supervised knowledge transfer for deep learning from private training data. In International Conference on Learning Representations , 2017.

[PHYS24] Francesco Pinto, Yaxi Hu, Fanny Yang, and Amartya Sanyal. PILLAR: How to make semi-private learning more effective. In 2nd IEEE Conference on Secure and Trustworthy Machine Learning , 2024.

[PSM + 18] Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Úlfar Erlingsson. Scalable private learning with PATE. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018.

[PVX + 23] Natalia Ponomareva, Sergei Vassilvitskii, Zheng Xu, Brendan McMahan, Alexey Kurakin, and Chiyaun Zhang. How to dp-fy ml: A practical tutorial to machine learning with differential privacy. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD '23, page 5823-5824, New York, NY, USA, 2023. Association for Computing Machinery.

[RS12] Alexander Rakhlin and Karthik Sridharan. Statistical learning theory and sequential prediction. Lecture Notes in University of Pennsyvania , 44, 2012.

[SCZ + 20] Dianbo Sui, Yubo Chen, Jun Zhao, Yantao Jia, Yuantao Xie, and Weijian Sun. FedED: Federated learning via ensemble distillation for medical relation extraction. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 2118-2128, Online, November 2020. Association for Computational Linguistics.

[Sel23] Mark Sellke. On size-independent sample complexity of relu networks. arXiv preprint arXiv:2306.01992 , 2023.

[SSBD14] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

[SST10] Nathan Srebro, Karthik Sridharan, and Ambuj Tewari. Optimistic rates for learning with a smooth loss. arXiv preprint arXiv:1009.3896 , 2010.

[SSTT21] Shuang Song, Thomas Steinke, Om Thakkar, and Abhradeep Thakurta. Evading the curse of dimensionality in unconstrained private glms. In International Conference on Artificial Intelligence and Statistics , pages 2638-2646. PMLR, 2021.

[SU15] Thomas Steinke and Jonathan Ullman. Between pure and approximate differential privacy. Journal of Privacy and Confidentiality , 7, 01 2015.

[VC71] N. Vapnik and A. Ya. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability & Its Applications , 16(2):264, 1971.

[Ver18] Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

[YNB + 22a] Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. Differentially private fine-tuning of language models. In International Conference on Learning Representations , 2022.

[YNB + 22b] Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. Differentially private fine-tuning of language models. In International Conference on Learning Representations , 2022.

[YZCL21] Da Yu, Huishuai Zhang, Wei Chen, and Tie-Yan Liu. Do not let privacy overbill utility: Gradient embedding perturbation for private learning. In International Conference on Learning Representations , 2021.

[ZWB21] Yingxue Zhou, Steven Wu, and Arindam Banerjee. Bypassing the ambient dimension: Private sgd with gradient subspace identification. In International Conference on Learning Representations , 2021.

A Additional Preliminaries

Theorem 8. [RS12, Theorem 12.7] For H ⊆ [ -R,R ] X , m ∈ N , p ≥ 1 , 0 < α ≤ R , we have that

.

Further, for any τ ∈ (0 , 1) ,

where c, c ′ , C and C ′ are absolute constants.

Theorem 9. [SST10, Lemma A.3] For any hypothesis class H , any sample size m and any α > R m ( H ) , we have that,

Theorem 10. [GRS18, Theorem 1] Let M ∈ N and ( R j ) M j =1 be a sequence of scalars. The Rademacher complexity of the class of depth M neural networks with 1 -Lipschitz, positivehomogeneous activation function, H , with weights ‖ W j ‖ F ≤ R j is bounded as,

Here ‖ · ‖ F denotes the Frobenius norm.

Lemma2. [DSS + 15, Implied by Lemmas 5 and 14] Let f : {± 1 } n ↦→ R and define g : [ -1 , 1] → R as g ( p ) = E S ∼D n p [ f ( S )] , where D p is as defined in Appendix B. Then for a, b ∈ R , b > a , and µ ∼ Unif ([ a, b ]) ,

Lemma 3. [FS17, Lemma A.1] Fix µ, glyph[epsilon1], δ, ∆ ∈ R . Let X and Y be random variables supported on [ µ -∆ , µ + ∆] . Suppose that X and Y are ( glyph[epsilon1], δ ) -indistinguishable, that is for any E ⊆ R , e -glyph[epsilon1] ( P [ X ∈ E ] -δ ) ≤ P [ Y ∈ E ] ≤ e glyph[epsilon1] P [ X ∈ E ] + δ. Then

B Missing proofs from Section 3

B.1 Lower Bounds for Mean Estimation

As stated previously, our SCO result follows primarily from new lower bound for PA-DP mean estimation. Here, we consider the setting where D is supported on the glyph[lscript] 2 ball of radius R > 0 . We define the mean of D as µ ( D ) . Our lower bound for PA-DP mean estimation follows much the same form as our SCO bound.

Theorem 11. Let δ ≤ 1 16 nd , glyph[epsilon1] ≤ 1 . For any ( glyph[epsilon1], δ ) -PA-DP algorithm, there exists a distribution D such that E [ ‖A ( S pub , S priv ) -µ ( D ) ‖ ] = Ω( R · Ψ( n pub , n, glyph[epsilon1], δ )) .

We present the full proof momentarily and provide a more detailed discussion on the challenges of establishing this lower bound in Appendix B.2. We highlight key ideas here. As with many other lower bounds in differential privacy, we leverage a construct known as fingerprinting codes [BUV14, DSS + 15]. A key aspect of our analysis is showing that fingerprinting distributions can be

used to recover the optimal non-private lower bound for mean estimation. This allows us to create a problem which is 'hard' both privately and non-privately. The analysis works by first showing that any sufficiently accurate algorithm must strongly correlate with the sampled datapoints. Next, we show upper bounds on how strongly the output of the algorithm correlates with the sampled dataset. The method for upper bounding this correlation varies depending on whether a given datapoint is considered public or private. Combining these upper and lower bounds on correlation yields the claimed result.

To obtain the √ log (1 /δ ) factor term in the lower bound, we use similar ideas to those in [SU15, CWZ21]. However, the introduction of public data leads to complications in prior methods. As such, we show that by analyzing the correlation of the coordinate wise clipping of the algorithms output, we are able to get bounds that appropriately scale with the accuracy.

Proof of Theorem 11 Before proceeding, we introduce the so-called fingerprinting distribution which will be the basis of our hard instance for mean estimation [BUV14, DSS + 15]. Towards this end, for any vector µ ∈ [ -1 , 1] d we define D µ as the product distribution where, for any j ∈ [ d ] , a sample has its j 'th coordinate as 1 with probability (1+ µ j ) / 2 and as -1 with probability (1 -µ j ) / 2 . As shorthand, we denote R √ d D µ as the distribution which samples a vector from D µ and then scales it by R √ d . For notational convenience, for a set E , we will also use Unif ( E ) to denote the uniform distribution over elements of the set.

The theorem follows from two theorems which have different restrictions on the problem parameters. In addition to the following two theorems, Theorem 11 incorporates the classic R √ n statistical lower bound that holds even non-privately. The first theorem we present holds for a larger range of parameters but does not achieve the dependence on log (1 /δ ) .

Theorem 12. Let glyph[epsilon1] > 0 , δ ≤ 1 16 n and A be an ( glyph[epsilon1], δ ) -PA-DP algorithm. For any setting of min { √ d n priv glyph[epsilon1] , 1 √ n pub } ≤ M ≤ 1 , if µ ∼ Unif ([ -M,M ] d ) and ( S pub , S priv ) ∼ R √ d D n µ it holds that

In application to Theorem 11, we use ( e glyph[epsilon1] -1) ≤ 2 glyph[epsilon1] whenever glyph[epsilon1] ≤ 1 . The second theorem requires d ≥ nglyph[epsilon1] but has the benefit of scaling with log (1 /δ ) .

Theorem 13. Let δ ≤ 1 3 dn , glyph[epsilon1] ≤ 1 , d ≥ 120 2 nglyph[epsilon1] , and n pub ≤ nglyph[epsilon1] 120 2 log ( 1 / [ √ ndδ ] ) , and A an ( glyph[epsilon1], δ ) -PADP algorithm. Then there exists M > 0 such for µ ∼ Unif ([ -M,M ] d ) and ( S priv , S pub ) ∼ R √ d D n µ it holds that

A crucial part of the analysis is leveraging the so called fingerprinting lemma, which roughly states that any accurate algorithm given a dataset sampled from D µ must strongly correlate with vectors in the dataset. Particularly pertinent to our analysis is achieving such a correlation even when the components of the mean µ are much smaller than 1 . Towards this end, we leverage the robust distribution framework of [DSS + 15] to achieve the following version of the fingerprinting lemma.

Lemma 4 (Fingerprinting Lemma) . Let M ∈ [0 , 1] and µ be sampled uniformly from [ -M,M ] d . Let A satisfy E S ∼D n µ [ ‖A ( S ) -µ ‖ ] ≤ α (for any µ ∈ [ -1 , 1] d ). Then one has

n

Proof. In the following we treat A as a deterministic function and bound E S,µ [ ∑ i =1 〈A ( S ) , x i -µ 〉 ] . This is sufficient to bound E A ,S,µ [ ∑ n i =1 〈A ( S ) , x i -µ 〉 ] for randomized A , since the analysis holds for

any function (i.e. the distribution does not depend on A ). Further, we start with the one dimensional case such that µ ∈ R . Define g ( µ ) = E S ∼D n µ [ A ( S )] . We start by applying results developed in +

[DSS 15],

Above, ( i ) comes from [DSS + 15, Lemma 5] and ( ii ) comes from [DSS + 15, Lemma 14], which we have collectively restated in Lemma 2. We now have

Above we use the fact that | µ | ≤ M and the definition of g .

We can now extend the above analysis to higher dimensions. For µ ∈ R d , the above holds for each µ j , j ∈ [ d ] . For convenience define ¯ M = ( M,...,M ) ∈ R d . Summing over d dimensions we have

This proves the claim.

We now turn towards proving Theorems 12 and 13. We start with the simpler proof of Theorem 12.

Proof of Theorem 12. For our proof we will use a dataset of vectors in {± 1 } d , and as such the glyph[lscript] 2 bound on the data is √ d . The final result will follow from rescaling by R √ d .

Let S = { x 1 , x 2 , . . . x n } = ( S pub , S priv ) ∼ D n µ be the concatenation of the public and private datasets. We also define α = E [ ‖A ( S pub , S priv ) -µ ‖ ] for notational convenience.

Define the following statistics,

where S ∼ i is the dataset formed by replacing i -th data point of S priv with x ′ i ∼ D µ . We have,

The lower bound proceeds by providing upper and lower bounds on the above sum. We first have

where the first inequality used Cauchy-Schwartz.

For the second term in Equation (2), we utilize differential privacy. Specifically, [FS17, Lemma A.1], restated in Lemma 3, gives that

Above we use that Var ( Z ′ i ) ≤ 4 α 2 since ‖ x i -µ ‖ ∞ ≤ 4 . Plugging the above two in Equation (2) yields,

We now use the fingerprinting lemma, Lemma 4, to lower bound the correlation. In this regard, note E S,µ [ 〈 µ, ∑ n i =1 ( x i -µ ) 〉 ] = 0 . Thus

Plugging the obtained upper bound on the left hand side gives us,

Above the first implication uses the assumption that δ ≤ 1 16 n priv . The last implication uses the fact that M √ d ≤ √ d M since M ≤ 1 . Rescaling by a R √ d factor yields the bound

Observe that any setting of M ≥ min { √ d n priv glyph[epsilon1] , 1 √ n pub } realizes the bound claimed in the theorem statement.

We now turn towards achieving a dependence on δ to prove Theorem 13. To do this, we leverage the the idea of filling a dataset with copies of each fingerprinting code seen in previous work [SU15, CWZ21]. However, in our case the introduction of public data makes this argument more delicate and leads to modified techniques for upper bounding the correlation statistics. See our discussion in Section B.2 for more details on why this is necessary.

Proof of Theorem 13. Let α = E [ ‖A ( S pub , S priv ) -µ ‖ ] , α ∗ = 1 125 min { d √ log(1 /δ ) nglyph[epsilon1] , √ d n pub } and assume by way of contradiction that α < α ∗ . Let k = 1 3 glyph[epsilon1] log ( 1 / [ √ dnδ ] ) . Let m = n k . We set

M = 4 √ d ( α ∗ + √ d m ) . Let µ ∼ Unif ([ -M,M ] d ) , S z = { z 1 , ..., z m } ∼ D µ . Sample S priv , S pub ∼ Unif ( { z 1 , ..., z m } ) and denote the combined dataset as S = { x 1 , . . . , x n } = ( S pub , S priv ) . Note that as in the proof of Theorem 12, we are starting by showing a lower bound for the case where the data is drawn from D µ instead of R √ d D µ , and will rescale at the end of the proof.

To prove our lower bound, we will provide upper and lower bounds on correlation statistics w.r.t. the intermediate dataset S z . We will also introduce a clipping procedure which helps better control the upper bound on correlation. In this regard, for each j ∈ [ m ] define Z j = 〈glyph[floorleft]A ( S pub , S priv ) glyph[floorright] M , z j -µ 〉 , where glyph[floorleft] v glyph[floorright] M denotes the operation of clipping every element of v to [ -M,M ] . In the following, we will provide upper and lower bounds on E [ ∑ m j =1 Z i ] and use this to show that α ≤ α ∗ implies a contradiction.

Lower Bound on Correlation We now want to lower bound E [ ∑ m j =1 Z j ] . Towards this end, we can apply fingerprinting lemma, Lemma 4, to the algorithm which outputs the clipping. For ̂ α > 0 , if E A ,S [ ‖glyph[floorleft]A ( S pub , S priv ) glyph[floorright] M -µ ‖ ] ≤ ̂ α , then this yields,

Now observe that

In the last step we use the assumed contradiction that α ≤ α ∗ . Thus it suffices to set ̂ α = α ∗ + √ d m . Now by the setting M = 4 √ d ̂ α and ̂ α ≤ √ d 12 , Eqn. (3) implies

Bounding the Number of Copies in the Dataset We now turn towards the more involved process of upper bounding E S,µ [ ∑ n i =1 Z i ] . To do this however, it will be first helpful to show that no datapoint in S z is copied into S too many times.

The first step is showing that no point is copied too many times into S . For j ∈ [ m ] , let Z j = { i ∈ [ n ] : x i = z j } . Observe

The second inequality follows from Bernstein's inequality for the sum of n Bernoulli random variables with mean 1 /m and the fact that E [ | Z j | ] = n m = k . Set τ = √ 8 glyph[epsilon1] log( dm ) 3 log(1 /δ ) and note since glyph[epsilon1] ≤ 1 and log ( dm ) ≤ log ( dn ) ≤ log (1 /δ ) (since δ ≤ 1 dn ), we have that τ ≤ 2 . Thus, denoting E as the event where no point in S z is copied into S more than 3 k times, we establish

glyph[negationslash]

Upper Bound on Correlation Under our model, we assume that A must treat all data in S priv as private. We will in fact only need to use the privacy property for a subset of samples in S priv to prove the correlation upper bound. Let I priv ⊆ [ m ] denote the set of indices s.t. j ∈ I priv if every copy of z j sampled into the overall dataset is in the private dataset S priv ; that is I priv = { j : ( ∀ x ∈ S pub ) x = z j } . Let I pub = [ m ] \ I priv . Observe that I priv may contain indices for points in S z which are never sampled into S . We will see this does not affect our analysis.

We have

The first term on the RHS can be bounded using the privacy property of A . For any fixed j ∈ I priv , let S ′ priv denote the dataset which replaces every instance of z j in S priv with a copied fresh sample from D µ . By the above analysis, conditional on the event E , at most 3 k such points need to be replaced conditional on the event E . Since A ( S pub , S ′ priv ) is independent of z j , by the Chernoff Hoeffding bound,

Since A satisfies k -group privacy with parameters ̂ glyph[epsilon1] ≤ 3 kglyph[epsilon1] and ̂ δ = e 3 kglyph[epsilon1] δ , we have

Setting τ = M √ d log (1 /δ ) , we obtain

The last inequality comes from the setting of k = 1 3 glyph[epsilon1] log ( 1 √ dnδ ) and the fact that M ≥ 1 √ n . Repeating this argument for each j ∈ I we get

The last inequality uses the bound established on each E [ Z j | E ] , j ∈ I priv, above and the bound on P [ E c ] from Eqn. (4).

To bound the correlation over the remaining vectors, we have

Above we have used the fact that |I pub | ≤ n pub because i ∈ I pub only if at least one copy of z i is sampled into S pub. Combining the above we have

Combining Bounds: The previously derived lower bound in Eqn. (3) establishes that E [ ∑ j ∈I priv Z j + ∑ j ∈I pub Z j ] ≥ d 3 . Using the above derived upper bounds we have the following manipulations,

The second line above uses that M = 4 √ d ( α ∗ + √ d m ) . Under the condition that n pub ≤ m 120 2 ≡ n pub ≤ 3 nglyph[epsilon1] 120 2 log ( 1 / [ √ ndδ ] ) , which is satisfied under by assumption in the theorem statement, we have

Now applying the assumption d ≥ 120 2 nglyph[epsilon1] = ⇒ m ≤ d 120 2 log(1 /δ ) we obtain

This establishes a contradiction, and thus α ≥ α ∗ = 1 125 min { d √ log(1 /δ ) nglyph[epsilon1] , √ d n pub } . Rescaling by R √ d then yields the claimed result.

B.2 Discussion of Lower Bound Analysis

We here provide more details on why the particular lower bound techniques we present were chosen. Our aim for the following discussion is to elucidate some of the subtleties of leveraging the fingerprinting code framework when public data is present, with the hope that it will aid future work on the characterization of PA-DP problems.

One crucial challenge in developing the mean estimation lower bounds in Appendix B.1 is ensuring that the correlation sum, traditionally defined as E [∑ x ∈ S 〈A ( S ) , x -µ 〉 ] , scales with the accuracy, α , of the algorithm. Previous work, such as [CWZ21], achieves this by setting the underlying distribution, D , to be a mixture distribution which, for some p = o (1) , samples a 0 vector with probability (1 -p ) and samples from the non-trivial distribution, D µ , with probability p . However, now the variance satisfies E x ∼D [ ‖ x -E [ x ] ‖ 2 ] ≤ 2 pR 2 meaning that when public data is present it

holds that E [ ‖ 1 n pub ∑ x ∈ S pub x -E x ∼D [ x ] ‖ ] ≤ 2 pR √ n pub = o ( R √ n pub ) , and one cannot hope to achieve the desired lower bound. Alternatively, by instead analyzing the sum E [∑ x ∈ S 〈A ( S ) -µ, x -µ 〉 ] , as seen for example in [KU20], we are able to avoid sampling from a mixture distribution. Further, by leveraging the flexibility of the strong distribution framework from [DSS + 15], we are able to still ensure ‖ µ ( D ) ‖ = o (1) , as needed for the SCO reduction; see Section B.3. These techniques lead to the result in Theorem 12.

Unfortunately, with regards to obtaining the √ log (1 /δ ) improvement in Theorem 13, the property E [ ‖A ( S ) -µ ‖ ] ≤ α does little to help establish the needed tail bound; see Eqn. (5). By clipping the components of A ( S ) to to the range [ -O ( α ) , O ( α )] , we are able to able to obtain the desired concentration. Unfortunately, this clipping technique in combination with the intermediate distribution, Unif ( S z ) , leads to the restrictions that d ≥ nglyph[epsilon1] and n pub ≤ n log(1 / [ ndδ ]) . These restrictions occur because of the need for the 'additional error' introduced by the intermediate distribution to be negligible. To see this, observe the intermediate distribution leads to E [ ‖A ( S ) -µ ‖ ] ≥ 1 √ m since A ( S ) depends on only m vectors from D µ , and the analysis in the proof of Theorem 12 (with n priv = 0 ) shows us that even non-private algorithms cannot do better on this distribution. We remark that [CWZ21] avoids this issue, and hence the restriction on d and n pub, because of the fact that one only actually needs ‖ E [ A ( S )] -µ ‖ ≤ α for the fingerprinting lemma to hold, and ‖ E [ A ( S )] -µ ‖ ≤ E [ ‖A ( S ) -µ ‖ ] . However, after clipping it is possible that ‖ E [ glyph[floorleft]A ( S ) glyph[floorright] M ] -µ ‖ ≥ ‖ E [ A ( S )] -µ ‖ .

B.3 Missing proofs from Section 3.1

Proof of Theorem 1. We use the instance in [BST14], glyph[lscript] ( w ; x ) = G 〈 w,x 〉 and W = { x ∈ R d : ‖ x ‖ ≤ 1 } . By a standard rescaling argument, we only need to consider G = D = 1 . We will consider the re-scaled data distribution used in Theorem 12, where { z 1 , ..., z n } i.i.d. ∼ D µ and the dataset S has x j = 1 √ d z j for j ∈ n . Here µ ∼ Unif ([ -M,M ] d ) where M will be chosen later.

First note by Lemma 5 we have that P [ ∣ ∣ ‖ µ ‖ -√ 2 3 M ∣ ∣ ≥ M 256 ] ≤ 1 512 so long as d is larger than some constant. Define this event as E and E ′ its complement. Thus we have

Thus it suffices to lower bound the conditional excess risk.

The optimal solution under the aforementioned loss is w ∗ = -µ ‖ µ ‖ , since the constraint set is a ball of radius 1 . We can see that

We will now lower bound ‖A ( S ) -w ∗ ‖ by using the lower bound for mean estimation developed in Theorem 11. Let the mean estimate candidate is ¯ µ ( S ) = ¯ µ = -√ 2 3 M A ( S ) . Under the event E ,

The above follows from the definition of w ∗ and since the algorithm's output is considered in a ball of radius 1 , so ‖A ( S ) ‖ ≤ 1 .

Combining the above inequalities (6) and (7) then taking expectation we have,

To bound E [ ‖ ¯ µ -µ ‖ 2 | E ] , observe

Rearranging we have

We will finish the bound by applying either Theorem 12 or Theorem 13.

Via Theorem 12: Set M = min { √ d 8 n priv , 1 √ n pub } . Under this setting of M , Theorem 12 implies that the lower bound on mean estimate distance satisfies E [ ‖ ¯ µ -µ ‖ ] ≥ M 8 , and thus E [ ‖ ¯ µ -µ ‖ 2 | E ] ≥ M 2 128 by Eqn. (9) above. Plugging into Eqn. (8) we have

Via Theorem 13: In Theorem 13, the setting of M used is

The inequality holds under the conditions d ≥ 120 2 nglyph[epsilon1] and n pub ≤ n 120 2 log ( 1 / [ √ ndδ ] ) . Thus we have under this setting of M that E [ ‖ ¯ µ -µ ‖ ] ≥ M 8 . Applying Eqns. (9) and (8) as in the previous case we have (providing the above conditions on d and n pub hold)

Lemma 5. For z ∼ Unif ([ -1 , 1] d ) , we have that ‖ z ‖ ∈ √ 2 d √ 3 ± √ 3 ln (1 /γ ) 2 , with probability at least 1 -γ .

Proof. This follows from standard concentration of norm results. We have that,

As in [Ver18, proof of Theorem 3.1.1], we use the simple fact that | x -1 | > δ = ⇒ ∣ ∣ x 2 -1 ∣ ∣ > max( δ, δ 2 ) for any x, δ ≥ 0 , to get,

We substitute ¯ δ = 2 δ 3 and apply Bernstein's inequality for i.i.d sub-exponential random variables z 2 i . Since, z i ∈ [ -1 , 1] , the sub-exponential norm ≤ 1 . Applying Corollary 2.8.3 from [Ver18], we get that,

This gives us that

Hence, with probability, at least 1 -γ , we have that ‖ z ‖ ∈ √ 2 d √ 3 ± √ 3 ln (1 /γ ) 2 , which completes the proof.

Proof of Theorem 2. We use the squared loss instance as in [BST14, Section 5.2]; glyph[lscript] ( w ; z ) = λ 2 ‖ w -z ‖ 2 , with ‖ z ‖ ≤ G 2 λ . The loss is G -Lipschitz and λ strongly convex on the domain of unit ball at zero of radius G 2 λ . Given a datasets S = { z 1 , z 2 , . . . , z n } , the population risk minimizer is simply the population mean µ ( D ) . Further, it is straightforward to verify that the excess population risk a re-scaling of the mean estimation error

Substituting the mean estimation lower bounds, Theorem 11, completes the proof.

C Missing Proofs from Section 4

C.1 Proof of Theorem 3

Define the orthogonal projection matrix P X pub = UU glyph[latticetop] . Note that the feature vectors in ˜ S priv = { ( U glyph[latticetop] x i , y i ) } n priv i =1 are bounded. In particular ∥ ∥ U glyph[latticetop] x ∥ ∥ 2 = x glyph[latticetop] ( UU glyph[latticetop] ) x = x glyph[latticetop] P X pub x ≤ ‖ x ‖ 2 , since P X pub is an orthogonal projection onto span ( W∩ S pub ) . Further, since ˜ w ∈ ˜ W , we have that there exists ˚ w ∈ W such that ˜ w = U glyph[latticetop] ˚ w . Finally, ̂ w = U ˜ w = UU glyph[latticetop] ˚ w = P X pub ˚ w ∈ W since the range of P X pub ⊆ W .

The privacy guarantee follows from the privacy guarantee of sub-routine ˜ A . For utility, we define w ∗ ∈ arg min w ∈W L ( w ; D ) and ˜ w ∗ ∈ arg min w ∈ ˜ W L ( w ; U glyph[latticetop] D ) , where U glyph[latticetop] D denotes the distribution which first samples from D then project using U glyph[latticetop] . Let ˚ w ∗ ∈ W such that ˚ w ∗ = U ˜ w ∗ .

Note that from the GLM structure, L ( ˜ w ∗ ; U glyph[latticetop] D ) = L ( ˚ w ∗ ; D ) . We have,

with probability at least 1 -β/ 4 . In the above, we control the generalization gap via uniform convergence and concentration for the fixed ˚ w ∗ with respect to S priv .

The last term ̂ L ( ˜ w ; ˜ S priv ) -min w ∈ ˜ W ̂ L ( w ; ˜ S priv ) is bounded by the guarantee of the private sub-routine with probability at least 1 -β/ 4 ,

Finally, for any ¯ w ∗ such that ¯ w ∗ ∈ U ˜ W , with probability at least 1 -β/ 2 , from G -Lipschitznes, we have

where the last inequality follows essentially from Lemma 6 and Lemma 1 for n pub = O ( max ( R 2 log(2 /β ) α 2 , min { m : log 3 ( n pub ) R 2 n pub ( H ) ≤ α 2 })) . To elaborate, the first step holds since ˚ w ∗ = U ˜ w ∗ and ˜ w ∗ is the the minimizer of risk over ˜ W . Now, Lemma 1 guarantees that for any w ∗ ∈ W , there exists a ¯ w ∗ in its α -cover with respect to ‖·‖ 2 ,X pub , with ‖ w ∗ -¯ w ∗ ‖ 2 , D X ≤ α . To argue why span ( X pub ) is an α -cover, from Lemma 6, we have that from any α -cover ¯ W , of W w.r.t. ‖·‖ 2 ,X pub , we can remove elements which do not lie in span ( S pub ) and still have an α -cover. Hence, the superset used in Algorithm 1, which essentially is, ¯ W = P X pub W , is indeed an α -cover. The n pub we get is,

where in the above, we plug in the Rademacher complexity of bounded linear predictor, R m ( H ) = Θ ( D ‖X‖ m ) . Plugging the above in Equation (10),

where the above follows by setting α = D ‖X‖ (log(1 /δ )+log(2 /β )) 1 / 4 √ n priv glyph[epsilon1] . This yields the claimed rate. The resulting public sample complexity is

This completes the proof.

Lemma 6. Let ˜ H be a α -cover of H with respect ‖·‖ 2 ,X pub . Then, ¯ H = ˜ H∩ span ( X pub ) is also an α -cover.

Proof. Given two h 1 , h 2 ∈ H , we have,

where w 1 and w 2 are the vectors corresponding to linear functions h 1 and h 2 and X pub denote the matrix of public feature vectors.

Given any h ∈ H , let ˜ h denote the element closest to it in the cover ˜ H ; we have, ‖ h -˜ h ‖ 2 ,X pub ≤ α . Consider the singular value decomposition, X pub = V Σ U glyph[latticetop] , where U and V are orthogonal matrices and Σ is a diagonal matrix. Define ¯ h = P X pub ( ˜ h ) = UU glyph[latticetop] ˜ h . Note that U is an orthogonal projection onto span ( X pub ) and P X pub is the corresponding projection matrix. We have,

Since by construction ¯ h also lies in span ( X pub ) , this proves the claim.

C.2 Proof of Theorem 6

We state the complete version of this theorem and then present its proof.

Theorem 14. Let glyph[epsilon1] > 0 , δ > 0 and glyph[epsilon1] ≤ log (1 /δ ) . For a G -Lipschitz, B -bounded non-negative H -smooth loss function, Algorithm 1 satisfies ( glyph[epsilon1], δ ) -DP. If the private subroutine ˜ A guarantees Equation (1) with probability at least 1 -β , then with n pub = ˜ O ( ( HD ‖X‖ ) 2 / 3 ( n priv glyph[epsilon1] ) 2 / 3 G 2 / 3 (log(1 /δ )) 1 / 3 + √ Hn priv glyph[epsilon1] √ L ( w ∗ ; D ) G √ log(1 /δ )) ) , with probability at least 1 -β , L ( ̂ w ; D ) -L ( w ∗ ; D ) is at most

where w ∗ and ̂ w ∗ are population and empirical minimizers with respect to D and S priv respectively.

Proof of Theorem 14. The privacy guarantee follows from the privacy guarantee of sub-routine ˜ A . The proof of the utility guarantee proceeds similar to that of Theorem 3. We define w ∗ ∈ arg min w ∈W L ( w ; D ) and ˜ w ∗ ∈ arg min w ∈ ˜ W L ( w ; U glyph[latticetop] D ) . Let ˚ w ∗ ∈ W such that ˚ w ∗ = U ˜ w ∗ . From the GLM structure, L ( ˜ w ∗ ; U glyph[latticetop] D ) = L ( ˚ w ∗ ; D ) . We have,

The last term ̂ L ( ˜ w ; ˜ S priv ) -min w ∈ ˜ W ̂ L ( w ; ˜ S priv ) is bounded by the guarantees of the private subroutine with probability at least 1 -β/ 4 ,

To bound the term L (˚ w ∗ ; D ) -L ( w ∗ ; D ) in Equation (11), we apply smoothness to get,

where the above holds for any ¯ w ∗ ∈ H such that ¯ w ∗ ∈ U ˜ W by optimality of ˜ w ∗ in ˜ W . The second inequality holds from H -smoothness, the third and fourth from CauchySchwarz, the fifth from self-bounding property of smooth non-negative losses (Lemma 4.1 in [SST10]). The final step holds with probability 1 -β/ 2 from Lemma 1 with n pub = O ( max ( R 2 log(2 /β ) α 2 , min { m : log 3 ( m ) R 2 m ( H ) ≤ α 2 } )) together with that since ˜ W is an α -cover

of H , together with Lemma 6 which shows that ˜ W is a valid α -cover of W . Therefore, there exists ¯ h ∗ ∈ ˜ H with ∥ ∥ ¯ h ∗ -h ∗ ∥ ∥ 2 ,S pub ≤ α .

Further, applying AM-GM inequality, we get

The first two terms in Equation (11) are bound via uniform convergence for smooth non-negative losses, (Theorem 1 in [SST10]) and Bernstein's inequality as follows; with probability at least 1 -β/ 4 , we have,

where the second equality follows from Equation (12), instantiating the Rademacher complexity of linear predictors, concavity of x ↦→ √ x and AM-GM inequality. The third equality follows concavity of x ↦→ √ x and Bernstein's inequality, the fourth follows from Equation (14) and AM-GM inequality.

Plugging the above, Equation (13) and Equation (12) into Equation (11), we get that with n pub = O ( max ( ‖X‖ 2 D 2 log(2 /β ) α 2 , min { m : log 3 ( n pub ) R 2 n pub ( H ) ≤ α 2 })) , the following holds with probability at least 1 -β ,

The public sample complexity is,

This completes the first part of the theorem. For thee second part, we start from Equation (16),

where the second inequality holds form optimality of w ∗ , the third from uniform convergence, Theorem 1 in [SST10] and AM-GM inequality, and the last by plugging in the following public sample complexity.

This completes the proof.

Theorem 15. In the setting of Theorem 14 with the additional assumption that the global minimizer of risk L , w ∗ lies in W , we get that with n pub = ˜ O ( ( HD ‖X‖ ) 2 / 3 ( n priv glyph[epsilon1] ) 2 / 3 G 2 / 3 (log(1 /δ )) 1 / 3 ) , with probability at least

1 -β ,

Further,

where w ∗ and ̂ w ∗ are population and empirical minimizers with respect to D and S priv respectively.

Proof. The proof is almost identical to that of Theorem 15. We repeat the steps pointing out the differences and how the expressions change. We continue till Equation (12). Next, we apply smoothness which results in the key difference between the analyses,

where last equality uses the fact that ∇ L ( w ∗ ; D ) = 0 since w ∗ is the unconstrained minimizer. Continuing, we get,

This yields,

The public sample complexity is,

The second part follows similarly.

C.3 Lower bounds

C.3.1 Proof of Theorem 4

To establish the GD √ ‖X‖ n term in the lower bound, we consider a one-dimenisonal problem where the loss φ y ( ̂ y ) = -Gy ̂ y and marginal distribution D x as the point distribution on ‖X‖ such that the overall loss is E x,y [ glyph[lscript] ( w, ( x, y ))] = E y [ y · w ‖X‖ G ] . We further set W = [ -D,D ] and consider D y to be the distribution which as 1 with probability P [ y = 1] = (1 + µ ) / 2 and P [ y = 1] = (1 -µ ) / 2 for some µ ∈ [ -1 , 1] . Note the minimizer w ∗ = D µ | µ | achieves population risk -µGD ‖X‖ . Classic results in information theory establish if µ is sampled uniformly from {± 1 √ 6 n } , no algorithm can estimate the sign of µ with probability better than 1 / 2 (see [Duc23, Section 8.3]). Thus it must be that for any algorithm E A ,S [ L ( A ( S ); D ) -min w ∈ R d L ( w ; D )] = Ω ( GD √ ‖X‖ n ) .

The GD ‖X‖ min { 1 √ nglyph[epsilon1] , √ d nglyph[epsilon1] } term in the lower bound is essentially a corollary of [ABG + 22, Theorem 6]. We provide further remarks here. The loss function used is,

Define d ′ := min( d, nglyph[epsilon1] ) and p := min ( 1 , d ′ nglyph[epsilon1] ) . The (known) marginal distribution D X is described as: with probability 1 -p , x = glyph[vector] 0 , otherwise, x ∼ Unif ( ‖X‖{ e j } d ′ j =1 ) where e j 's are canonical basis vectors. The (unknown) conditional distribution of the response y is as follows. Sample a 'fingerprinting code', z ′ ∈ { 0 , 1 } d ′ with mean µ ′ ∈ [0 , 1] d ′ where each co-ordinate µ ′ j ∼ Beta (0 . 0625 , 0 . 0625) i.i.d. Embed z ′ in d dimensions as z and let µ be the corresponding mean vector. Finally, define y = D 〈 z,x 〉 √ d ′ . The proof in [ABG + 22, Theorem 6] then proceeds by lower bounding the loss by bounding the ability of any differential private algorithm to estimate the fingerprinting code z .

Since the rank of E x ∼D X [ xx glyph[latticetop] ] = d ′ , the result [ABG + 22, Theorem 6] then yields a lower bound on the unconstrained excess risk ,

but also guarantees that the global minimizer has norm at most D . Thus, we achieve the same lower bound for E A ,S [ L ( A ( S ); D ) -min w ∈W L ( w ; D )] by setting W to be the ball of radius D .

C.3.2 Proof of Theorem 5

The proof uses the lower bound instance in the DP-SCO lower bound with public data, Theorem 1. We consider the case where D y is the point distribution on 1 . Then for any y ∈ Y , Y = { 1 } , the loss function is then glyph[lscript] ( w ; ( x, y )) = y 〈 w,x 〉 = 〈 w,x 〉 , as in Theorem 1. Hence, a labeled and unlabeled sample have the same information. We also set W to be the ball of radius D .

Assume by contradiction there exists an ( glyph[epsilon1], δ ) -PA-DP algorithm, A , which achieves rate O ( GD ‖X‖ ( 1 √ n priv + √ log(1 /δ ) √ n priv glyph[epsilon1] ) ) with o ( n priv glyph[epsilon1]/ log (1 /δ )) public samples. Since n pub = o ( n priv glyph[epsilon1]/ log (1 /δ )) and d = ω ( nglyph[epsilon1] ) , Theorem 1 gives a lower bound on E [ A ( X pub , S priv ; D ) -min w ∈W { L ( w ; D ) } ] of

Since glyph[epsilon1] ≤ 1 , this is a contradiction.

D Missing proofs for Section 4.2

D.1 Proof of Theorem 7

Proof. The privacy proof follows from the guarantee of exponential mechanism [MT07]. In particular, the sensitivity of the score function is at most 2 n priv min( B,GR ) where the first follows from the loss bound of B and the second from the Lipschitzness and bound on predictors. Let h ∗ ∈ arg min h ∈H L ( h ; D ) and ˜ h ∗ ∈ arg min h ∈ ˜ H L ( h ; D ) . From standard analysis based on uniform convergence, we have

where the above holds with probability at least 1 -β/ 2 and follows from guarantee of exponential mechanism [MT07] and uniform convergence ([SSBD14], see Theorem 16). We further have that log ( | ˜ H| ) = ˜ O ( fat cα ( H )) from Lemma 7.

For the L ( ˜ h ∗ ; D ) -L ( h ∗ ; D ) term, we have

where the first step holds for any ¯ h ∗ ∈ ˜ H by optimaility of ˜ h ∗ over ˜ H , the second holds from the G -Lipschitzness of the loss function, the third from Jensen's inequality. The final step holds with probability 1 -β/ 2 from Lemma 1 with n pub = O ( max ( R 2 log(2 /β ) α 2 , min { m : log 3 ( m ) R 2 m ( H ) ≤ α 2 } )) together with the fact that since ˜ H is an α -cover of H , hence there exists ¯ h ∗ ∈ ˜ H with ∥ ∥ ¯ h ∗ -h ∗ ∥ ∥ 2 ,X pub ≤ α .

Plugging the above in Equation (19), we get with probability at least 1 -β ,

which finishes the proof.

Theorem 16. [SSBD14] Let H ⊆ [ -R,R ] X . For any G -Lipschitz, B -bounded loss function, any probability distribution D over X ×Y , given m i.i.d. samples from S , with probability at least 1 -β , the following holds for all h ∈ H ,

Proof. This is a classical result in learning theory which follows directly Theorem 26.5 in [SSBD14] together with the contraction lemma (Lemma 26.9 in [SSBD14]) for Lipschitz losses.

Lemma 7. Let ˜ H be an α -cover of H with respect to ‖·‖ 2 ,X pub . The size of ˜ H is bounded as,

where c is an absolute constant.

Proof. This follows directly from Theorem 8,

Proof of Lemma 1 For h ∈ H , let ˜ h ∈ arg min ¯ h ∈ ˜ H ‖ ¯ h -h ‖ 2 ,X pub . Since ˜ H is an τ -cover, this gives us that ‖ h -˜ h ‖ 2 ,X pub ≤ τ . We have,

The first term above can be seen as uniform deviation between the empirical and population risk, of another prediction problem, with squared loss, in the the realizable setting (with the responses generated by h ). The squared loss is 1 2 -smooth and non-negative, so we can apply result of Theorem 1 in [SST10] instantiated in the realizable setting, which gives us that with probability at least 1 -β ,

Choosing n pub such that n pub = O ( max ( R 2 log(1 /β ) α 2 , min { m : log 3 ( m ) R 2 m ( H ) ≤ α 2 } )) , we get the claimed result.

D.2 Optimistic rates with smooth non-negative losses

Algorithm 2 achieves optimistic rates on risk depending on realizability/interpolation conditions, that is, whenever L ( h ∗ ; D ) or ̂ L ( ̂ h ; S priv ) is small.

Theorem 17. Algorithm 2 with γ = 2 B n priv glyph[epsilon1] satisfies glyph[epsilon1] -PA-DP. For n pub = O ( max ( R 2 log(2 /β ) α 2 , min { m : log 3 ( m ) R 2 m ( H ) ≤ α 2 } )) < ∞ and any α > 0 , with probability at least 1 -β , we have

where h ∗ and ̂ h ∗ are population and empirical minimizers with respect to D and S priv respectively, and c is an absolute constant.

Proof of Theorem 17. The privacy proof is the same as that of Theorem 7. For utility, let h ∗ ∈ arg min h ∈H L ( h ; D ) and ˜ h ∗ ∈ arg min h ∈ ˜ H L ( h ; D ) .

We start with the proof of the first part of the theorem. We have,

From the guarantee of exponential mechanism together with log (∣ ∣ ∣ ˜ H ∣ ∣ ∣ ) = ˜ O ( fat cα ( H )) from Lemma 7, we have that with probability at least 1 -β/ 4 ,

For the L ( ˜ h ∗ ; D ) -L ( h ∗ ; D ) term in Equation (20), we apply smoothness to get,

where the above holds for any ¯ h ∗ ∈ ˜ H . The second inequality holds from H -smoothness, the third and fourth from Cauchy-Schwarz, the fifth from self-bounding property of smooth nonnegative losses [SST10]. The final step holds with probability 1 -β/ 2 from Lemma 1 with n pub = O ( max ( R 2 log(2 /β ) α 2 , min { m : log 3 ( m ) R 2 m ( H ) ≤ α 2 } )) together with the fact that since ˜ H is an α -cover of H , so there exists ¯ h ∗ ∈ ˜ H with ∥ ∥ ¯ h ∗ -h ∗ ∥ ∥ 2 ,X pub ≤ α .

An application of AM-GM inequality further yields,

The first two terms in Equation (20) are bound using uniform convergence for smooth non-negative losses, Theorem 1 in [SST10] and Bernstein's inequality as follows; with probability at least 1 -β/ 4 , we have,

where the second equality follows from Equation (21), concavity of x ↦→ √ x and AM-GM inequality. The third equality follows concavity of x ↦→ √ x and Bernstein's inequality, the fourth follows from Equation (23) and AM-GM inequality.

Plugging the above, Equation (22) and Equation (21) into Equation (20) yields the following with probability at least 1 -β ,

This completes the first part of the theorem. For the second part, we proceed from Equation (25) onwards

where the second inequality follows form optimality of h ∗ , the third from uniform convergence, Theorem 1 in [SST10] and AM-GM inequality. This completes the proof.

D.3 Proof of Corollary 2

We use the result from [GRS18], restated as Theorem 10. Further, note that range bound on the hypothesis class is simply R ≤ ‖X‖ ∏ d j =1 R j . Instantiating our general result Theorem 7 with the above together with the relation between fat-shattering dimension and Rademacher complexity (Theorem 9) yields the following excess risk bound,

The number of public samples then is

D.4 Proof of Corollary 3

Note that R ≤ D ‖X‖ . Further, we have [KST08, FGV17],

Moreover, we have from Theorem 9, for any α > R m ( H ) ,

Choose m = n pub and α = (log ( n pub )) 3 / 2 R n pub ( H ) , to get that fat α ( H ) = ˜ O ( n pub ) . Plugging this in Theorem 7, we get,

where in the last step, we plug in n pub = O ( ( n priv glyph[epsilon1] ) r/ (1+ r ) log (2 /β ) ) , yielding α = O ( D ‖X‖ ( n priv glyph[epsilon1] ) 1 r +1 ) . The number of public samples simplifies as,

which is satisfied from our choice since r ≥ 2 .

D.5 Additional Results

Corollary 4. In the setting of Theorem 7 together with X = { x ∈ R d : ‖ x ‖ ≤ ‖X‖ } and H = { x ↦→〈 w,x 〉 , x ∈ X , ‖ w ‖ ≤ D } . With α = ( D 2 ‖X‖ 2 n priv glyph[epsilon1] ) 1 / 3 and n pub = ˜ O ( ( D ‖X‖ ) 2 / 3 ( n priv glyph[epsilon1] ) 2 / 3 log (2 /β ) ) , with probability at least 1 -β ,

Proof. In this setting, we have that R ≤ D ‖X‖ . Further, it is known [KST08],

Plugging this in Theorem 7, we get,

where in the last step, we plug in α = ( D 2 ‖X‖ 2 n priv glyph[epsilon1] ) 1 / 3 . The number of public samples simplifies as,

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Claims made are given in Sections 3 and 4 proofs are provided in the Appendix Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The specific assumptions made are detailed in the preliminaries and theorem statements.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Statements are proved or cite a relevant reference. Many of these proofs can be found in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: There are no experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: There is no associated code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: There are no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: There are no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: There are no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The theoretical nature of the results means there are minimal ethical concerns.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The theoretical nature of the work means that any societal impact would be very indirect.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No such assets are used as a part of this research.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: No such assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No such assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No human subjects were involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No human subjects were involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents some new lower bounds for public-data assisted private SCO, when some public examples are available, either with or without labels.\nIn the unlabeled case, a simple algorithm for GLM assisted with unlabeled public examples is presented and analyzed (and it is shown that this removes a dimension dependent term, similar to previous results).\n\nSTRENGTHS:\n- The problem is important.\n- The new results shed additional light on the limits of public-data assisted private SCO.\n- In particular, they partially closes the gap between lower and upper bounds, in certain regimes of $d$ and $n_{pub}$.\n- There is good commentary following most of the results.\n\nWEAKNESSES:\n- The presentation is poor. Little effort was made to summarize the main results (and how they improve on previous bounds) in an easily accessible form. This could be remedied with a table summarizing assumptions, new and existing results, conditions on $n_{priv}, n_{pub}, \\epsilon, d$, ranges in which the bounds are tight, etc.\n- For example, it was unclear from the abstract and introduction in which regimes the new bounds are tight. This is only discussed later in the text. Similarly, there are boundedness assumption that should have been clarified early on.\n- In Theorems 3, 6, 7, it seems there are missing conditions on $n_{pub}$. For example, according to the theorem statement, nothing prevents one from taking $n_{pub} = 1$, or even 0, and it seems one would still get the improved bound. This should not be the case. Please clarify in the rebuttal.\n- The significance of Theorem 3 is unclear. Similar algorithms and similar improvements of dimension dependence were obtained in other works. The authors mention line 245 that their result avoids \"many of the strong assumptions seen in previous work\" without giving any details. What are these assumptions and how does Theorem 3 compare to existing results?\n- Corollaries 2 and 3 are frankly hard to interpret, and their significance is unclear to me. Maybe more effort is needed in this section.\n- The general message of the paper, that \"more sophisticated attempts at leveraging public data will yield no benefit\" (in the labeled case), should be toned down a bit. While the new bounds indeed show some fundamental limits, these come under assumptions, and improvements were in fact shown for certain classes of problems like linear regression, as in some of the papers that were cited lines 28-29, (most of these were cited without discussion unfortunately).\n\nMinor:\n- What do the authors mean by \"the non trivial regime $n_{pub} = \\Theta(n)$ and $n_{pub} = o(n)$\"? Do you mean either of these is true? (if so, it would be clearer to use or, not and).\n- There are minor formatting issues like a broken reference line 359 and some typos.\n- In the discussion following Cor.3, how do the $log d$ terms appear?\n\nQUESTIONS:\nPlease see questions in the weaknesses.\n\nLIMITATIONS:\nAs mentioned above, the paper should be clearer about its assumptions, regimes in which its bounds are tight, etc.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates a public-data-assisted differential privacy problem. Firstly, for labeled public data, the author introduces a novel mean estimation lower bound of $\\tilde{\\Omega}\\left(\\min \\left{\\frac{1}{\\sqrt{n_{\\mathrm{pub}}}}, \\frac{1}{\\sqrt{n}}+\\frac{\\sqrt{d}}{n \\epsilon}\\right}\\right)$. Secondly, when considering unlabeled public data with $n_{\\text {pub }} \\geq n_{\\text {priv }}$, the paper presents a dimension-independent rate of $\\tilde{O}\\left(\\frac{1}{\\sqrt{n_{\\text {priv }}}}+\\frac{1}{\\sqrt{n_{\\text {priv }} \\epsilon}}\\right)$ given $\\tilde{O}\\left(n_{\\text {priv }} \\epsilon\\right)$ unlabeled public data. Additionally, the results for unlabeled public data are extended to general hypothesis classes with bounded fat-shattering dimensions.\n\nSTRENGTHS:\n1. The paper provides a technically solid and well-structured analysis of the public-data-assisted differential privacy problem. The most intriguing part is the Private Supervised Learning with Unlabeled Public Data. In this setting, the paper leverages $n_{\\text {pub }} \\geq n_{\\text {priv }}$, showing a dimension-independent rate and demonstrating that an increasing number of unlabeled public data cannot improve the results because unlabeled public data can only reveal information about the marginal distribution.\n\n2. The emergence of the dimension-independent rate from \"public data can be used to identify a low-dimensional subspace, which under the appropriate metric acts as a cover for the higher-dimensional space,\" is particularly interesting. Could the author provide more explanations and insights on this?\n\n3. Algorithm 1 projects the private data to an orthogonal space, derives the solution, and then projects back to the original space to obtain the result. This approach is novel to me, but I have some questions below.\n\nWEAKNESSES:\nN/A\n\nQUESTIONS:\nFor Algorithm 1, why does the author perform the projection before applying the differential privacy subroutine? What is the function of the projection here, or how does it help the result? I am not sure if it is because the author uses the unlabeled dataset to construct a data space.\n\nLIMITATIONS:\nRefers to the weakness and questions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the effectiveness of using public data to assist private convex optimization. Private convex optimization gives the solver query access to a convex function $f:W\\times X \\to \\mathbb{R}$ and samples $S$ drawn i.i.d. from some unknown distribution $D$. The solver is asked to find some point $w\\in W$ that minimizes the population risk $\\mathbb{E}_{x\\sim D}[f(w,x)]$, but to do so in a manner that is private with respect to the sample $S$. This is an important problem that has received much interest in the community. The notion of using public data that one expects has been drawn from the same distribution has been explored earlier, and the authors continue this line of work.\n\nThe main contributions of this work are:\n1. Stronger lower bounds than prior work showing that in the general setting even for approximate DP there is a dependence on the dimension $d$ of the sample space $X = \\mathbb{R}^d$ (typically one assumes data is drawn from some bounded $d$-dimensional ball in the DP setting). They also complement prior work and extent a lower bound for the pure DP setting that held previously only for a limited range of values. These lower bounds show that the naive approach of either treating all data (public and private) as private or ignoring the private data and using SCO on public data is asymptotically optimal.\n2. A new algorithm for the unlabelled setting showing that one can gainfully use unlabelled public data to perform dimension reduction for labelled private data and achieve a risk bound that has no dependence on the ambient dimension. This is in contrast to the labelled setting where their lower bounds showed a dependence on the ambient dimension $d$.\n\nSTRENGTHS:\n1. I think this problem is well-motivated and of interest to the DP community.\n2. The asymptotic separation between the labelled and unlabelled settings is interesting, and the improvement of lower bounds compared to prior work is also good to see.\n3. The paper is in general well-written and easy to follow.\n\nWEAKNESSES:\n1. I think that in this case it becomes important to understand the exact constants that occur in the upper bound in the labelled case - even if asymptotically one does not hope to see any improvement over the naive strategies, I would imagine that in practice one still does gain some improvement in the public case. Maybe this point could be better made.\n2. Although the analyses seem to be formal and strong (and relatively assumption free), there are not necessarily any new high-level ideas in this work (if you would like to emphasize any new highlights or unexpected outcomes that would be great to see though). The technical work is still interesting however.\n\nQUESTIONS:\nThere is one point that I don't understand, I think I might be confused about the nature of the results. In principle, if one treats public labelled data as unlabelled and attempts to follow a similar approach of using them to learn a good lower dimensional representation of the labelled private data, is it easy to see why this does not lead to an asymptotic improvement over ignoring the public data entirely, or treating it as private?\n\nLIMITATIONS:\nNo additional limitations need to be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}