Title: Revisiting Differentially Private ReLU Regression
Venue: NeurIPS 2024
Accepted: yes
Keywords: Differential Privacy

Generalized Linear Model
Abstract: As one of the most fundamental non-convex learning problems, ReLU regression under differential privacy (DP) constraints, especially in high-dimensional settings, remains a challenging area in privacy-preserving machine learning. Existing results are limited to the assumptions of bounded norm $ \|\mathbf{x}\|_2 \leq 1$, which becomes meaningless with increasing data dimensionality. In this work, we revisit the problem of DP ReLU regression in high-dimensional regimes. We propose two innovative algorithms DP-GLMtron and DP-TAGLMtron that outperform the conventional DPSGD. 
DP-GLMtron is based on a generalized linear model perceptron approach, integrating adaptive clipping and Gaussian mechanism for enhanced privacy. To overcome the constraints of small privacy budgets in DP-GLMtron, represented by $\widetilde{O}(\sqrt{1/N})$ where $N$ is the sample size, we introduce DP-TAGLMtron, which utilizes a tree aggregation protocol to balance privacy and utility effectively, showing that DP-TAGLMtron achieves comparable performance with only an additional factor of $O(\log N)$ in the utility upper bound.
Moreover, our theoretical analysis extends beyond Gaussian-like data distributions to settings with eigenvalue decay, showing how data distribution impacts learning in high dimensions. Notably, our findings suggest that the utility upper bound could be independent of the dimension $d$, even when $d \gg N$. 
Experiments on synthetic and real-world datasets also validate our results.

Revisiting Differentially Private ReLU Regression

Abstract

As one of the most fundamental non-convex learning problems, ReLU regression under differential privacy (DP) constraints, especially in high-dimensional settings, remains a challenging area in privacy-preserving machine learning. Existing results are limited to the assumptions of bounded norm ∥ x ∥ 2 ≤ 1 , which becomes stringent and strong with increasing data dimensionality. In this work, we revisit the problem of DP ReLU regression in overparameterized regimes. We propose two innovative algorithms, DP-GLMtron and DP-TAGLMtron, that outperform the conventional DPSGD. DP-GLMtron is based on a generalized linear model perceptron approach, integrating adaptive clipping and Gaussian mechanism for enhanced privacy. To overcome the constraints of small privacy budgets in DPGLMtron, represented by ˜ O ( √ 1 /N ) where N is the sample size, we introduce DP-TAGLMtron, which utilizes a tree aggregation protocol to balance privacy and utility effectively, showing that DP-TAGLMtron achieves comparable performance with only an additional factor of O (log N ) in the utility upper bound. Moreover, our theoretical analysis extends beyond Gaussian-like data distributions to settings with eigenvalue decay, showing how data distribution impacts learning in high dimensions. Notably, our findings suggest that the utility bound could be independent of the dimension d , even when d ≫ N . Experiments on synthetic and real-world datasets also validate our results.

1 Introduction

Protecting individual privacy in data analysis has emerged as a critical concern. In light of the vast quantities of personal and sensitive information involved, traditional methods of ensuring privacy are encountering significant challenges. Differential Privacy (DP), introduced by [13], has gained widespread recognition as a method for preserving privacy by adding a controlled amount of random noise to the data or query responses, thereby effectively concealing the details of any individual.

Differentially Private Stochastic Optimization (DP-SO) and its empirical form, Differentially Private Empirical Risk Minimization (DP-ERM), are foundational models extensively studied in the DP community. Although there is a substantial body of research on DP-SO and DP-ERM with convex loss functions [10, 46, 7, 16, 35, 37, 6, 9, 26, 36, 38, 39], the understanding of nonconvex optimization in a DP context remains limited. Recent efforts have developed private algorithms with established proven utility bounds for differentially private nonconvex optimization [51, 46, 43, 52, 8, 47]. However, most of the current work employs the gradient norm of the population risk function as a measurement instead of excess population risk, commonly used in convex scenarios. Recently, [33] comprehensively studied different instances of non-convex learning, such as generalized linear

models, ReLU regression, and multi-layer neural networks. However, for each problem, they imposed different strong assumptions, which left a big room to further understanding DP non-convex learning. In this paper, we consider the most fundamental one, the ReLU (Rectified Linear Unit) regression model.

As one of the most fundamental non-convex models, ReLU regression stands out due to its widespread use and effectiveness in deep learning. Despite its prevalence in industry, theoretical exploration of ReLU regression has been relatively limited. [33] studies the DP ReLU regression problem in both well-specified and misspecified settings. However, the problem is still far from well-understood. By and large, several challenges need to be addressed. Their analysis, for instance, heavily relies on strong assumptions about bounded norms of feature vectors and labels, i.e. ∥ x ∥ 2 ≤ O (1) and | y | ≤ O (1) , which does not hold even for normal Gaussian distributions. Secondly, while they provide a utility upper bound of ˜ O ( 1 √ N +min { d 1 / 2 ( Nε ) 1 / 2 , 1 ( Nε ) 2 / 3 } ) with sample size N and dimension d , their dimension-independent results heavily rely on their data assumption, leading to a dimensiondependent upper bound when considering Bernoulli or O (1) -subGaussian data where ∥ x ∥ 2 ≤ O ( √ d ) with high probability. That means when d ≫ N , i.e., when in the overparameterization regime, previous results become trivial.

In this paper, we revisit the DP ReLU regression problem in the overparameterized regime. Specifically, we provide a general analysis under the well-specified setting and propose two novel algorithms (DP-GLMtron and DP-TAGLMtron). Our contributions are three-fold, see Table 1 for details.

1) We propose an innovative algorithm, DP-GLMtron, built upon the Generalized Linear Model Perceptron (GLM-tron) algorithm of [23]. Specifically, DP-GLMtron operates by making a single pass over the dataset and processes one data point at a time by integrating appropriate clipping and adding Gaussian noise to maintain privacy. Additionally, we provide an excess population risk upper bound of ˜ O ( D eff N + D pri eff N 2 ε 2 ) where D eff ( D pri eff ) is the (private) effective dimension. It is noticed that D eff and D pri eff are defined in a way that does not directly rely on the dimension d . Furthermore, our results are applicable not only to Gaussian-like data, typically considered in the traditional differential privacy community, but also to data distributions that exhibit either polynomial or exponential decay. In these cases, we demonstrate that the utility bound is independent of the dimension d , which marks a significant departure from the traditional one (see more discussions in Remark 4). Finally, we also show that our analysis on DP-GLMtron is almost tight by providing a lower bound of ˜ Ω( D eff N + D pri eff N 2 ε 2 ) .

2) A significant issue with DP-GLMtron is its upper bound only holds with a small privacy budget ε = ˜ O ( 1 √ N ) (for further details, see Remark 3), which is due to privacy amplification via shuffling. To tackle this limitation, we propose DP-TAGLMtron, which utilizes the tree aggregation protocol to introduce noise into the sum of mini-batch gradients instead of using privacy amplification. Our analysis reveals that the utility upper bound of DP-TAGLMtron is closely comparable to that of DP-GLMtron only with an additional factor of O (log N ) . This finding indicates that DP-TAGLMtron can offer similar performance benefits while potentially being applicable in settings with larger privacy budgets.

3) We conducted experiments with both synthetic data and real data on the ReLU regression model across various algorithms and privacy budgets. Specifically, in the first part, synthetic data was generated under two eigenvalue decay scenarios, λ i ∝ i -2 and λ i ∝ i -3 . The results revealed that faster eigenvalue decay consistently resulted in lower excess risk compared to slower decay, indicating that it may suffer less from dimensionality with respect to utility. Additionally, in both synthetic and real data, DP-GLMtron and DP-TAGLMtron outperformed DP-SGD and DP-FTRL [22], especially as the sample size increased, highlighting the effectiveness of our proposed methods.

Due to space limitations, we have included the algorithms and proofs in the appendix.

2 Related Work

Private Nonconvex Optimization Previous research focusing on the utility of DP-SO/DP-ERM with convex loss functions predominantly employed excess population risk as the metric of utility. In contrast, for non-convex scenarios, utility assessment can be broadly categorized into three types: first-order stationary-based, second-order stationary-based, and excess population risk. The first-order stationary-based method [43, 53, 34, 8, 52, 50] involves evaluating the ℓ 2 -norm of the gradient of

the population risk function. This method, however, encounters several challenges. For example, [2] shows that the gradient norm tends to approach zero as the sample size increases indefinitely, while there exists no assurance that a differentially private estimator will converge to or even be near any significant local minimum. The Second-order stationary-based method [42, 45] employs the norm of the gradient and the Hessian matrix minimal eigenvalue of the population risk function. However, this approach is particularly suited to specific scenarios where any second-order stationary point is a local minimum of the problem, and all the local minima are the global minimum, such as matrix completion and dictionary learning. The third approach involves directly employing the excess population risk as the measure of utility [33, 43] and our work aligns with this direction. However, as we mentioned these bounds either need strong assumptions or become trivial when d ≫ N .

Private Generalized Linear Models DP-SO and DP-ERM with generalized linear loss (DP-GLL) and generalized linear models (DP-GLM) have gained significant attention in recent years. The key developments in this area can be regarded as three aspects: 1) For convex loss functions, an early study on DP-GLL [21] revealed that, assuming features have bounded l 2 norm, the error bound can reach ˜ O ( 1 √ Nε ) , contrasting with the general convex DP-ERM bound of O ( √ d Nε ) . A subsequent study [25] found that in constrained cases, the error bound depends on the Gaussian width of the constraint set. [34] improved the bound to O ( √ θ Nε ) under the assumption that the data space is a bounded set, where θ is the rank of the expectation of the data matrix. 2) For constrained DP-GLM, [8] explored various scenarios, including smooth/non-smooth losses and losses in ℓ p space for 1 ≤ p ≤ 2 . [5] provided the optimal rates of DP-GLM in unconstrained settings, identifying different rates depending on the nature of the loss function (smooth and non-negative but not necessarily Lipschitz, or Lipschitz). 3) For non-convex losses, [34, 7] provided dimension-independent bounds for the gradient ℓ 2 norm of the population risk function. As a specific instance of GLM, this work focuses on the overparameterized regime with sub-Gaussian data rather than bounded data.

3 Preliminaries

Notations : In this paper, we adhere to a consistent notation style for clarity. We use boldface lower letters such as x , w for vectors, and boldface capital letters (e.g. A , H ) for matrices. Let ∥ A ∥ 2 denote the spectral norm of A . For two matrices A and B of appropriate dimension, their inner product is defined as ⟨ A , B ⟩ := tr( A ⊤ B ) . For a positive semi-definite (PSD) matrix A and a vector v of appropriate dimension, we write ∥ v ∥ 2 A := v ⊤ Av . The outer product is denoted by ⊗ .

Given a dataset D = { ( x i , y i ) } N i =1 with each data point has a feature vector x i ∈ X ⊆ R d and a response variable y i ∈ Y . In this paper, we assume that { ( x i , y i ) } N i =1 are i.i.d. sampled from a ReLU regression model, i.e., each ( x i , y i ) is a realization of the ReLU regression model y = ReLU( x ⊤ w ∗ ) + z , where ReLU( · ) := max {· , 0 } is the Rectified Linear Unit (ReLU) ; z is a zero mean randomized noise; w ∗ ∈ R d is the optimal model parameter. Our goal is to output a model w priv that maintains privacy while minimizing the excess population risk, i.e., L ( w priv ) -L ( w ∗ ) , where

In this paper, we will focus on classic Differential Privacy to ensure privacy.

Definition 3.1 (Differential Privacy [13]) . Given a data universe X , we say that two datasets D,D ′ ⊆ X are neighbors if they differ by only one element, which is denoted as D ∼ D ′ . A randomized algorithm A is ( ε, δ ) -differentially private (DP) if for all adjacent datasets D,D ′ and for all events S in the output space of A , we have P ( A ( D ) ∈ S ) ≤ e ε · P ( A ( D ′ ) ∈ S ) + δ.

In the following, we will introduce some definitions and assumptions related to the model.

Definition 3.2 (Well-specified Condition) . Assume that there exists a parameter w ∗ ∈ R d such that E [ y | x ] = ReLU( x ⊤ w ∗ ) , and the variance of the model noise can be denoted by σ 2 := E [ ( y -ReLU( x ⊤ w ∗ )) 2 ] .

In the literature, the well-specified setting is also extensively referred to as the 'noisy teacher' setting [18] or the well-structured noise mode [19]. It has been widely studied previously [56, 41, 33, 12].

Assumption 3.3 (Data Covariance) . Define H := E [ xx ⊤ ] as the expected data covariance matrix.

Denote the eigen decomposition of the data covariance by H = ∑ i λ i v i v ⊤ i , where ( λ i ) i ≥ 1 are eigenvalues in a nonincreasing order and ( v i ) i ≥ 1 are the corresponding eigenvectors. Define H k 1 : k 2 as H k 1 : k 2 := ∑ k 1 <i ≤ k 2 λ i v i v ⊤ i , and allow k 2 = ∞ to imply that H k : ∞ = ∑ i>k λ i v i v ⊤ i .

In the case of a Gaussian distribution, the data covariance matrix reduces to an identity matrix scaled by variance.

Assumption 3.4 (Fourth moment conditions) . Define that the fourth moment of x as M :

(A) There exists a constant α > 0 such that for any Positive Semi-Definite (PSD) matrix A , the following holds:

(B) There exists a constant β > 0 , such that for every PSD matrix A , the following holds:

Remark 1. For Gaussian distribution, it can be verified that Assumption 3.4 holds with α = 3 and β = 1 [56]. Furthermore, the assumption of sub-Gaussianity in data, specifically that H -1 2 x is a sub-Gaussian random vector, as posited in previous work [44, 41, 54, 28, 55, 20, 32, 31] for private linear regression model, can be shown to imply Assumption 3.4 (A) [56]. It is important to note that Assumption 3.4 (B) is specifically utilized in our analysis for establishing the lower bound only.

Definition 3.5 ( ( H , C 2 , a, b ) -Tail) . Let both H and M exist and are finite. A random vector x satisfies ( H , C 2 , a, b ) -Tail if the the following holds:

∃ a > 0 s.t. with probability ≥ 1 -b ,

That is, for any fixed v , with probability ≥ 1 -b :

Both conditions above provide Gaussian-like tail bounds determining the resilience of the dataset. Note that Definition 3.5 is widely used in the recent literature on DP analysis for the sub-Gaussian type of data such as [41, 29, 27]. In this work, we will assume each sample x satisfying ( H , C 2 , a, b ) -Tail and the distribution of the inherent noise z satisfies ( σ 2 , C 2 , a, b ) -Tail, which could capture more statistical properties beyond expectations. Additionally, according to Assumption 3.4 (A), it holds that ∥ x ∥ 2 2 ≤ α tr( H ) · log 2 a (1 /b ) in Equation (2).

Assumption 3.6 (Symmetricity conditions) . Assume that for every u , v ∈ R d , it holds that:

✶ ✶ Remark 2. Here we require that both the second and fourth moments of x remain symmetric. It can be shown that Assumption 3.6 is satisfied when x and -x follow the same distribution, which can apply to symmetric sub-Gaussian such as symmetric Bernoulli or Gaussian distributions.

We have,

2.0

1.5

1.0

0.5

0.0

-0.5

1.0

1.5

1.05

0.60-

-2.0

1.20-

0.45

0.30

DP-SGD

DP-GLMtron

Optimal

Saddle

0.90

.15

2.0-1.5-1.0-0.50.00.5

1.0

1.5

(a) Trajectory with smaller noise

2.0

1.5

33

95

2.40

3.153.30

DP-SGD

DP-GLMtron

1.0

B

2.85

1.95

Optimal

Saddle

0.5

0.0

2

2.10

1.50

1.20

0.75

1.65

0.45

-0.5

-1.0

25

1.05

0.60

1.5

0.90

0.15

0.30

-2.0

=2.0

0-1.5-1.0-0.50.0

0.5

1.0

1.5

0.75

2.0

(b) Trajectory with larger noise

4 Proposed DP-GLMtron Algorithm

Before presenting the analysis of DP ReLU regression problem, we first introduce our novel algorithm: DP-GLMtron (Algorithm 1), which is inspired by and built upon the Generalized Linear Model Perceptron (GLM-tron) algorithm of [23].

The fundamental distinction between SGD and GLMtron lies in their respective update rules. Specifically, it takes the following rules:

where the algorithm commence from an initial point w 0 and iterate from t = 0 to t = N with stepsize η . In contrast to the update rule of SGD, GLMtron diverges by changing the derivative of the ReLU function in its update mechanism. Typically, in neural network training, the gradient of the activation function, such as ReLU, is a critical component of the backpropagation process. The exclusion of this derivative in GLMtron's framework not only simplifies the computational process but also enhances efficiency. Our motivation for developing the DP-GLMtron algorithm stems from observations regarding the limitations of DP-SGD, as it sometimes fails to reach the optimal solution and can struggle with convergence under conditions of high noise, often settling at a saddle point instead (as illustrated in Figure 4). This issue is particularly pronounced when considering a low privacy budget. Furthermore, [23] demonstrates that this specific omission plays a significant role in GLMtron's ability to efficiently identify a predictor that closely approximates the optimal solution.

Building upon these foundations, we now present the detailed implementation of our proposed DP-GLMtron. For Algorithm 1, we begin with a random permutation of the dataset, ensuring that each data point is treated independently and uniformly. Also, such shuffling can amplify privacy[17]. Subsequently, the algorithm performs a single pass over the dataset. Throughout this pass, we need to determine the clipping threshold before undergoing iterative updates for w . Establishing an appropriate clipping threshold is crucial for balancing the trade-off between utility and privacy. If the clipping threshold is set excessively low, it results in loss of gradient information, consequently leading to high bias [30, 3]. Consequently, we propose an advanced Algorithm 2 based on the residual estimator in [28] to achieve an adaptive clipping [4] by utilizing a small batch of data points.

Recall that the clipping term is a product of the residual, (ReLU( x ⊤ t w t ) -y t ) , and the covariate, x ⊤ t . Based on Definition 3.5 and Assumption 3.4, we understand that ∥ x t ∥ 2 2 ≤ α tr( H ) · log 2 a (1 /b ) with a probability of at least 1 -b . Thus, it is sufficient to get an upper bound of | ReLU( x ⊤ t w t ) -y t | . To bound this, we consider three cases, 1 ) x ⊤ t w t > 0 , x ⊤ t w ∗ > 0 ; 2 ) x ⊤ t w t < 0 , x ⊤ t w ∗ > 0 ; 3 ) x ⊤ t w t > 0 , x ⊤ t w ∗ < 0 , corresponding to d 1 t , d 2 t , and d 3 t in step 2 of Algorithm 2. It is noticed that the inherent model noise is not considered when initially calculating the residual for the third case at step 2, while this noise is taken into account at step 15. Moreover, when considering a special case that x ⊤ t w t < 0 , x ⊤ t w ∗ < 0 , the clipping threshold is integrated into the third scenario. For each residual, we first partition the dataset into K batches, compute an estimate for each batch, and form a histogram with over those K estimates. We then apply a private histogram mechanism with geometrically increasing bin sizes. The bin with the most estimates is used, ensuring a constant factor

4898

33645

3.00

2.55

1.80-

1.35

2.85

2.40

75

1.50

2.0

approximation of the distance to the optimum. Finally, we take the maximum among the three private residual estimates.

Upon determining the clipping threshold γ t , it is subsequently incorporated into the existing clipping list γ . Following this, Gaussian noise with a standard deviation of 2 fγ t is introduced to the clipped l t and then the weight vector w t +1 will be updated. Finally, the algorithm culminates by computing the average of the iterates.

Theorem 4.1 (Privacy Guarantee) . Suppose DP-GLMtron is applied to N input samples with a noise multiplier set to f = O ( log( N/δ ) ε √ N ) . Under these conditions, the algorithm satisfies ( ε, δ ) -differential √

Remark 3. For general data distributions, DP-GLMtron necessitates a more complex approach to clipping decisions. To this end, we incorporate Algorithm 2, which uses the magnitude of the residual along with privacy parameters ε ′ and δ ′ , as a means to estimate the private threshold. Additionally, we take only one pass over random shuffling data for Algorithm 1 [15, 17, 41], which is a wellknown method of privacy amplification. Hence, we need to add noise O ( γ t ε √ N ) , where γ t is the √

clipping threshold and ε must be restricted to 1 / N , to satisfy the requirement of [17] and to ensure differential privacy.

Theorem 4.2 (Utility Guarantee Upper) . Consider a dataset D = { ( x i , y i ) } N i =1 , where each x i is drawn from a distribution D satisfying ( H , C 2 , a, b ) -Tail and the distribution of the inherent noise z satisfies ( σ 2 , C 2 , a, b ) -Tail. Given Assumptions 3.4 and 3.6, if we set a step size η ≤ 1 2 α tr( H ) log 2 a ( N ) , a noise multiplier f = O ( log( N/δ ) ε √ N ) , estimating data size m = Ω( log( N ) log( N/δ ′ ) ε ′ ) , then with a probability of 1 -1 /N , the output of Algorithm 1 will satisfy:

where the bias, variance, and private terms are upper bounded by

where the effective dimensions are given by

with k ∗ = max { k : λ k ≥ 2 ηN } and Γ = max { γ t } T -1 t =0 .

Remark 4. The utility bound is typically decomposed into three terms: the bias term, which is associated with the initial value of the model; the variance term, stemming from the inherent noise intrinsic to the model itself; and the privacy term, introduced by the privacy-preserving mechanism. Notably, the results contain two crucial factors: the effective dimension D eff and the private effective dimension D pri eff , which can be regarded as the function of data covariance matrix's eigenvalues.

Technical Understanding Under Simplified Cases To provide a more intuitive explanation, we explore two simplified scenarios. 1) ∥ x ∥ 2 ≤ 1 : In this scenario, where the norm of the data is constrained, the trace of the covariance matrix tr( H ) ≤ 1 . This condition indicates that the sum of the eigenvalues, λ i , is less than 1. Given that the eigenvalues are ordered non-increasingly (refer to Assumption 3.3), the majority of these eigenvalues must be very small. Assuming the step size η is sufficiently small and scaled with the square root of the data size, √ N , the optimal number of iterations k ∗ must be confined within a scope proportional to √ N . If not, the inequality tr( H ) ≥ k ∗ · 2 Nη ≥ 1 would contradict the data assumptions. Furthermore, the clipping threshold, which depends on tr( H ) , remains a constant term, and the private effective dimension D pri eff is bounded above by N . Therefore, the total utility is ˜ O ( 1 √ N + tr( H -1 ) N 2 ε 2 ) . 2) sub-Gaussian tail: In this case, the eigenvalue of the covariance matrix corresponds to the variance of sub-Gaussian data. Both the effective dimension, D eff, and the private effective dimension, D pri eff , effectively reduce to the feature dimension d . Moreover, the clipping threshold contributes a factor of d , leading to a total utility bound of ˜ O ( d √ N + d 2 N 2 ε 2 ) .

The Role of Eigenvalue in High-dimensional Setting Existing work in the DP community primarily relies on data assumptions satisfying Gaussian-like distributions, as discussed in Remark 1, which reduces the data covariance matrix to an identity matrix scaled by variance. Such an assumption may cause the case to be overlooked when the spectrum exhibits decay. Theorem 4.2 offers a novel perspective by elucidating the interplay between the feature space and the utility bound. Notice the crucial cut-off index k ∗ = max { k : λ k ≥ 2 ηN } , which separates the entire space into a k ∗ dimensional subspace. It enables us to obtain a bound that does not directly depend on the feature dimension d in the overparameterized regime. To achieve a diminishing bound, it is necessary for k ∗ to be o ( N ) and for the tail summation of eigenvalues to be o (1 /N ) , which indicates that it is possible to achieve improved results without suffering the dimension d in the overparameterized regime.

To better understand the effective dimension, we offer examples of data spectra where the excess risk diminishes under certain conditions.

Corollary 4.3. Given the same assumptions in Theorem 4.2 and suppose ∥ w 0 -w ∗ ∥ 2 is bounded, it holds that:

If the spectrum of H satisfies λ k = k -r for some r > 1 , then with probability at least 1 -1 /N , the utility is upper bounded by ˜ O ( N -r 1+ r (1 + ( ε -2 N r 1+ r ) r 1+2 r )) .

If the spectrum of H satisfies λ k = e -k , then with probability at least 1 -1 /N , the utility is upper bounded by ˜ O ( N -1 (1 + ( ε -2 N ) 1 2 )) .

Remark 5. Corollary 4.3 indicates that utility bound in the overparameterized setting can still vanish if the spectrum exhibits either polynomial or exponential decay. Specifically, a constant privacy budget ε = O (1) can yield utility bounds of ˜ O ( N -r/ (1+2 r ) ) for polynomial decay and ˜ O ( N -1 / 2 ) for exponential decay. Notably, faster eigenvalue decay (e.g. r = 2 , 3 ) consistently resulted in lower excess risk compared to slower decay, indicating that it may suffer less from dimensionality with respect to utility. This insight cannot be derived when considering only traditional Gaussian-like data. Our experiment in Section 6 also validates the conclusion. Moreover, increasing the privacy budget to ε = O ( N r/ (2+2 r ) ) for polynomial decay and ε = O ( N 1 / 2 ) for exponential decay improves utility bounds to ˜ O ( N -r/ (1+ r ) ) and ˜ O ( N -1 ) , respectively, matching non-private and linear regression scenario results [49, 48].

In the following, we show that our previous analysis for Algorithm 1 is almost tight.

Theorem 4.4 (Utility Guarantee Lower) . Consider a dataset D = { ( x i , y i ) } N i =1 , where each x i is drawn from a distribution D satisfying ( H , C 2 , a, b ) -Tail and the distribution of the inherent noise z satisfies ( σ 2 , C 2 , a, b ) -Tail. Given Assumptions 3.4 and 3.6, if we set the same parameters as in Theorem 4.2, then with a probability of 1 -1 /N , the output of Algorithm 1 will satisfy:

where

where the effective dimensions, k ∗ m are defined same as in Theorem 4.2 and denotes ˜ Γ = min { γ t } T -1 t =0 .

Remark 6. Similar to the Theorem 4.2, the lower bound stated here consists of three terms: the bias term, the variance term, and the privacy term. It is noticed that Theorem 4.4 provides the algorithmic lower bound Algorithm 1, applicable in both under-parameterized and over-parameterized settings. Notably, our lower bound aligns with the upper bound, differing only by absolute constants.

5 Advanced DP-TAGLMtron

A pivotal concern with Algorithm 1 DP-GLMtron lies in its primary applicability in the setting where the privacy budget ε is small, which could potentially limit its practical utility (see Remark 3 for more details). To address this challenge, we introduce DP-TAGLMtron (Algorithm 3) in this section, which is designed to attain an improved trade-off between privacy and utility.

In Algorithm 3, the process starts with setting the initial weights. For each iteration t , a private residual is determined by Algorithm 2 on the estimating samples, current weights, and privacy parameters. Subsequently, the clipping bound is computed as the product of the private residual and the covariate, analogous to the previous case. Once this clipping bound γ t for the current iteration is computed, it is added to the existing clipping list γ . With γ t in place, the clipped "gradient" ( l t ) is calculated. The maximum value from this list is selected as the clipping threshold Γ for our next step. The "gradient" l t is then forwarded to the differentially private tree-aggregation mechanism with noise multiplier f and the maximal clipping threshold Γ .

It is noticed that our proposed algorithm, DP-TAGLMtron, does not depend on privacy amplification. Instead, it primarily involves employing tree aggregation [14, 11, 22, 40] to introduce noise into the sum of mini-batch gradients. At the beginning of training, we construct a binary tree consisting of (2 ⌈ log 2 N ⌉ +1 -1) nodes, corresponding to N leaves. Each leaf node is assigned a label l i , and the value of each internal node represents the sum of its child nodes. In each iteration, the tree receives a vector l t , derived from the current data pair ( x t , y t ) , which is used to update the tree, incorporating the new node. The final output, denoted as l ′ ≤ t , is obtained by aggregating the gradients accumulated in the binary tree and adding Gaussian noise to this aggregate. As a result, we can understand that l ′ ≤ t is the private estimation of ∑ t i =0 l i generated by tree aggregation protocol as follows:

and v t is a combination of at most O ( ⌈ log 2 t ⌉ ) Gaussian. Therefore, each l i influences only its ancestor nodes, affecting at most ⌈ log t ⌉ +1 nodes. By introducing Gaussian noise with a standard deviation of O (log t/ε ) to each node, the entire tree achieves ( ε, δ ) -differential privacy.

When receiving the output of Algorithm 4, the optimization objective at step 9 is to determine the weight parameter w . This objective function is composed of two key components: the first term, ( l ′ ≤ t , w ) , represents the inner product of l ′ ≤ t and the weight parameter w . The second term, 1 2 η ∥ w ∥ 2 2 , introduces a regularization mechanism, where η controls the balance between fitting the training data and constraining the magnitude of the parameters to prevent overfitting. One common method to find the optimal solution for this optimization problem is to take the derivative of the objective function and set it to zero. Therefore, for the given problem at step 9, we have the following update:

It should be noted that in Algorithm 1, if the initialization is set to w 0 = 0 , then the update rule of w can be considered equivalent to the one presented here, provided that the variance of the noise and the clipping threshold are disregarded. Upon completing all iterations, the final model weights are determined by averaging all the updated weights from each iteration.

Theorem 5.1 (Privacy Guarantee) . Suppose DP-GLMtron is applied to N input samples with a noise multiplier set to f = √ 2 ⌈ lg( N +1) ⌉ ln(1 /δ ) ε . Under these conditions, the Algorithm 3 satisfies ( ε, δ ) -differential privacy, with ε ′ = ε √ 8 N log(2 /δ ) and δ ′ = δ 2 N in Algorithm 2.

(

Remark 7. Differing from Algorithm 1, the update rule of DP-TAGLMtron mainly utilizes the DP-Tree-Aggregation mechanism, which involves aggregating the "gradients" collected in a binary tree and then adding Gaussian noise to this aggregate. Based on [22], Algorithm 4 achieves ε, δ ) -differential privacy with appropriate noise multiplier f = √ 2 ⌈ lg( N +1) ⌉ ln(1 /δ ) ε , ensuring that Algorithm 3 also upholds ( ε, δ ) -DP under similar settings with ε ′ , δ ′ as Algorithm 2.

Theorem 5.2 (Utility Guarantee) . Consider a dataset D = { ( x i , y i ) } N i =1 , where each x i is drawn from a distribution D satisfying ( H , C 2 , a, b ) -Tail and the distribution of the inherent noise z satisfies ( σ 2 , C 2 , a, b ) -Tail. Given Assumptions 3.4 and 3.6 hold, if we set a stepsize η < 1 / ( α tr( H )) , a noise multiplier f = √ 2 ⌈ lg( N +1) ⌉ ln(1 /δ ) ε , then with a probability of 1 -1 /N , the output of Algorithm 3 will satisfy:

where

where the effective dimensions and k ∗ are defined same as in Theorem 4.2.

Remark 8. It is noteworthy that in Theorem 5.2, both the bias term and the variance term are consistent with those in Theorem 4.2. This alignment extends to the non-private results, with the only differences being in the absolute constants. Additionally, the private term in Theorem 5.2 closely aligns with that in Theorem 4.2, with the primary difference being a factor of log 2 N . It is intuitive that the tree aggregation mechanism utilized in DP-TAGLMtron indicates that the noise added for privacy is derived from a combination of at most ⌈ log 2 N +1 ⌉ and at least one random Gaussian vector. This aspect suggests that DP-TAGLMtron diverges from the upper bound of DP-GLMtron at most by a factor of log 2 N and adheres to the same lower bound as DP-GLMtron with a different noise variance. Moreover, in contrast to Algorithm 1, DP-TAGLMtron does not necessitate ε = O (1 / √ N ) to maintain differential privacy, allowing a broader range of privacy budget settings in terms of flexibility and applicability.

6 Empirical Simulation

In this section, we first conduct experiments with synthetic data to validate the theoretical insights related to the role of eigenvalues in a high-dimensional setting. Additionally, due to space constraints, we present a real-data experiment on the MNIST dataset in Appendix B to demonstrate the performance of our proposed method.

100

200

300

400

500

Epoch

0.5

1.0

1.5

2.0

Excess Risk

i

i

2

(

=0.05)

DP-SGD

DP-GLMtron

DP-TAGLMtron

DP-FTRL

(a)

100

200

300

400

500

Epoch

0.4

0.6

0.8

1.0

1.2

1.4

Excess Risk

i

i

2

(

=0.2)

DP-SGD

DP-GLMtron

DP-TAGLMtron

DP-FTRL

(b)

100

200

300

400

500

Epoch

0.4

0.6

0.8

1.0

1.2

1.4

Excess Risk

i

i

2

(

=0.5)

DP-SGD

DP-GLMtron

DP-TAGLMtron

DP-FTRL

(c)

100

200

300

400

500

Epoch

0.4

0.6

0.8

1.0

Excess Risk

i

i

3

(

=0.05)

DP-SGD

DP-GLMtron

DP-TAGLMtron

DP-FTRL

(d)

100

200

300

400

500

Epoch

0.4

0.6

0.8

1.0

Excess Risk

i

i

3

(

=0.2)

DP-SGD

DP-GLMtron

DP-TAGLMtron

DP-FTRL

(e)

100

200

300

400

500

Epoch

0.4

0.6

0.8

Excess Risk

i

i

3

(

=0.5)

DP-SGD

DP-GLMtron

DP-TAGLMtron

DP-FTRL

(f)

Experiment Setup. The experiments were designed with varying privacy budgets ( ε ) set at 0.05, 0.2, and 0.5 with δ = 1 n 1 . 1 to observe the impact of differential privacy constraints on the learning process. We generated the data with two eigenvalue decay scenarios, λ i ∝ i -2 , i -3 , simulating different distributions of feature importance. The dimensionality of the data was fixed at 1,024 to maintain consistency across trials. The learning rate was initially set to 10 -2 , with N representing

the sample size, which varied from 50 to 550 and increased in steps to 100. We utilized a ReLU regression model with noise to be trained using the previously mentioned algorithms, integrating privacy-preserving noise adjustments. The algorithms underwent a single iteration over generated data. The primary metric for evaluation was the average excess risk across varying dataset sizes and ε values. For each experiment, we also consider DP-SGD [1] and DP-FTRL [22] as baselines.

Empirical Findings. Compared with Figures (2a-2c), which illustrate data with eigenvalue decay λ i ∝ i -2 , Figures (2f-2e), which illustrate data with eigenvalue decay λ i ∝ i -3 exhibit the lower excess risk under the same training algorithms and conditions regardless of privacy budget. This indicates that fast eigenvalue decay suffers less from the dimensionality with respect to utility, thereby validating our results in Section 4. Additionally, results in both data with eigenvalue decay λ i ∝ i -2 and data with eigenvalue decay λ i ∝ i -3 show that DP-GLMtron and DP-TAGLMtron consistently outperform DP-SGD and DP-FTRL regarding excess risk, particularly as the sample size increased. Moreover, it is noticed that the DP-SGD and DP-FTRL exhibit poor convergence performance even with comparatively large data sizes. Finally, in both figures, increasing the privacy budget from ε = 0 . 05 to ε = 0 . 5 resulted in lower excess risks for both algorithms. This observation confirms the anticipated trade-off between privacy and learning performance, with a looser privacy constraint leading to more accurate models.

7 Conclusions

In conclusion, our study significantly advances the field of differential privacy in ReLU regression problems. Our first proposed algorithm, DP-GLMtron, provides an excess population risk upper bound of ˜ O ( D eff N + D pri eff N 2 ε 2 ) , offering a novel perspective on utility decomposition and highlighting the impact of eigenvalues in overparameterized settings. Our results apply not only to Gaussian-like data, typically considered in traditional differential privacy studies, but also to data distributions with polynomial or exponential decay. To address limitations with small privacy budgets, we developed DP-TAGLMtron, which offers performance comparable to DP-GLMtron but supports broader privacy budgets. Empirical results from both synthetic and real data experiments demonstrate that DPGLMtron and DP-TAGLMtron consistently outperform DP-SGD and DP-FTRL.

Acknowledgments

The research of Meng Ding and Jinhui Xu was supported in part by KAUST through grant CRG104663.2. Di Wang was supported in part by the baseline funding BAS/1/1689-01-01, funding from the CRG grand URF/1/4663-01-01, REI/1/5232-01-01, REI/1/5332-01-01, FCC/1/1976-49-01 from CBRC of King Abdullah University of Science and Technology (KAUST). Di Wang was also supported by the funding RGC/3/4816-09-01 of the SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence (SDAIA-KAUST AI).

Impact Statements

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

Limitations

While this study advances previous work by relaxing certain assumptions and providing both synthetic and real-data validations of the theoretical findings, the analysis is specifically confined to the ReLU regression problem. Consequently, it remains uncertain whether these conclusions can be generalized to other nonconvex optimization scenarios. Future research could explore whether similar patterns hold across a broader range of nonconvex problems.

References

Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pages 308-318, 2016.

Naman Agarwal, Zeyuan Allen-Zhu, Brian Bullins, Elad Hazan, and Tengyu Ma. Finding approximate local minima faster than gradient descent. In Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing , pages 1195-1199, 2017.

Kareem Amin, Alex Kulesza, Andres Munoz, and Sergei Vassilvtiskii. Bounding user contributions: A bias-variance trade-off in differential privacy. In International Conference on Machine Learning , pages 263-271. PMLR, 2019.

Galen Andrew, Om Thakkar, Brendan McMahan, and Swaroop Ramaswamy. Differentially private learning with adaptive clipping. Advances in Neural Information Processing Systems , 34:17455-17466, 2021.

Raman Arora, Raef Bassily, Cristóbal Guzmán, Michael Menart, and Enayat Ullah. Differentially private generalized linear models revisited. Advances in Neural Information Processing Systems , 35:22505-22517, 2022.

Hilal Asi, John Duchi, Alireza Fallah, Omid Javidbakht, and Kunal Talwar. Private adaptive gradient methods for convex optimization. In International Conference on Machine Learning , pages 383-392. PMLR, 2021.

Raef Bassily, Vitaly Feldman, Kunal Talwar, and Abhradeep Guha Thakurta. Private stochastic convex optimization with optimal rates. Advances in neural information processing systems , 32, 2019.

Raef Bassily, Cristóbal Guzmán, and Michael Menart. Differentially private stochastic optimization: New results in convex and non-convex settings. Advances in Neural Information Processing Systems , 34:9317-9329, 2021.

Raef Bassily, Cristóbal Guzmán, and Anupama Nandi. Non-euclidean differentially private stochastic convex optimization. In Conference on Learning Theory , pages 474-499. PMLR, 2021.

Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th annual symposium on foundations of computer science , pages 464-473. IEEE, 2014.

T-H Hubert Chan, Elaine Shi, and Dawn Song. Private and continual release of statistics. ACM Transactions on Information and System Security (TISSEC) , 14(3):1-24, 2011.

Meng Ding, Kaiyi Ji, Di Wang, and Jinhui Xu. Understanding forgetting in continual learning with linear regression. arXiv preprint arXiv:2405.17583 , 2024.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3 , pages 265-284. Springer, 2006.

Cynthia Dwork, Moni Naor, Toniann Pitassi, and Guy N Rothblum. Differential privacy under continual observation. In Proceedings of the forty-second ACM symposium on Theory of computing , pages 715-724, 2010.

Úlfar Erlingsson, Ilya Mironov, Ananth Raghunathan, and Shuang Song. That which we call private. arXiv preprint arXiv:1908.03566 , 2019.

Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimization: optimal rates in linear time. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing , pages 439-449, 2020.

Vitaly Feldman, Audra McMillan, and Kunal Talwar. Hiding among the clones: A simple and nearly optimal analysis of privacy amplification by shuffling. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS) , pages 954-964. IEEE, 2022.

Spencer Frei, Yuan Cao, and Quanquan Gu. Agnostic learning of a single neuron with gradient descent. Advances in Neural Information Processing Systems , 33:5417-5428, 2020.

Surbhi Goel and Adam R Klivans. Learning neural networks with two nonlinear layers in polynomial time. In Conference on Learning Theory , pages 1470-1499. PMLR, 2019.

Lijie Hu, Shuo Ni, Hanshen Xiao, and Di Wang. High dimensional differentially private stochastic optimization with heavy-tailed data. In Proceedings of the 41st ACM SIGMODSIGACT-SIGAI Symposium on Principles of Database Systems , pages 227-236, 2022.

Prateek Jain and Abhradeep Guha Thakurta. (near) dimension independent risk bounds for differentially private learning. In International Conference on Machine Learning , pages 476484. PMLR, 2014.

Peter Kairouz, Brendan McMahan, Shuang Song, Om Thakkar, Abhradeep Thakurta, and Zheng Xu. Practical and private (deep) learning without sampling or shuffling. In International Conference on Machine Learning , pages 5213-5225. PMLR, 2021.

Sham M Kakade, Varun Kanade, Ohad Shamir, and Adam Kalai. Efficient learning of generalized linear and single index models with isotonic regression. Advances in Neural Information Processing Systems , 24, 2011.

Vishesh Karwa and Salil Vadhan. Finite sample differentially private confidence intervals. arXiv preprint arXiv:1711.03908 , 2017.

Shiva Prasad Kasiviswanathan and Hongxia Jin. Efficient private empirical risk minimization for high-dimensional learning. In International Conference on Machine Learning , pages 488-497. PMLR, 2016.

Janardhan Kulkarni, Yin Tat Lee, and Daogao Liu. Private non-smooth empirical risk minimization and stochastic convex optimization in subquadratic steps. arXiv preprint arXiv:2103.15352 , 2021.

Xiyang Liu, Prateek Jain, Weihao Kong, Sewoong Oh, and Arun Suggala. Label robust and differentially private linear regression: Computational and statistical efficiency. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Xiyang Liu, Prateek Jain, Weihao Kong, Sewoong Oh, and Arun Sai Suggala. Near optimal private and robust linear regression. arXiv preprint arXiv:2301.13273 , 2023.

Xiyang Liu, Weihao Kong, Prateek Jain, and Sewoong Oh. Dp-pca: Statistically optimal and differentially private pca. Advances in Neural Information Processing Systems , 35:29929-29943, 2022.

H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. arXiv preprint arXiv:1710.06963 , 2017.

Yuan Qiu, Jinyan Liu, and Di Wang. Truthful generalized linear models. In Kristoffer Arnsfelt Hansen, Tracy Xiao Liu, and Azarakhsh Malekian, editors, Web and Internet Economics - 18th International Conference, WINE 2022, Troy, NY, USA, December 12-15, 2022, Proceedings , volume 13778 of Lecture Notes in Computer Science , pages 369-370. Springer, 2022.

Yuan Qiu, Jinyan Liu, and Di Wang. Truthful and privacy-preserving generalized linear models. Information and Computation , 301:105225, 2024.

Hanpu Shen, Cheng-Long Wang, Zihang Xiang, Yiming Ying, and Di Wang. Differentially private non-convex learning for multi-layer neural networks. arXiv preprint arXiv:2310.08425 , 2023.

Shuang Song, Thomas Steinke, Om Thakkar, and Abhradeep Thakurta. Evading the curse of dimensionality in unconstrained private glms. In International Conference on Artificial Intelligence and Statistics , pages 2638-2646. PMLR, 2021.

Shuang Song, Om Thakkar, and Abhradeep Thakurta. Characterizing private clipped gradient descent on convex generalized linear problems. arXiv preprint arXiv:2006.06783 , 2020.

Jinyan Su, Lijie Hu, and Di Wang. Faster rates of differentially private stochastic convex optimization. Journal of Machine Learning Research , 25(114):1-41, 2024.

Jinyan Su and Di Wang. Faster rates of differentially private stochastic convex optimization. arXiv preprint arXiv , 2108, 2021.

Jinyan Su, Changhong Zhao, and Di Wang. Differentially private stochastic convex optimization in (non)-euclidean space revisited. In Uncertainty in Artificial Intelligence , pages 2026-2035. PMLR, 2023.

Youming Tao, Yulian Wu, Xiuzhen Cheng, and Di Wang. Private stochastic convex optimization and sparse learning with heavy-tailed data revisited. In IJCAI , pages 3947-3953. ijcai.org, 2022.

Hoang Tran and Ashok Cutkosky. Momentum aggregation for private non-convex erm. Advances in Neural Information Processing Systems , 35:10996-11008, 2022.

Prateek Varshney, Abhradeep Thakurta, and Prateek Jain. (nearly) optimal private linear regression via adaptive clipping. arXiv preprint arXiv:2207.04686 , 2022.

Di Wang, Changyou Chen, and Jinhui Xu. Differentially private empirical risk minimization with non-convex loss functions. In International Conference on Machine Learning , pages 6526-6535. PMLR, 2019.

Di Wang and Jinhui Xu. Differentially private empirical risk minimization with smooth nonconvex loss functions: A non-stationary view. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 1182-1189, 2019.

Di Wang and Jinhui Xu. On sparse linear regression in the local differential privacy model. In International Conference on Machine Learning , pages 6628-6637. PMLR, 2019.

Di Wang and Jinhui Xu. Escaping saddle points of empirical risk privately and scalably via dptrust region method. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2020, Ghent, Belgium, September 14-18, 2020, Proceedings, Part III , pages 90-106. Springer, 2021.

Di Wang, Minwei Ye, and Jinhui Xu. Differentially private empirical risk minimization revisited: Faster and more general. Advances in Neural Information Processing Systems , 30, 2017.

Lingxiao Wang, Bargav Jayaraman, David Evans, and Quanquan Gu. Efficient privacypreserving stochastic nonconvex optimization. In Uncertainty in Artificial Intelligence , pages 2203-2213. PMLR, 2023.

Lingxiao Wang, Difan Zou, Kumar Kshitij Patel, Jingfeng Wu, and Nathan Srebro. Private overparameterized linear regression without suffering in high dimensions, 2024.

Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Sham M Kakade. Finite-sample analysis of learning high-dimensional single relu neuron. 2023.

Hanshen Xiao, Zihang Xiang, Di Wang, and Srinivas Devadas. A theory to instruct differentiallyprivate learning via clipping bias reduction. In 2023 IEEE Symposium on Security and Privacy (SP) , pages 2170-2189. IEEE Computer Society, 2023.

Jiaqi Zhang, Kai Zheng, Wenlong Mou, and Liwei Wang. Efficient private erm for smooth objectives. arXiv preprint arXiv:1703.09947 , 2017.

Qiuchen Zhang, Jing Ma, Jian Lou, and Li Xiong. Private stochastic non-convex optimization with improved utility rates. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence , 2021.

Yingxue Zhou, Zhiwei Steven Wu, and Arindam Banerjee. Bypassing the ambient dimension: Private sgd with gradient subspace identification. arXiv preprint arXiv:2007.03813 , 2020.

Liyang Zhu, Meng Ding, Vaneet Aggarwal, Jinhui Xu, and Di Wang. Improved analysis of sparse linear regression in local differential privacy model. arXiv preprint arXiv:2310.07367 , 2023.

Liyang Zhu, Amina Manseur, Meng Ding, Jinyan Liu, Jinhui Xu, and Di Wang. Truthful high dimensional sparse linear regression. arXiv preprint arXiv:2410.13046 , 2024.

Difan Zou, Jingfeng Wu, Vladimir Braverman, Quanquan Gu, and Sham Kakade. Benign overfitting of constant-stepsize sgd for linear regression. In Conference on Learning Theory , pages 4633-4635. PMLR, 2021.

A Algorithms

Algorithm 1 DP-GLMtron

1: Input : Training Samples: { ( x i , y i ) } N -1 i =0 , Estimating Samples: { ( x ′ i , y ′ i ) } i m =1 , Learning Rate: η , DP Noise Multiplier: f , Expected x Norm: √ α tr( H ) , Privacy Parameters: ε ′ , δ ′ , Clipping list γ = {}

2: Randomly permute { ( x i , y i ) } N -1 i =0

3: Initialize w 0 → 0

4: for t = 0 , . . . , N -1 do

5: s t ← DP-Threshold ( { ( x ′ i , y ′ i ) } i m =1 , w t , ε ′ , δ ′ )

6: γ t = √ 2 α tr( H ) C 2 log 2 a N · s t

7: Sample g t ∼ N (0 , I d × d )

8: w t +1 ← w t -η ( l t +2 fγ t g t )

9: where l t = clip γ t ( x ⊤ t (ReLU( x ⊤ t w t ) -y t ))

11: return w ← 1 N ∑ N -1 t =0 w t

10: end for

Algorithm 2 DP-Threshold

1: Input : Estimating Samples: { ( x ′ i , y ′ i ) } i m =1 , Privacy Parameters: ε ′ , δ ′ , Model w , Distribution Parameters: a, b z in Definition 3.5

3: Partition { d 1 i } i m =1 , { d 2 i } i m =1 and { d 3 i } i m =1 into three K = ⌈ C 1 log( N/ ( δ ′ )) /ε ′ ⌉ subsets of equal size separately, namely { D 1 j } K j =1 , { D 2 j } K j =1 and { D 3 j } K j =1

2: ∀ i ∈ [ m ] , d 1 i ← ( x ⊤ i w -y i ) 2 , d 2 i ← y 2 i , d 3 i ← ( x ⊤ i w ) 2 , ∀ i ∈ [ m ]

4: for p=1,2,3 do

5: ( d p j ) ′ ← ∑ i ∈ D p j d p i / | D p j | , ∀ j ∈ [ K ]

6: Partition [0 , ∞ ) into geometrically increasing intervals Ω p := { . . . , [2 -1 , 1) , [1 , 2) , [2 , 2 2 ) , . . . } ∪ { [0 , 0] }

7: Compute d p k ← ∑ j ∈ D p k d p j ( d p j ≤ ( d p k ) ′ ) / | D p k | , where D p k is the interval in Ω p

11: ˜ d p k ← 0

10: else

12: end if

14: if p=3 then

13: Let [ s p 1 , s p 2 ] be the nonempty interval containing the largest ˜ d p k

16: else

17: set s p = √ 2 s p 2

18: end if

19: end for

20: return s = max { s p } 3 p =1

Algorithm 3 DP-TAGLMtron

1: Input : Training Samples: { ( x i , y i ) } N -1 i =0 , Estimating Samples: { ( x ′ i , y ′ i ) } i m =1 , Learning Rate: η , DP Noise Multiplier: f , Expected x Norm: √ α tr( H ) , Privacy Parameters: ε ′ , δ ′ , Clipping list γ = {}

2: Initialize w 0 ← 0

3: for t = 0 . . . N -1 do

4: s t ← DP-Threshold ( { ( x ′ i , y ′ i ) } i m =1 , w t , ε ′ , δ ′ )

5: γ t ← √ 2 α tr( H ) C 2 log 2 a N · s t

6: add γ t to the list γ and Γ = max γ

8: l ′ ≤ t ← DP-Tree-Aggregation ( l ≤ t , f, Γ)

7: l t ← clip γ t ( x t (ReLU( x ⊤ t w t ) -y t ))

9: Update w t +1 ← argmin w ⟨ w , l ′ t ⟩ + 1 2 η ∥ w ∥ 2 2

10: end for

11: return w N ← 1 N ∑ N -1 t =0 w t

Algorithm 4 DP-Tree-Aggregation

1: Input : Vectors: { l i } t -1 i =0 , Noise Paramters: f, Γ

2: Initialization Create a binary tree T of size 2 ⌈ log 2 N ⌉ +1 -1 with N leaf nodes, and each node is sampled i.i.d. from N (0 , f 2 Γ 2 I d × d ) .

3: At t -th iteration, do:

4: Accept l t and add it to all the nodes along the path to the root of T .

5: Let [ e 1 , . . . , e h ] be the list of nodes from the root of T to the t -th leaf node, with node e 1 being the root node and node e h being the leaf node, where h is the height of tree.

6: Let l ′ ≤ t = 0 d and { b 1 , . . . , b h } be the h bit binary representation of t .

7: for j ∈ [ h ] do

8: if b j = 1 then

9: l ′ ≤ t ← l ′ ≤ t + e j , where e j is the value in left sibling of e j node.

10: end if

11: end for

12: return l ′ ≤ t

B Additional Experiment

0

1000

2000

3000

4000

Epoch

20

25

30

35

Excess Risk

=0.05

DP-SGD

DP-GLMtron

DP-TAGLMtron

DP-FTRL

(a)

0

1000

2000

3000

4000

Epoch

20

22

24

26

Excess Risk

=0.2

DP-SGD

DP-GLMtron

DP-TAGLMtron

DP-FTRL

(b)

0

1000

2000

3000

4000

Epoch

20

22

24

Excess Risk

=0.5

DP-SGD

DP-GLMtron

DP-TAGLMtron

DP-FTRL

(c)

In our second experiment, we evaluate the performance of four different differential privacy (DP) algorithms-DP-SGD, DP-GLMtron, DP-TAGLMtron, and DP-FTRL-using the MNIST dataset across varying sample sizes and privacy budgets denoted by ε , with values set at 0.05, 0.2, and 0.5. The learning rate was initially set to 0 . 05 , with N representing the sample size, which varied from 0 to 1000 and increased in steps to 100. We utilized a ReLU regression model with noise to be trained using the previously mentioned algorithms, integrating privacy-preserving noise adjustments. The primary metric for evaluation was the average excess risk across varying dataset sizes and ε values.

The key insights from our findings are: Under Tight Privacy Constraints ( ε = 0 . 05 ): DP-TAGLMtron demonstrated superior performance across all sample sizes, achieving the lowest excess risk, while DP-GLMtron's excess risk increased with larger sample sizes. Under Moderate Privacy Constraints ( ε = 0 . 2 ): The performance differential between our DP-GLMtron and the baseline narrowed, and DP-TAGLMtron continued to exhibit lower excess risks. DP-SGD's performance suffered with larger sample sizes, highlighting its sensitivity to sample size under moderate privacy levels. Under Relaxed Privacy Constraints ( ε = 0 . 5 ): Our proposed methods markedly outperformed both DP-SGD and DP-FTRL, showcasing their enhanced risk mitigation capabilities when privacy constraints are loosened. These insights demonstrate the effectiveness of our proposed algorithms, DP-TAGLMtron and DP-GLMtron, in optimizing the balance between privacy protection and model accuracy.

200

400

Epoch

1.00

1.20

1.40

1.60

1.80

Excess Risk

DP-SGD

=0.05

=0.20

=0.50

200

400

Epoch

0.40

0.60

0.80

1.00

Excess Risk

DP-GLMtron

=0.05

=0.20

=0.50

200

400

Epoch

0.93

0.95

0.98

1.00

1.02

1.05

Excess Risk

DP-FTRL

=0.05

=0.20

=0.50

200

400

Epoch

0.40

0.50

0.60

Excess Risk

DP-TAGLMtron

=0.05

=0.20

=0.50

Sensitivity of Privacy Budgets. Here, we also included additional experimental results related to the performance of different algorithms under various privacy budgets. we demonstrate the sensitivity of the privacy budget for each algorithm, and it is clear that DP-SGD faces the same issue. When ε is small, the tree mechanism in DP-FTRL allows it to add less noise compared to the naive Gradient Descent or GLMtron algorithm, resulting in similar performance between DP-FTRL and DP-GLMtron. Additionally, it is important to note that DP-FTRL should be compared with DP-TAGLMtron rather than DP-GLMtron, as both DP-FTRL and DP-TAGLMtron utilize the tree mechanism, whereas DP-GLMtron does not. Moreover, even though DP-FTRL converges at the end of the training, it can be observed that the excess risk for DP-FTRL is significantly larger than that of DP-TAGLMtron, indicating that it may converge to a suboptimal solution.

C Supported Lemmas

Here, we provide some relaxation for Assumption 3.6.

Assumption C.1 (Moment symmetricity conditions [49]) . Assume that:

For every u ∈ H , it holds that

✶ 2. For every u ∈ H and v ∈ H , it holds that

For every u ∈ H , it holds that

For every u ∈ H and v ∈ H

✶ ✶ Lemma C.2 ([49]) . The following results are direct consequences of Assumption C.1.

Under Assumption C.1, it holds that: for every vector u ∈ H ,

Under Assumption C.1, it holds that: for every vector u ∈ H

D DP-GLMtron

D.1

Privacy Guarantee

To guarantee the privacy of Algorithm 1, we should first ensure that each step of DP-GLMtron is private.

Lemma D.1. Each update step of DP-GLMtron (Algorithm 1)ensures ( ε 0 , δ 0 ) -differential privacy, provided that f = c ε 0 , where c ≥ √ 2 log(1 . 25 /δ 0 ) , and γ t denotes the clipping norm.

Proof. We first consider { w t } N -1 t =0 .

Each update step (excluding the DP-noise addition) is of the form:

where clip γ t ( v ) = v · max { 1 , γ t ∥ v ∥ 2 } . Consequently, the local L 2 sensitivity of w t +1 is determined by analyzing the variation in the t th iteration data sample, as follows:

Moreover, denoting ε ′ = ε 0 / √ 8 N log(2 /δ 0 ) and δ ′ = δ 0 / (2 N ) , according to Lemma 2.3 in [24], we could know that { s t } N t =0 is ( ε ′ , δ ′ ) -DP.

Lemma D.2 ([17]) . For a domain D , let R ( i ) : f ×D → S ( i ) for i ∈ [ n ] (where S ( i ) is the range space of R ( i ) ) be a sequence of algorithms such that R ( i ) ( z 1: i -1 , · ) is a ( ε 0 , δ 0 ) -DP local randomizer

for all values of auxiliary inputs z 1: i -1 ∈ S (1) ×··· × S ( i -1) . Let A s : D n →S (1) ×··· × S ( n ) be the algorithm that given a dataset x 1: n ∈ D n , samples a uniformly random permutation π , then sequentially computes z i = R ( i ) ( z 1: i -1 , x π ( i ) ) for i ∈ [ n ] , and outputs z 1: n . Then for any δ ∈ [0 , 1] such that ε 0 ≤ log( n 16 log(2 /δ ) ) , A s is ( ε, δ + O ( e ε δ 0 n )) -DP where ε is:

We are now prepared to prove Theorem 4.1, utilizing the lemmas discussed above.

Firstly, we reformulate the update rule into a sequence of one-step algorithms as follows:

where u denotes auxiliary inputs, and π ( t ) represents the sample at the t -th iteration after randomly permuting the input data.

From Lemma D.1, each R ( t +1) ( u 0: t , · ) is a ( ε 0 , δ 0 ) -DP local randomizer algorithm, where ε 0 ≤ log( N 16 log(2 / ̂ δ ) ) . The output of DP-GLMtron is derived through post-processing of the shuffled outputs u t +1 = R ( t +1) ( u 0: t , ( x , y )) for t ∈ 0 , . . . , N -1 . Therefore, by Lemma D.2, Algorithm DP-GLMtron adheres to ( ̂ ε, ̂ δ + O ( e ̂ ε δ 0 N )) -DP, where:

Assuming ε 0 ≤ 1 2 , we can infer the existence of some constant c 1 > 0 such that:

By setting f = 2 log(1 . 25 /δ 0 ) ε 0 in Lemma D.1, we ensure that each update step of DP-GLMtron independently satisfies ( ε 0 , δ 0 ) -DP, based on standard Gaussian mechanism. Replacing ε 0 = √ 2 log(1 . 25 /δ 0 ) f , we obtain:

To satisfy overall ( ε, δ ) -DP, set ̂ δ = δ 2 , and δ 0 = c 2 · δ e ̂ εN for some constant c 2 > 0 . From this, we have:

For any ε ≤ 1 , setting f = c 3 · log( N/δ ) ε √ N ≥ c 3 log( N/δ ) log(1 /δ ) ε √ N for a sufficiently large c 3 > 0 ensures that ̂ ε ≤ ε . Additionally, to fulfill Lemma D.2's assumption, ε 0 < 1 2 must be satisfied, which is attainable by setting ε = O ( √ log( N/δ ) N ) .

This implies that for f = Ω( log( N/δ ) ε √ N ) , DP-GLMtron achieves ( ε, δ ) -DP as long as ε = O ( √ log( N/δ ) N ) , thereby completing the proof.

D.2 Utility Guarantee

Here we first provide several results that will be used in our DP-GLMtron utility bound. Lemma D.3 ([28]) . Let m = Ω( b log( b/δ ′ ) log N/ε ′ ) and denote V = max {∥ w ∗ -w t ∥ 2 H + σ 2 , ∥ w ∗ ∥ 2 H + σ 2 , ∥ w t ∥ 2 H + σ 2 } , with probability at least 1 -1 /b , Algorithm 2 returns s t such that V ≤ γ 2 t ≤ 2 V .

Proof. Consider each sample x i ∼ D satisfying ( H , C 2 , a, b ) -Tail and the inherent noise z i satisfying ( σ 2 , C 2 , a, b ) -Tail. It implies the following facts with Assumption 3.4

∃ a > 0 s.t. with probability ≥ 1 -b z , such that ∥ z ∥ 2 2 ≤ σ 2 · log 2 a (1 /b z ) .

Additionally, we know that

According to the aforementioned facts, it holds that with probability ≥ 1 -b x , such that

Moreover,

The second part of the proof can be directly derived from Lemma 2.3 in [24] and Theorem 5 in [28], because each part of d p i , with high probability, holds that

Therefore, we have the following clipping results:

Lemma D.4 (Generic bounds on the DP-GLMtron iterates [49]) . Suppose that Assumption 3.6 holds. Considering the DP-GLMtron algorithm, we have the following recursion:

Now consider the recursion of A t given in Lemma D.4. Note that A t is related to A t -1 through a linear operator, therefore A t can be understood as the sum of two iterates, i.e., A t := B t + C t , where

and

where

Besides, since our DP-GLM-tron is run with constant stepsize η and outputs the average of the iterates:

Then, the following lemma holds:

Lemma D.5. Suppose that Assumption 3.6 hold. For w N defined in Equation (9) , we have that

Proof.

The remaining proof simply follows [56].

From the decomposition presented in Equation (7) and Equation (8), we know that ∑ N t =0 A t = ∑ N t =0 B t + ∑ N t =0 C t . With this foundation, we can now bound the bias and variance terms separately.

Upper bound for variance error

We then assume that C t -1 ⪯ ησ 2 1 -η ( α tr( H )) I + 4 η Γ 2 f 2 1 -η ( α tr( H )) H -1 , and exam C t based on Equation (7):

For the simplicity, we define Σ := σ 2 H +4Γ 2 f 2 I . By the definitions of T and ˜ T , we have:

where the last inequality is due to the fact that ˜ M is a PSD mapping. Then by the iteration of variance, we have for all t ≥ 0 ,

Consequently, the variance error can be represented as follows, in accordance with Lemma D.5:

Final results can be obtained:

variance error:

where the second inequality holds since tr( H -1 0: k ∗ ) = ∑ i =1 λ -1 i ≤ Nη · k ∗ and λ i ≥ 1 ηN .

Lower bound for variance error

Similarly, we derive a recursion for the variance term to establish the lower bound:

It implies that

where the last inequality comes from η I -η 2 4 H ⪯ η I .

Applying Lemma D.5, we could derive the lower bound for variance error:

By defining f ( x ) := ∑ N -1 t =0 (1 -(1 -x ) N -t -1 )(1 -(1 -x ) t ) and according to Lemma C.3 in [56], we have:

Upper and Lower bound for bias error

According to Theorem B.5 in [49], the upper and lower bounds on the bias error can be directly derived:

Lemma D.6. Suppose Assumption 3.6 holds and the step size satisfies η ≤ 1 /λ 1 , then it holds that:

By integrating the aforementioned results, we thus complete the proof.

E DP-TAGLMtron

E.1 Privacy Guarantee

Lemma E.1 ([22]) . Algorithm 4 guarantees ( τ, τ ⌈ lg( N +1)] 2 σ 2 ) -Rényi differential privacy, where N is the number of samples. Setting σ = √ 2 ⌈ lg( N +1) ⌉ ln(1 /δ ) ε , one can guarantee ( ε, δ ) -differential privacy, for ε ≤ 2 ln(1 /δ ) .

To guarantee the privacy of Algorithm 3, we should first ensure that each step of DP-TAGLMtron is private.

We first consider { l t } N -1 t =0 . According to Lemma E.1, it ensures that DP-tree-aggregation step is ( ε, δ ) -DP with f = √ 2 ⌈ lg( N +1) ⌉ ln(1 /δ ) ε .

Moreover, denoting ε ′ = ε/ √ 8 N log(2 /δ ) and δ ′ = δ/ (2 N ) , according to Lemma 2.3 in [24], we could know that { s t } N t =0 is ( ε ′ , δ ′ ) -DP.

Combining with the advanced composition theorem, it holds that Algorithm 3 is ( ε, δ ) -DP.

E.2 Utility Guarantee

Here we introduce several lemmas for future proof.

Lemma E.2. Given a sequence of Positive Semi-Definite (PSD) matrices P t , t = 0 . . . N , where P t = ( I η 2 · T (2 η )) ◦ P t -1 and P 0 = I , if η ≤ 1 / (4 β tr( H ) log N ) , then for any t ∈ [0 , N ] , the following inequality holds:

where k ∗ ∈ [ N ] can be arbitrarily chosen.

Lemma E.3. Under Assumption 3.4(A), for every A ⪰ 0 , it holds that

Lemma E.4 (Generic bounds on the GLM-tron iterates [49]) . Suppose that Assumption 3.6 holds. Consider the GLM-tron algorithm, we have the following recursion:

where A t := E ( w t -w ∗ )( w t -w ∗ ) ⊤ , t ≥ 0 .

Lemma E.5 (Generic bounds on the DP-TAGLMtron iterates) . Suppose that Assumptions 3.6 (A) hold. Consider the DP-TAGLMtron algorithm, we have the following recursion:

Proof of Lemma E.5. According to the update rule of DP-TAGLMtron, we know:

where l ′ ≤ t is the private estimation of ∑ t i =0 l i generated by tree aggregation protocol. From [22], we notice l ′ ≤ t = ∑ t i =0 l i + v t , where l i = ( x t (ReLU( x ⊤ t w t ) -y t )) and v t is a combination of at most O ( ⌈ log 2 t ⌉ ) Gaussian noise. If we set w 0 = 0 , then it holds that:

By denoting Γ t = v t -v t -1 and noticing that v t is a combination of at most O ( ⌈ log 2 t ⌉ ) Gaussian noise, it implies that Γ t is a combination of at most ⌈ log 2 t + 1 ⌉ random Gaussian vectors, i.e. Γ t = ∑ q ∈Q ( t ) γ ( q )Γ f 2 g q , where Q ( t ) is an index set and Q ( t ) = { 2 Q -1 , 2 Q -1 +2 Q -2 , . . . , 2 Q -1 + 2 Q -2 + · · · +2 0 } when t = 2 Q with Q ∈ N and γ ( q ) = max { γ 1 , . . . , γ q } . Moreover, we know g q ∼ N (0 , I d × d ) , indicating:

✶ ✶ Let us consider the expected outer product: A t = E [ ∥ w t -w ∗ ∥ 2 2 ] = E [( w t -w ∗ )( w t -w ∗ ) ⊤ ] = E (( I -η ✶ [ x ⊤ t w t -1 > 0] · x t x ⊤ t ) · ( w t -1 -w ∗ ))(( I -η ✶ [ x ⊤ t w t -1 > 0] · x t x ⊤ t ) · ( w t -1 -w ∗ )) ⊤ ︸ ︷︷ ︸ (quadratic term 1) + η 2 · E ( ✶ [ x ⊤ t w ∗ > 0] -✶ [ x ⊤ t w t -1 > 0]) 2 · ( x t x ⊤ t w ∗ )( x t x ⊤ t w ∗ ) ⊤ ︸ ︷︷ ︸ (quadratic term 2) +2 η · E ( ✶ [ x ⊤ t w ∗ > 0] -✶ [ x ⊤ t w t -1 > 0]) · ( I -η ✶ [ x ⊤ t w t -1 > 0] · x t x ⊤ t )( w t -1 -w ∗ )( x t x ⊤ t w ∗ ) ⊤ ︸ ︷︷ ︸ (crossing term 1) -η · ( E [( I -η ✶ [ x ⊤ t w t -1 > 0] x t x ⊤ t )( w t -w ∗ ) Γ ⊤ t ] + E [ Γ t ( w t -w ∗ ) ⊤ ( I -✶ [ x ⊤ t w t -1 > 0] η x t x ⊤ t )]) ︸ ︷︷ ︸ (crossing term 2) -η · ( E [( ✶ [ x ⊤ t w ∗ > 0] -✶ [ x ⊤ t w t -1 > 0])( x t x ⊤ t w ∗ ) Γ ⊤ t ] + E [ Γ t ( x t x ⊤ t w ∗ ) ⊤ ( ✶ [ x ⊤ t w ∗ > 0] -✶ [ x ⊤ t w t -1 > 0])]) ︸ ︷︷ ︸ (crossing term 3) + η 2 · E ( σ 2 x ⊤ t x t ) ︸ ︷︷ ︸ + η 2 · E [ Γ t Γ ⊤ t ] ︸ ︷︷ ︸ .

It can be observed that quadratic terms 1,2,3 and crossing term 1 are original terms for non-private problems, where the remaining are introduced by the private noise.

According to Assumption 3.6, Lemma C.2 and the independence of w ∗ , we have:

(crossing term 3) = 0 ,

✶ ✶ Now we examine the crossing term 2, denoting R t := ( I -η ✶ [ x ⊤ t w t -1 > 0] x t x ⊤ t ) . According to the update rule, we have:

Hence, we have:

By multiplying both sides by A and taking the expected value, we can obtain:

We first consider the second term, notice that:

E [ η R t x t -1 y t -1 Γ ⊤ t ] = E [ η R t ( ✶ [ x ⊤ t w ∗ > 0] · x t x ⊤ t w ∗ + z t x t ) Γ ⊤ t ] = 0 . Furthermore, considering Γ t and Γ j , when j < t and j / ∈ Q ( t ) , it follows that E [ Γ j Γ ⊤ t ] = 0 , otherwise, we have:

These results arise because only Γ q is dependent on g q for q ∈ Q ( t ) . Hence, denoting R [ t : j ] = R t · · · R j , we have:

̸

For the last term of crossing term 3, we could also derive:

Combining the above results with the symmetric of R t , it implies that:

Recall R t := ( I -η ✶ [ x ⊤ t w t -1 > 0] x t x ⊤ t ) and the lemma in [49]. It indicates that:

We first consider P p = ( I η 2 · T (2 η )) ◦ P p -1 and P 0 = I . Then, it follows that:

where the term HP p -1 can be bounded by Lemma E.2. Now, we could derive:

As a results, the generic bounds on the DP-TAGLMtron iterate could have the following recursion:

We now move on to the second part. In light of the Γ t definition, we know it is at least a random Gaussian vector such that Γ t = ˜ Γ g t , where ˜ Γ = min γ . Moreover, it holds that

̸

Since E [ Γ j Γ ⊤ t ] = 0 for any j = t . Therefore, the iteration of A t will be the same case in DPGLMtron.

Consequently, analogous to the previous case, we can decompose A t into bias term B t and variance term C t .

where

Lemma E.6. Let M represent a union upper bound for M 0 , . . . , M N -1 . Under Assumptions (A) and 3.6, and provided that the step size η satisfies η ≲ 1 / (tr( H ) log N ) , the following holds:

Proof of Lemma E.6. Our proofs follow the main idea of [48], we include them here for completeness. According to the update rule of C t , it follows that:

It can be observed that C 0 ⪯ C 1 ⪯ · · · ⪯ C N since ( I η 2 · T (2 η )) is a PSD mapping.

Let M be an union upper bound of { M 0 , . . . , M N -1 } , we will have:

The last inequality comes from ( ˜ T - T ) , ( I η 2 · T (2 η )) and ( I η 2 · ˜ T (2 η )) are PSD mappings. Hence, it implies that:

where the last inequality comes from Lemma E.3. Therefore, we have:

Lemma E.7. Variance error] If the step size satisfies η ≲ 1 / (tr( H ) log N ) , then the following holds for any k ≥ 1 ,

where

Proof. According to Lemma D.5, we have:

Furthermore, Lemma E.5 implies that, for any t ≤ N :

Now we consider an arbitrary PSD mapping A commuting with H , it holds that:

Moreover, noting that for any x ∈ (0 , 2 /η ) , we have 1 -(1 -ηx 2 ) N ≤ min { 1 , Nηx/ 2 } , then

for any k ∗ ≥ 0 . Therefore, we can define a new matrix as follows:

Combining the previous results, we obtain variance

Now, we introduce the lemma in [49] to complete the proof.

Lemma E.8 ([49]) . If the step size satisfies η ≤ 1 /λ 1 , then it holds that for any k ∗ ≥ 1 ,

E.3 Specific Disrtibutions

Proof of Corollary. 1. if λ k = k -r , by the definition of k ∗ , we have k ∗ = ( Nη 2 ) 1 /r , then:

For the sake of simplicity, we assume ∥ w 0 -w ∗ ∥ 2 ≤ W 2 , then it hold that:

If we choose the step size as follows:

which indicates:

Thus, it holds that Utility = ˜ O ( N -r 1+ r · (1 + ( f 2 N r 1+ r ) r 1+2 r )) .

If λ k = e -k , we have k ∗ = log( Nη ) , then:

Hence,

If we choose the following step size η ≃ min { 1 , 1 √ Nf } , then we complete our results:

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The paper's main contributions and scope are included in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper provides a discussion on the limitation and future direction.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions and proofs are included in the submission.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All experiments can be reproduced.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All data and code will be released after acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the detailed experiment setup

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The factors that will influence the experiment are discussed.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The Experiments Compute Resources are discussed in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All related works are cited correctly and explicitly.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of learning a planted (student-teacher) ReLU regression model, privately. The paper proposes two algorithms (DP-GLMtron and DP-TAGLMtron) to do this. For both algorithms, privacy utility tradeoffs are computed. The results do not explicitely contain the ambient dimension $d$, while they contain other terms as the \"effective dimension\", which allow to have bounds explicitely vacuous in the regime where there are more parameters (in this case, equal to the number of input dimensions $d$) than number of samples $N$\n\nSTRENGTHS:\nThis paper tackles a difficult and theoretically interesting problem, i.e. (in its generality) private, non-convex optimization. This paper focuses on the case of a planted ReLU regression model. The initial assumptions (see basically page 4) are, in my opinion, reasonable for this setting.\n\nWEAKNESSES:\nThe paper has, in my opinion, 4 major weaknesses:\n\n- Its fit with the related literature is not always clear. The main improvement with respect to previous results is remarked in Table1, and the baseline is the recent and unpublished work [1]. In line 98, results on convex optimization are presented [2, 3]. This results are not presented with the dependence on the $l_2$ norm of the solution, and it is not mentioned that both results assume the input samples to have bounded $l_2$ norm, which is not the case in this work. This makes the comparison with the convex baseline unclear. Furthermore, comparison with [4] is not provided, which definitely offers a more solid baseline than [1].\n\n- The analysis is not well motivated. The authors contribution is based on the two proposed algorithms, namely DP-GLMtron and DP-TAGLMtron. Considering the arguable practicality of such algorithms in realistic deep learning problems, why didn't the authors focused, for example, on providing bounds on DP-SGD instead? This makes the results being very specific to the toy setting of ReLU regression, for which the two algorithms are specifically designed for.\n\n- While it is true that the provided bounds do not contain explicitely $d$, it is unclear the true dependence with the ambient dimension through other terms. Namely, in line 222 the authors set $\\|x\\|_2 \\leq 1$, argued to be the improvement with respect to [1]. This allows to remove the dependence of $d$ from the trace of the covariance $H$. Authors also assume $\\|w_0 - w^*\\|_2$ to be bounded in corollary 4.3. Furthermore, the bounds also depend on $\\Gamma$, which does not receive discussion on how this term is reasonably small.\n\n- Empirical validation of the claims is relatively weak. In particular, in the main body, empirical results are presented on a synthetic dataset with quickly decaying covariance eigenvalues. The experiments on MNIST are pushed in Appendix B (there might be a typo in the third line of the caption of Figure 2, where it's mentioned that the data is Bernoulli), where the results are, in my opinion, puzzling. The baseline of DP-SGD is presented (which has to be improved with the two proposed algorithms to motivate the analysis), and the excess risk for $\\varepsilon = 0.5$ increases as the number of samples increases (!). Furthermore, this excess risk is approximately $\\sim 24$, which seems larger than simply random guessing. This dubious results make me suspect of a bad comparison with standard and well established algorithms as DP-SGD, pointing towards the second weakness mentioned in this list.\n\n\n\n[1] - https://arxiv.org/pdf/2310.08425\n\n[2] - https://proceedings.mlr.press/v32/jain14.pdf\n\n[3] - https://arxiv.org/pdf/2006.06783\n\n[4] - https://proceedings.mlr.press/v97/wang19c/wang19c.pdf\n\nQUESTIONS:\nCan the authors provide extensive comparison with [4]?\n\nWhy didn't the authors focused on DP-SGD? Why it is useful to introduce these two algorithms?\n\nCan the authors comment on their implicit assumption on $\\Gamma$?\n\nCan the authors elaborate on the experimental settings used for DP-SGD on MNIST data?\n\nLIMITATIONS:\nNo potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-19"}

comments: SUMMARY:
The paper provides an algorithm for differentially private RELU regression. They claim that their results outperform DPSGD. Additionally, for the case of a small privacy budget, they provide a tree aggregation protocol that balances privacy and utility. Finally, extensive experimental results are provided to support the claims.

STRENGTHS:
1) A novel algorithm is provided for differentially private RELU regression which outperforms DPSGD.

2) Detailed theoretical analysis is provided for the proposed algorithms.

3) Sufficient experimental evaluation is provided for the proposed algorithms with experiments on both synthetic and real-world data

WEAKNESSES:
1) There are a lot of recent works on privacy-utility tradeoffs which seem relevant to this paper. A literature survey involving works in that area can be helpful for future readers to further extend this work.

QUESTIONS:
1) Can the authors propose ideas on how the proposed approach be extended to models with higher complexity such as deep neural networks?
2) The authors analyze the privacy-utility tradeoff in their work. They should also mention similar papers involving privacy-utility tradeoff algorithms such as

[a] Alireza Fallah, Ali Makhdoumi, Azarakhsh Malekian, and Asuman Ozdaglar Optimal and Differentially Private Data Acquisition: Central and Local Mechanisms Operations Research 2024 72:3, 1105-1123
[b] Ameya Anjarlekar, Rasoul Etesami, & R. Srikant. (2023). Striking a Balance: An Optimal Mechanism Design for Heterogenous Differentially Private Data Acquisition for Logistic Regression.
[c] Andrew Lowy, Zeman Li, Tianjian Huang, & Meisam Razaviyayn. (2024). Optimal Differentially Private Model Training with Public Data.

3) Typo on line 335

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper revisits the problem of differentially private (DP) ReLU regression in overparameterized regimes. The authors propose two novel algorithms, DP-GLMtron and DP-TAGLMtron, which outperform conventional methods like DPSGD. The paper provides theoretical analysis of these algorithms, including privacy guarantees and utility bounds. The authors extend their analysis beyond Gaussian-like data distributions to settings with eigenvalue decay, showing how data distribution impacts learning in high dimensions. The paper also includes empirical results on both synthetic and real-world datasets to validate their theoretical findings.\n\nSTRENGTHS:\n- The proposed algorithm is noval and intuitive.\n- The authors provide comprehensive theoretical analysis, and the results are concisely conveyed.\n- The writing is good.\n\nWEAKNESSES:\n- The assumptions used in this paper are, though relaxed, strong and may not always hold in practice.\n- The simulation is weak and contains some mismatches with the theory.\n\nQUESTIONS:\n- Can the authors provide the specific location of the claimed rate $\\widetilde{O}(\\frac{1}{\\sqrt{N}}+\\min \\left(\\frac{d^{1 / 2}}{(N \\varepsilon)^{1 / 2}}, \\frac{1}{(N \\varepsilon)^{2 / 3}}\\right)) $ in [1]? I scanned the paper and don't see how this rate appears. Thanks in advance :).\n- What is the role of ReLU regression? Does it serve as a fundamental simplified problem for neural network optimization?\n- The authors provide special cases of $D_{eff}^{priv}$. However, I would appreciate an explanation of its form. For instance, compared to $D_{eff}$, why is it not squared over $\\lambda_i$, and why did the sum of inverse $\\lambda_i$ appear?\n- In the simulation (Figure 1), I noticed that in some cases (especially Figure 1(a)), the excess risk is increasing with respect to sample sizes. How does this happen?\n- Again in the simulation, DP-GLMtron gradually matches the performance of DP-TAGLMtron as $\\varepsilon$ increases, while this is not true for the theory, where DP-GLMtron does not have a guarantee when $\\varepsilon$ is large. How does this happen?\n- Following the last two questions, it seems that DP-GLMtron and DP-TAGLMtron are more sensitive to variations of $\\varepsilon$ than DP-SGD. This is also true for the theory (for instance, Table 1 suggests that for the proposed methods and DPSGD, dependence is $\\varepsilon^2$ and $\\varepsilon^{2/3}$, respectively).\n- Why did the authors choose the settings for $i^{-2}$ and $i^{-3}$ in the simulation?\n\n\n[1] Differentially Private Non-convex Learning for Multi-layer Neural Networks.\n\nLIMITATIONS:\nThe limitations should appear in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-16"}