Title: Label Robust and Differentially Private Linear Regression: Computational and Statistical Efficiency
Venue: NeurIPS 2023
Accepted: yes
Keywords: Differential Privacy; Private Estimation
Abstract: We study the canonical problem of linear regression under $(\varepsilon,\delta)$-differential privacy when the datapoints are sampled i.i.d.~from a distribution and a fraction of response variables are adversarially corrupted. We provide the first provably efficient -- both computationally and statistically -- method for this problem, assuming standard assumptions on the data distribution. Our algorithm is a variant of the popular differentially private stochastic gradient descent (DP-SGD) algorithm with two key innovations: a full-batch gradient descent to improve sample complexity and a novel adaptive clipping to guarantee robustness. Our method requires only linear time in input size, and still matches the information theoretical optimal sample complexity up to a data distribution dependent condition number factor.  Interestingly, the same algorithm, when applied to a setting where there is no adversarial corruption, still improves upon the existing state-of-the-art and achieves a near optimal sample complexity.

Label Robust and Differentially Private Linear Regression: Computational and Statistical Efficiency

Abstract

Westudy the canonical problem of linear regression under ( ε, δ ) -differential privacy when the datapoints are sampled i.i.d. from a distribution and a fraction of response variables are adversarially corrupted. We provide the first provably efficient - both computationally and statistically - method for this problem, assuming standard assumptions on the data distribution. Our algorithm is a variant of the popular differentially private stochastic gradient descent (DP-SGD) algorithm with two key innovations: a full-batch gradient descent to improve sample complexity and a novel adaptive clipping to guarantee robustness. Our method requires only linear time in input size, and still matches the information theoretical optimal sample complexity up to a data distribution dependent condition number factor. Interestingly, the same algorithm, when applied to a setting where there is no adversarial corruption, still improves upon the existing state-of-the-art and achieves a near optimal sample complexity.

1 Introduction

Differential Privacy (DP) [33] is a standard notion of privacy widely adopted by both industry and government [76, 35, 36, 2]. With widespread usage of ML and statistical techniques, DP becomes even more critical to ensure private information of participating individuals is not revealed in any form via the learned model. An statistical estimator is said to be ( ε, δ ) -differentially private if presence/absence of an individual's data point in the dataset does not significantly change the estimated output. Smaller ε > 0 and δ ∈ [0 , 1] imply stronger privacy guarantees.

While privacy preserving statistical estimators have been studied extensively in recent past, several critical questions remain open (see App. A for a survey). Consider the canonical statistical task of linear regression with n i.i.d. samples, { ( x i ∈ R d , y i ∈ R ) } n i =1 , drawn from x i ∼ N (0 , Σ) , y i = x ⊤ i w ∗ + z i , z i ∼ N (0 , σ 2 ) and E [ x i z i ] = 0 for some true parameter w ∗ ∈ R d . The error is measured in (1 /σ ) ∥ ˆ w -w ∗ ∥ Σ := (1 /σ ) ∥ Σ 1 / 2 ( ˆ w -w ∗ ) ∥ , which correctly accounts for the signalto-noise ratio in each direction; in the direction of large eigenvalue of Σ , we have larger signal in x i

but the noise z i remains the same. We expect smaller errors in those directions, which is accounted for in the error measure (1 /σ ) ∥ ˆ w -w ∗ ∥ Σ .

Minimax optimal sample complexity for estimating the optimal linear regression model with DP was recently established. For the lower bound, using recently introduced score attack technique, [16, Theorem 3.1] shows that n = Ω( d/α 2 + d/ ( εα )) samples are necessary to achieve an error of (1 /σ ) ∥ ˆ w -w ∗ ∥ Σ = α (in expectation). For the matching upper bound, High-dimensional ProposeTest-Release (HPTR) in [61] and Robust-to-Private in [10] show that n = ˜ O ( d/α 2 + d/ ( εα )) samples are also sufficient. The first term of d/α 2 is the fundamental sample complexity even if privacy is not required, and the second term of d/ ( εα ) is the cost of privacy.

This implies that, statistically, the problem appears to be solved. However, computationally, the problem is still open despite multiple studies of the problem. That is, the statistical optimal algorithms still take exponential time.

After a series of efforts in computationally efficient approaches as surveyed in App. A, [81] achieves the best known sample complexity of n = ˜ O ( d/α 2 + κd/ ( εα ) + κ 2 d/ε ) , where κ is the condition number of the covariance Σ of the covariates. Compared to HPTR, the cost of computational efficiency is factor of κ in the second term and the third term that is unnecessary. As the condition number can be quite large, improving the dependence on κ is of utmost importance. Furthermore, the technique of [81] strictly requires sampling without replacement, whose analysis relies on having an explicit form of the end-to-end update. In particular, their analysis technique is not applicable to the case with corrupted samples.

In contrast, we propose a novel method (Alg. 1) that builds upon full-batch gradient descent and applies a carefully chosen adaptive clipping which is a general technique used in practice as well [1]. Together with an intuitive but intricate analysis technique, we improve the sample complexity to n = ˜ O ( d/α 2 + κ 1 / 2 d/ ( εα )) .

Corollary 1.1 (Corollary of Thm. 3.1 for sub-Gaussian data) . Alg. 1 is ( ε, δ ) -DP. Let S = { ( x i , y i ) } n i =1 be a dataset of i.i.d. samples with x i ∼ N (0 , Σ) , y i = x ⊤ i w ∗ + z i and z i ∼ N (0 , σ 2 ) for some unknown true parameter w ∗ = Σ -1 E [ y i x i ] ∈ R d and unknown Σ and σ 2 . Then n = ˜ O ( d/α 2 + κ 1 / 2 d/ ( εα )) samples are sufficient for Alg. 1 to achieve (1 /σ ) ∥ ˆ w -w ∗ ∥ Σ = ˜ O ( α ) with high probability, where κ := λ max (Σ) /λ min (Σ) .

Due to space constraints, we focus on sub-Gaussian distributions in the main text and provide comparisons to prior work in Tab. 1. Our analysis in App. H applies to a more general family of light-tailed distributions, called sub-Weibull. Next, when the noise in the samples is heavy-tailed , a similar algorithm can be applied with carefully chosen clipping thresholds to account for the heavier tail. Concretely, for k -th moment bounded distributions, the tail of the distribution gets increasingly heavier with smaller k . This would require larger number of samples to achieve the same accuracy, which is captured in our sample complexity of n = ˜ O ( d/α 2 k/ ( k -1) + κ 1 / 2 d/ ( εα k/ ( k -1) )) . We explain the heavy-tailed setting, provide a detailed analysis and a proof, and discuss the results in App. L. This is the first efficient algorithm with provable guarantees achieving ( ε, δ ) -DP.

Corollary 1.2 (informal version of Coro. L.7 for heavy-tailed noise) . Alg. 4 is ( ε, δ ) -DP. Let S = { ( x i , y i ) } n i =1 be a dataset of i.i.d. samples with x i ∼ N (0 , Σ) , y i = x ⊤ i w ∗ + z i , and the zero-mean, independent, and heavy-tailed noise z i satisfies E [ | z/σ | k ] = O (1) for some unknown true parameter w ∗ ∈ R d and unknown Σ and σ 2 . Then n = ˜ O ( d/α 2 k/ ( k -1) + κ 1 / 2 d/ ( εα k/ ( k -1) )) samples are sufficient for Alg. 4 in App. L to achieve an error rate of (1 /σ ) ∥ ˆ w -w ∗ ∥ Σ = ˜ O ( α ) with high probability, where κ := λ max (Σ) /λ min (Σ) .

Perhaps surprisingly, we show that Alg. 1 is also robust against label-corruption, where an adversary selects an arbitrary α corrupt fraction of the data points and changes their response variables arbitrarily. Ideally, we want a robust algorithm against a stronger adversary who can corrupt the covariates also. However, even for a simpler problem of private mean estimation, achieving robustness against such a strong adversary with O ( d ) samples requires heavy machinery (convex relaxations of sum-of-squares optimization) with significantly more computations (although polynomial) [45].

Our lower bound in Prop. 3.9, together with the lower bound in [16] on the uncorrupted case, shows that n = Ω( d/α 2 + d/ ( εα )) samples are necessary to achieve an error rate of (1 /σ ) ∥ ˆ w -w ∗ ∥ Σ = O ( α + α corrupt ) . In particular, it is impossible to achieve an error below α corrupt even if we have

infinite samples (Prop. 3.9), and hence there is no need to aim for α < α corrupt . This lower bound is matched by exponential time approaches, HPTR in [61] and Robust-to-Private in [10], which also guarantee robustness. Currently, there is no efficient algorithm that can guarantee both privacy and robustness for linear regression. To this end, we provide the first efficient algorithm guaranteeing both, with a sample complexity that is optimal up to a κ 1 / 2 factor.

Corollary 1.3 (Corollary of Thm. H.3 for sub-Gaussian data with adversarial label corruption) . Under the hypotheses of Coro. 1.1, suppose α corrupt -fraction of the labels are corrupted arbitrarily. Then n = ˜ O ( d/α 2 + κ 1 / 2 d/ ( εα )) samples are sufficient for Alg. 1 to achieve an error rate of (1 /σ ) ∥ ˆ w -w ∗ ∥ Σ = ˜ O ( α + α corrupt ) with high probability, where κ := λ max (Σ) /λ min (Σ) .

When α corrupt = 0 , this recovers the non-robust result from Coro. 1.1. A similar robustness guarantee also holds for heavy-tailed settings. We provide a formal statement in App. L

Contributions. For a canonical problem of private linear regression under sub-Gaussian distributions, the best known efficient algorithm [81] requires

to achieve (1 /σ ) ∥ ˆ w -w ∗ ∥ Σ = α . We provide the first efficient algorithm that improves this to

which nearly matches the exponential-time algorithms [61, 10] and the lower bound [16] up to κ 1 / 2 in the second term. For the same problem, we show that the same algorithm is the first to achieve robustness against adversarial corruption of the labels.

Under a heavy-tailed distribution of the noise, we provide the first computationally efficient algorithm, to the best of our knowledge, that achieves a sample complexity close to that of an exponential-time algorithm of [61]. There is no matching lower bound in the heavy-tailed setting. This is also the first efficient algorithm to achieve robustness against adversarial corruption of the labels under heavy-tailed noise.

2 Problem formulation and background

When there is no adversary, we present our results under the standard linear model with sub-Gaussian covariates and noise. In App. H, we present a more general family of ( K,a ) -sub-Weibull distributions that recovers the standard sub-Gaussian family as a special case when a = 0 . 5 . The necessity of such assumptions on the tail is explained in Sec. 3.4.

Assumption 2.1 (sub-Gaussian model) . We have i.i.d. samples S = { ( x i ∈ R d , y i ∈ R ) } n i =1 from a distribution P Σ ,w ∗ ,σ 2 of a linear model y i = ⟨ x i , w ∗ ⟩ + z i , where the input vector x i has zero

mean E [ x i ] = 0 and a positive definite covariance Σ := E [ x i x ⊤ i ] ≻ 0 , and the (input dependent) label noise z i has zero mean E [ z i ] = 0 and variance σ 2 := E [ z 2 i ] . We further assume E [ x i z i ] = 0 , which is equivalent to assuming that the true parameter w ∗ = Σ -1 E [ y i x i ] . We assume the marginal distributions of x i and z i are K -sub-Gaussian with K = O (1) , as defined below.

Given a dataset S that is i.i.d. sampled from P Σ 2 ,w ∗ ,σ 2 satisfying Asmp. 2.1, our goal is to estimate w ∗ that minimizes (1 /σ ) ∥ ˆ w -w ∗ ∥ Σ which is also equivalent to minimize the excess population risk, i.e., L ( w ∗ ) -L ( ˆ w ) where L ( w ) := E ( x,y ) ∼P Σ ,w ∗ ,σ 2 [( y -⟨ w,x ⟩ ) 2 ] .

Notations. A vector x ∈ R d has the Euclidean norm ∥ x ∥ . For a matrix M , we use ∥ M ∥ 2 to denote the spectral norm. The error is measured in ∥ ˆ w -w ∗ ∥ Σ := ∥ Σ 1 / 2 ( ˆ w -w ∗ ) ∥ for some PSD matrix Σ . The identity matrix is denoted by I d ∈ R d × d . Let [ n ] = { 1 , 2 , . . . , n } . ˜ O ( · ) hides some constants terms, K = Θ(1) , and poly-logarithmic terms in n , d , 1 /ε , log(1 /δ ) , 1 /ζ , and 1 /α corrupt . For a vector x ∈ R d , we define clip a ( x ) := x · min { 1 , a/ ∥ x ∥} .

Background on DP. Differential Privacy is a standard measure of privacy leakage when data is accessed via queries, introduced by [33]. Two datasets S and S ′ are said to be neighbors if they differ at most by one entry, which is denoted by S ∼ S ′ . A stochastic query q is said to be ( ε, δ ) -differentially private for some ε > 0 and δ ∈ [0 , 1] , if P ( q ( S ) ∈ A ) ≤ e ε P ( q ( S ) ∈ A ) + δ , for all neighboring datasets S ∼ S ′ and all subset A of the range of the query. We build upon two widely used DP primitives, the Gaussian mechanism and the private histogram. A central concept in DP mechanism design is the sensitivity of a query, defined as ∆ q := sup S ∼ S ′ ∥ q ( S ) -q ( S ′ ) ∥ . We describe Gaussian mechanism and private histogram in App. B.

2.1 Comparisons with the prior work

The state-of-the-art approach introduced by [81] is based on DP-SGD [71], where privacy is ensured by gradient norm clipping and the Gaussian mechanism. Two additional technical components are adaptive clipping and streaming SGD. Adaptive clipping with an appropriate threshold θ t ensures that no data point is clipped (under the sub-Gaussian assumption), while providing a bound on the sensitivity of the average mini-batch gradient (to ensure we do not add too much noise). The streaming approach, where each data point is only touched once and discarded, ensures independence between the past iterate w t and the gradients at round t + 1 , which the analysis critically relies on. For T = ˜ Θ( κ ) iterations where κ is the condition number of the covariance Σ , the dataset S = { ( x i , y i ) } n i =1 is partitioned into { B t } T -1 t =0 subsets of equal size: | B t | = ˜ Θ( n/κ ) . At each round t , the gradients are clipped and averaged with additive Gaussian noise chosen to satisfy ( ε, δ ) -DP:

where ν t ∼ N (0 , I d ) . In [81], a slight variation of this streaming SGD is shown to achieve an error of (1 /σ ) ∥ w T -w ∗ ∥ Σ = α with n = ˜ O ( d/α 2 + κd/ ( εα ) + κ 2 d/ε ) samples (Row 3 in Tab. 1).

Our technical innovations. Our approach builds upon such gradient based methods but makes several important innovations. First, we use full-batch gradient descent, as opposed to the streaming SGD above. Using all n samples reduces the sensitivity of the per-round gradient average by a κ factor, and thus decreases the privacy noise added in each iteration. This improves the second term of sample complexity from κd/ ( εα ) to κ 1 / 2 d/ ( εα ) and removes the third term completely. However, full-batch GD loses the independence that the streaming SGD enjoyed between w t and the samples used in the round t +1 . This dependence makes the analysis more challenging. We instead propose using the resilience to precisely track the bias and variance of the (dependent) full-batch average gradient. Resilience is a central concept in robust statistics that links the tail-property of the distribution to the bias, which we explain in Sec. 5.

Next, one critical component in achieving this improved sample complexity is the new analysis technique we introduce for tracking the end-to-end gradient updates. Since our gradient descent algorithm is not guaranteed to make progress every step, we cannot use the vanilla one-step analysis.

Taking the full end-to-end analysis by expanding the whole gradient trajectory will introduce too many correlated cross-terms which are very hard to control. Therefore, we leverage an every κ -step analysis and show that the objective function at least decreases geometrically every κ steps. To be more specific, our analysis technique in App. H (steps 3 and 4) opens up the iterative updates from the beginning to the end, and exploits the fact that λ max (( η Σ) 1 / 2 (1 -η Σ) i ( η Σ) 1 / 2 ) is upper bounded by 1 / ( i +1) when ∥ η Σ ∥ ≤ 1 . This technique is critical in achieving the near-optimal dependence in κ . This might be of independent interest to other analysis of gradient-based algorithms. We refer to the beginning of step 3 in App. H for a detailed explanation.

Finally, we propose a novel clipping that separately clips x i and ( w ⊤ t x i -y i ) in the gradient, ( w ⊤ t x i -y i ) x i . This is critical in achieving robustness to label-corruption, as we explain in Sec. 3.1.

3 Label-robust and private linear regression

We introduce a novel gradient descent approach. This achieves an improved sample complexity compared to the state-of-the-art algorithm and robustness against label corruption.

3.1 Algorithm

The skeleton of our approach in Alg. 1 is the general DP-SGD [1, 71] with adaptive clipping [7]. We partition the dataset into three equal-sized subsets: S 1 , S 2 , S 3 . S 1 and S 2 are used in adaptively estimating the clipping thresholds, and S 3 is re-used every step to compute the average gradient.

The standard adaptive clipping, e.g., [7, 81], is not robust against label-corruption. Under subGaussian distribution, a positive fraction of the covariates, x i 's, can be close to the origin. If the adversary chooses to corrupt those points with small norm, ∥ x i ∥ , they can make large changes in the corrupted residual, ( y i -w ⊤ t x i ) , while evading the standard clipping by the norm of the gradient; the norm of the gradient, ∥ x i ( y i -w ⊤ t x i ) ∥ = ∥ x i ∥ | y i -w ⊤ t x i | , can remain under the threshold. This is problematic, since the bias due to the corrupted samples in the gradient scales proportionally to the magnitude of the residual (after clipping). To this end, we propose clipping the norm and the residual separately: clip Θ ( x i )clip θ t ( w ⊤ t x i -y i ) . This keeps the sensitivity of gradient average bounded by Θ( θ t ) . The subsequent Gaussian mechanism in line 11 ensures ( ε 0 , δ 0 ) -DP at each round. Applying advanced composition in Lemma B.5 of T rounds, this ensures end-to-end ( ε, δ ) -DP.

Novel adaptive clipping. When clipping with clip Θ ( x i ) , the only purpose of clipping the covariate by its norm, ∥ x i ∥ , is to bound the sensitivity of the resulting clipped gradient. In particular, we do not need to make it robust as there is no corruption in the covariates. Ideally, we want to select the smallest threshold Θ that does not clip any of the covariates. Since the norm of a covariate is upper bounded by ∥ x i ∥ 2 ≤ K 2 Tr(Σ)log(1 /ζ ) with probability 1 -ζ (Lemma J.3), we estimate the unknown Tr(Σ) using Private Norm Estimator in Alg. 3 in App. F and set the norm threshold Θ = K √ 2Γlog( n/ζ ) (Alg. 1 line 4). The n in the logarithm ensures that the union bound holds.

When clipping with clip θ t ( w ⊤ t x i -y i ) , the purpose of clipping the residual by its magnitude, | y i -w ⊤ t x i | = | ( w ∗ -w t ) ⊤ x i + z i | , is to bound the sensitivity of the gradient and also to provide robustness against label-corruption. We want to choose a threshold that only clips corrupt data points and at most a few clean data points. In order to achieve an error (1 /σ ) ∥ w T -w ∗ ∥ Σ = α , we know that any set of (1 -α ) fraction of the clean data points is sufficient to get a good estimate of the average gradient. By clipping at | ( w ∗ -w t ) ⊤ x i + z i | 2 ≤ ( ∥ w t -w ∗ ∥ 2 Σ + σ 2 ) CK 2 log(1 / (2 α )) , Lemma J.3 guarantees that the unclipped subset will be large enough, i.e., (1 -α ) n . At the same time, this threshold on the residual is small enough to guarantee robustness against the label-corrupted samples. We introduce the robust and DP Distance Estimator in Alg. 2 to estimate the unknown (squared and shifted) distance, ∥ w t -w ∗ ∥ 2 Σ + σ 2 , and set the distance threshold θ t = 2 √ 2 γ t √ 9 C 2 K 2 log(1 / (2 α )) (Alg. 1 line 7). Both norm and distance estimation rely on DP histogram (Lemma B.2), but over a set of statistics computed on partitioned datasets, which we explain in detail in App. C.

Algorithm 1: Robust and Private Linear Regression

3.2 Analysis without adversarial corruption

We show that Alg. 1 achieves an improved sample complexity. We provide the proof for a more general class of distributions in App. H and a sketch of the proof in Sec. 5. We address the necessity of the assumptions in Sec. 3.4, along with some lower bounds.

Theorem 3.1. Alg. 1 is ( ε, δ ) -DP. Under sub-Gaussian model of Asmp. 2.1, for any failure probability ζ ∈ (0 , 1) and target error rate α , if the sample size is large enough such that

with a large enough constant, then the choices of a step size η = 1 / ( Cλ max (Σ)) for any C ≥ 1 . 1 and the number of iterations, T = ˜ Θ( κ log ( ∥ w ∗ ∥ )) for a condition number of the covariance κ := λ max (Σ) /λ min (Σ) , ensures that, with probability 1 -ζ , Alg. 1 achieves

where the expectation is taken over the noise added for DP , and ˜ O and ˜ Θ( · ) hide logarithmic terms in K,σ,d,n, 1 /ε, log(1 /δ ) , 1 /α , and κ .

Remark 3.2 . Omitting some constant and logarithmic terms, Alg. 1 requires

samples to ensure an error rate of (1 /σ 2 ) E [ ∥ w T -w ∗ ∥ 2 Σ ] = ˜ O ( α 2 ) . From [16, Theorem 3.1], there exists an n = Ω( d/α 2 + d/ ( εα )) lower bound, and our upper bound matches this lower bound up to a factor of κ 1 / 2 in the second term and other logarithmic factors. Eq. (4) is the best known rate among all efficient private linear regression algorithms, strictly improving upon the state-of-the-art. The best existing efficient algorithm by [81] requires n = ˜ O ( d/α 2 + κd/ ( εα ) + κ 2 d/ε ) to achieve the same error rate. Compared to Eq. (4), the second term is larger by a factor of κ 1 / 2 compared to the second term in Eq. (4). Further, [81] requires κ 2 d/ε , which is not needed in Eq. (4).

Remark 3.3 . Consider the standard settings of linear regression with x i ∼ N (0 , I d ) and z i ∼ N (0 , σ 2 ) such that the condition number is one, our bound given by Eq. (4) nearly matches the lower bound ([16, Theorem 3.1]) up to logarithmic factors.

Remark 3.4 . Note that the leading term in Eq. (4) is the first term d/α 2 when target error α ≤ ε/κ 1 / 2 . Our first term is independent of κ , which matches the lower bound for non-private linear regression.

Remark 3.5 . The third term κ 2 d/ε in [ 81 ] is independent of error rate α but scales as κ 2 . This term is required to ensure the privacy noise added in each iteration is small enough for their DP-SGD to make progress (Appendix. B.2.2 in [81]). Our algorithm is based on full-batch gradient descent, which uses all n samples and thus reduces the sensitivity of gradient average by a κ factor. As a result, we show in Eq. (59) that our algorithm only requires n = ˜ O ((1 /ε ) √ κ 1 / 2 d/α ) to make progress for each iteration. This is strictly smaller than our dominant term κ 1 / 2 d/ ( εα ) and does not show up in our final guarantee. We provide a formal proof in App. H.

Remark 3.6 . One of the key innovations in Alg. 1 is the adaptive distance estimator (Alg. 2 in App. C). The goal is to privately estimate the (shifted) distance of the current estimate, i.e., ∥ w t -w ∗ ∥ Σ + σ 2 , without the knowledge of w ∗ . We show in Thm. C.1 that our novel distance estimator only requires an error-independent sample complexity n = ˜ O ( κ 1 / 2 d/ε ) to achieve a constant multiplicative error. Note that the DP-STAT (Algorithm 3 in [81]) can also be used to estimate the distance. But it requires the knowledge of domain size ∥ w ∗ ∥ Σ + σ . We completely remove this requirement, improve the dependence on K and log( n ) , and show it is also robust, as introduced in the next section. We provide the algorithms and analysis in App. C and the formal proof in App. D.

3.3 Robustness against label corruption

We assume there exists a good dataset S good that satisfies Asmp. 2.1. We only get access to a labelcorrupted dataset under the standard definition of label corruption , e.g., [15]. There are variations in literature on the definition, which we survey in App. A.

Assumption 3.7 ( α corrupt -corruption) . Given a dataset S good = { ( x i , y i ) } n i =1 , an adversary inspects all the data points, selects α corrupt n data points denoted as S r , and replaces the labels with arbitrary labels while keeping the covariates unchanged. We let S bad denote this set of α corrupt n newly labelled examples by the adversary. Let the resulting set be S corrupt := S good ∪ S bad \ S r .

Our goal is to estimate the unknown parameter w ∗ , given corrupted dataset S corrupt , distribution parameter K , and (an upper bound on) the corruption level α corrupt .

Under the non-private scenario , i.e., ε = ∞ , recent advances led to optimal algorithms for linear regression that are robust to label corruptions [15, 21]; if the corruption level is smaller than the target error rate, i.e., α corrupt ≤ α , then n = ˜ O ( d/α 2 ) samples are sufficient to achieve an error rate of (1 /σ ) ∥ ˆ w -w ∗ ∥ Σ = α . The sample complexity of d/α 2 is optimal as it matches the information theoretic lower bound. The condition α corrupt ≤ α is necessary since it is information theoretically impossible to achieve error α less than α corrupt , as we prove in Prop. 3.9. Setting the target error to the minimum possible value of α = α corrupt , we say that these algorithms achieve optimal robustness since the minimum robust error rate of (1 /σ ) ∥ ˆ w -w ∗ ∥ Σ = O ( α corrupt ) can be achieved with minimal sample complexity of n = ˜ O ( d/α 2 corrupt ) . We aim to achieve such optimal robustness simultaneously with differential privacy in a computationally efficient manner.

Theorem 3.8. Under sub-Gaussian model of Asmp. 2.1 and α corrupt -corruption of Asmp. 3.7, if the corruption level is below the target error rate, α ≥ α corrupt , then n = ˜ O ( d/α 2 + κ 1 / 2 d/ ( εα )) samples are sufficient for Alg. 1 to achieve an error rate of (1 /σ 2 ) E [ ∥ ˆ w -w ∗ ∥ 2 Σ ] = ˜ O ( α 2 ) .

This is the first efficient approach to achieve robustness and ( ε, δ ) -DP simultaneously. The existing such algorithms take exponential time [61, Corollary C.2] and [10], but achieve optimal sample complexity of n = O ( d/α 2 + d/ ( εα )) . Notice that there is no dependence on κ . It remains an open question if computationally efficient private linear regression algorithms can achieve such an optimal κ -independent sample complexity. We make the first advance towards this ambitious goal with the above theorem. Our sample complexity is sub-optimal only by a factor of √ κ in the second term. This is achieved by individually clipping the covariate, x i , and the residual, ( w ⊤ t x i -y i ) , in Alg. 1 and carefully tracking the bias of clipping with the use of resilience in the analysis in App. H.

3.4 Lower bounds

Necessity of our assumptions. A tail assumption on the covariate x i such as Asmp. 2.1 is necessary to achieve n = O ( d ) sample complexity in Eq. (4). Even when the covariance Σ is close to identity, without further assumptions on the tail of covariate x , the result in [13] implies that for δ < 1 /n , it is necessary for an ( ε, δ ) -DP estimator to have n = Ω( d 3 / 2 / ( εα )) samples to achieve

d=10, 0=1, K=1, ε=1

=1

=1

Error

10-1

Error

Error

ParameterEstimation

hation

10-3

ParameterEstimation

10-2

Estim

10-4

10-3

10-5

Parameter

10-6

10-3

10-7

105

106

107

10-4

10-3

10-2

10-1

100

100

101

Number ofSamples

a

K

Non Private OLS

DP-SSP

NonPrivateSGD

DP-AMBSSGD

DP-RobGD

DP-RobGD*

∥ ˆ w -w ∗ ∥ Σ = ˜ O ( α ) (see Eq. (3) in [83]). Note that this lower bound is a factor d 1 / 2 larger than our upper bound that benefits from the additional tail assumption.

A tail assumption on the noise z i such as Asmp. 2.1 is necessary to achieve n = O ( d/ ( εα )) dependence on the sample complexity in Eq. (4). For heavy-tailed noise, such as k -th moment bounded noise, the dependence can be significantly larger. [61, Proposition C.5] implies that for δ = e -Θ( d ) and 4 -th moment bounded x i and z i , any ( ε, δ ) -DP estimator requires n = Ω( d/ ( εα 2 )) , which is a factor of 1 /α larger, to achieve (1 /σ 2 ) ∥ ˆ w -w ∗ ∥ Σ = ˜ O ( α ) .

The assumption that only labels are corrupted is critical for Alg. 1. The average of the clipped gradients can be significantly more biased, if the adversary can place the covariates of the corrupted samples in the same direction. In particular, the bound on the bias of our gradient step in Eq. (44) in App. H would no longer hold. Against such strong attacks, one requires additional steps to estimate the mean of the gradients robustly and privately, similar to those used in robust private mean estimation [60, 56, 44, 8]. There is no known linear-time algorithm to achieve this, and this is outside the scope of this paper.

Lower bounds under label corruption. Under the α corrupt label corruption setting (Asmp. 3.7), even with infinite data and without privacy constraints, no algorithm is able to learn w ∗ with ℓ 2 error better than α corrupt . We provide a formal derivation for completeness.

Proposition 3.9. Let D Σ ,σ 2 ,w ∗ ,K be a class of distributions on ( x i , y i ) from sub-Gaussian model in Asmp. 2.1. Let S n,α be an α -corrupted dataset of n i.i.d. samples from some distribution D ∈ D Σ ,σ 2 ,w ∗ ,K under Asmp. 3.7. Let M be a class of estimators that are functions over S n,α . Then there exists a constant c such that min n, ˆ w ∈M max S n,α , D∈D Σ ,σ 2 ,w ∗ ,K ,w ∗ ,K E [ ∥ ˆ w -w ∗ ∥ 2 Σ ] ≥ c α 2 σ 2 .

A proof is provided in App. I.1. A similar lower bound can be found in [11, Theorem 6.1].

4 Experimental results

4.1 DP Linear Regression

We present experimental results comparing our proposed technique (DP-ROBGD) with other baselines. We consider non-corrupted regression in this section and defer corrupted regression to the App. K. We begin by describing the problem setup and the baseline algorithms first.

Experiment Setup. We generate data for all the experiments using the following generative model. The parameter vector w ∗ is uniformly sampled from the surface of a unit sphere. The covariates { x i } n i =1 are first sampled from N (0 , Σ) and then projected to unit sphere. We consider diagonal covariances Σ of the following form: Σ[0 , 0] = κ , and Σ[ i, i ] = 1 for all i ≥ 1 . Here κ ≥ 1 is the condition number of Σ . We generate noise z i from uniform distribution over [ -σ, σ ] . Finally, the response variables are generated as follows y i = x ⊤ i w ∗ + z i . All the experiments presented below are repeated 5 times and the averaged results are presented. We set the DP parameters ( ϵ, δ ) as ϵ = 1 , δ = min(10 -6 , n -2 ) . Experiments for ϵ = 0 . 1 can be found in Fig. 2 in the App. K. Baseline Algorithms. We compare our estimator with the following baseline algorithms:

Non private algorithms: ordinary least squares (OLS), one-pass stochastic gradient descent with tail-averaging (SGD). For SGD, step-size is 1 / (2 λ max ) and minibatch size is n/T , where T = 3 κ log n .

Private algorithms: sufficient statistics perturbation (DP-SSP) [38, 83], differentially private stochastic gradient descent (DP-AMBSSGD) [81]. DP-SSP had the best empirical performance among numerous techniques studied by [83], and DP-AMBSSGD has the best known theoretical guarantees. The DP-SSP algorithm involves releasing X T X and X T y differentially privately and computing ( ̂ X T X ) -1 ̂ X T y . DP-AMBSSGD is a private version of SGD where the DP noise is set adaptively according to the excess error in each iteration. For both algorithms, we use the hyper-parameters recommended in their respective papers. To improve the performance of DP-AMBSSGD , we reduce the theoretical clipping threshold by a constant factor.

DP-ROBGD. We implement Alg. 1 with the following key changes. Instead of relying on PrivateNormEstimator to estimate Γ , we set it to its true value Tr(Σ) . This is done for a fair comparison with DP-AMBSSGD which assumes the knowledge of Tr(Σ) . Next, we use 20% of the samples to compute γ t in line 5 (instead of the 50% stated in Alg. 1). In our experiments we also present results for a variant of our algorithm called DP-ROBGD* which outputs the best iterate based on γ t , instead of the last iterate. One could also perform tail-averaging instead of picking the best iterate. Both these modifications are primarily used to reduce the variance in the output of Alg. 1 and achieved similar performance in our experiments.

Results. Figure 1 presents the performance of various algorithms as we vary n, κ, σ . It can be seen that DP-ROBGD outperforms DP-AMBSSGD in almost all the settings (and DP-ROBGD* outperforms DP-ROBGD in all cases). DP-SSP has poor performance when the noise σ is low, but performs slightly better than DP-ROBGD in other settings. A major drawback of DP-SSP is its computational complexity which scales as O ( nd 2 + d ω ) . In contrast, the computational complexity of DP-ROBGD has smaller dependence on d and scales as ˜ O ( ndκ ) . Thus the latter is more computationally efficient for high-dimensional problems. More experimental results on both robust and private linear regression can be found in the App. K.

5 Sketch of the main ideas in the analysis

We provide the main ideas behind the proof of Thm. 3.1. The privacy proof is straightforward since no matter what clipping threshold we use the noise we add is always proportionally to the clipping threshold which guarantees privacy. In the remainder, we focus on the utility analysis.

The proof of the utility heavily relies on the resilience [73] (also known as stability [27]), which states that given a large enough sample set S , various statistics (for example, sample mean and sample variance) of any large enough subset of S will be close to each other. We define resilience as follows.

Definition 5.1 ([61, Definition 23]) . For some α ∈ (0 , 1) , ρ 1 ∈ R + , ρ 2 ∈ R + , and ρ 3 ∈ R + , ρ 4 ∈ R + , we say dataset S good = { ( x i ∈ R d , y i ∈ R ) } n i =1 is ( α, ρ 1 , ρ 2 , ρ 3 , ρ 4 ) -resilient with respect to ( w ∗ , Σ , σ ) for some w ∗ ∈ R d , positive definite Σ ≻ 0 ∈ R d × d , and σ > 0 if for any T ⊂ S good of size | T | ≥ (1 -α ) n , the following holds for all v ∈ R d :

We give an overview of the proof for non-robust case as follows. First, we introduce some notations. Let g ( t ) i := ( x ⊤ i w t -y i ) x i be the raw gradient and ˜ g ( t ) i := clip Θ ( x i )clip θ t ( x ⊤ i w t -y i ) be the clipped gradient. Note that when the data follows from our distributional assumption, with high probability, samples are not clipped by the norm: clip Θ ( x i ) = x i . We can write down one step of gradient update

(see Alg. 1) as follows:

In the above equation, the first term is a contraction, meaning w t is moving toward w ∗ . The second term captures the noise from the randomness in the samples. The third term captures the bias introduced by the clipping operation, and the fourth term captures the added noise for privacy. The second term is standard and relatively easy to control, and our main focus is on the last two terms.

The third term ( η/n ) ∑ i ∈ S ( g ( t ) i -˜ g ( t ) i ) can be controlled using the resilience property. We prove that with our estimated threshold, the clipping will only affect a small amount of datapoints, whose contribution to the gradient is small collectively.

Now we have controlled the deterministic bias. Then, we upper bound the fourth term, which is the noise for the purpose of privacy, and show the expected prediction error decrease in every gradient step. The difficulty is that, since our clipping threshold is adaptive, the decrease of the estimation error depends on the estimation error of all the previous steps. This causes that in some iterations, the estimation error actually increases. In order to get around this, we split the iterations into length κ chunks, and argue that the maximum estimation error in a chunk must be a constant factor smaller than the previous chunk. This implies we will reach the desired error within ˜ O ( κ ) steps.

6 Conclusion

We provide a novel variant of DP-SGD algorithm for differentially private linear regression under label corruption. We show the first near-optimal rate that achieves privacy and robustness to label corruptions simultaneously. When there is no label corruption, our result also improves upon the state-of-the-art method [81] in terms of the condition number κ . Compared to [81], our algorithm has two innovations: 1 ) we introduce a novel adaptive clipping, which is critical in achieving robustness against label corruptions; and 2 ) we use full batch gradient descent and a novel convergence analysis to get the near-optimal sample complexity.

References

Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pages 308-318, 2016.

John M Abowd. The US Census Bureau adopts differential privacy. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 2867-2867, 2018.

Anish Agarwal, Devavrat Shah, Dennis Shen, and Dogyoon Song. On robustness of principal component regression. Advances in Neural Information Processing Systems , 32, 2019.

Daniel Alabi, Pravesh K Kothari, Pranay Tankala, Prayaag Venkat, and Fred Zhang. Privately estimating a gaussian: Efficient, robust and optimal. arXiv preprint arXiv:2212.08018 , 2022.

Daniel Alabi, Audra McMillan, Jayshree Sarathy, Adam Smith, and Salil Vadhan. Differentially private simple linear regression. arXiv preprint arXiv:2007.05157 , 2020.

Kareem Amin, Matthew Joseph, Mónica Ribero, and Sergei Vassilvitskii. Easy differentially private linear regression. arXiv preprint arXiv:2208.07353 , 2022.

Galen Andrew, Om Thakkar, Brendan McMahan, and Swaroop Ramaswamy. Differentially private learning with adaptive clipping. Advances in Neural Information Processing Systems , 34, 2021.

Hassan Ashtiani and Christopher Liaw. Private and polynomial time algorithms for learning gaussians and beyond. In Conference on Learning Theory , pages 1075-1076. PMLR, 2022.

Hilal Asi, Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimization: Optimal rates in l1 geometry. In International Conference on Machine Learning , pages 393-403. PMLR, 2021.

Hilal Asi, Jonathan Ullman, and Lydia Zakynthinou. From robustness to privacy and back. arXiv preprint arXiv:2302.01855 , 2023.

Ainesh Bakshi and Adarsh Prasad. Robust linear regression: Optimal rates in polynomial time. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 102-115, 2021.

Raef Bassily, Vitaly Feldman, Kunal Talwar, and Abhradeep Guha Thakurta. Private stochastic convex optimization with optimal rates. Advances in Neural Information Processing Systems , 32, 2019.

Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th Annual Symposium on Foundations of Computer Science , pages 464-473. IEEE, 2014.

Kush Bhatia, Prateek Jain, Parameswaran Kamalaruban, and Purushottam Kar. Consistent robust regression. Advances in Neural Information Processing Systems , 30, 2017.

Kush Bhatia, Prateek Jain, and Purushottam Kar. Robust regression via hard thresholding. Advances in neural information processing systems , 28, 2015.

T Tony Cai, Yichen Wang, and Linjun Zhang. Score attack: A lower bound technique for optimal differentially private learning. arXiv preprint arXiv:2303.07152 , 2023.

Kamalika Chaudhuri, Claire Monteleoni, and Anand D Sarwate. Differentially private empirical risk minimization. Journal of Machine Learning Research , 12(3), 2011.

Yu Cheng, Ilias Diakonikolas, and Rong Ge. High-dimensional robust mean estimation in nearly-linear time. In Proceedings of the thirtieth annual ACM-SIAM symposium on discrete algorithms , pages 2755-2771. SIAM, 2019.

Yeshwanth Cherapanamjeri, Efe Aras, Nilesh Tripuraneni, Michael I Jordan, Nicolas Flammarion, and Peter L Bartlett. Optimal robust linear regression in nearly linear time. arXiv preprint arXiv:2007.08137 , 2020.

Christopher A. Choquette-Choo, Krishnamurthy Dvijotham, Krishna Pillutla, Arun Ganesh, Thomas Steinke, and Abhradeep Thakurta. Correlated noise provably beats independent noise for differentially private learning, 2023.

Arnak Dalalyan and Philip Thompson. Outlier-robust estimation of a sparse linear model using ℓ 1 -penalized huber's m-estimator. Advances in neural information processing systems , 32, 2019.

Jules Depersin. A spectral algorithm for robust regression with subgaussian rates. arXiv preprint arXiv:2007.06072 , 2020.

Ilias Diakonikolas, Samuel B Hopkins, Daniel Kane, and Sushrut Karmalkar. Robustly learning any clusterable mixture of gaussians. arXiv preprint arXiv:2005.06417 , 2020.

Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Ankur Moitra, and Alistair Stewart. Robust estimators in high-dimensions without the computational intractability. SIAM Journal on Computing , 48(2):742-864, 2019.

Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Jacob Steinhardt, and Alistair Stewart. Sever: A robust meta-algorithm for stochastic optimization. In International Conference on Machine Learning , pages 1596-1606, 2019.

Ilias Diakonikolas, Gautam Kamath, Daniel M Kane, Jerry Li, Ankur Moitra, and Alistair Stewart. Being robust (in high dimensions) can be practical. In International Conference on Machine Learning , pages 999-1008. PMLR, 2017.

Ilias Diakonikolas and Daniel M Kane. Recent advances in algorithmic high-dimensional robust statistics. arXiv preprint arXiv:1911.05911 , 2019.

Ilias Diakonikolas, Daniel M Kane, and Alistair Stewart. List-decodable robust mean estimation and learning mixtures of spherical gaussians. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing , pages 1047-1060, 2018.

Ilias Diakonikolas, Weihao Kong, and Alistair Stewart. Efficient algorithms and lower bounds for robust linear regression. In Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 2745-2754. SIAM, 2019.

Christos Dimitrakakis, Blaine Nelson, Aikaterini Mitrokotsa, and Benjamin IP Rubinstein. Robust and private bayesian inference. In International Conference on Algorithmic Learning Theory , pages 291-305. Springer, 2014.

Yihe Dong, Samuel Hopkins, and Jerry Li. Quantum entropy scoring for fast robust mean estimation and improved outlier detection. Advances in Neural Information Processing Systems , 32, 2019.

Cynthia Dwork and Jing Lei. Differential privacy and robust statistics. In Proceedings of the forty-first annual ACM symposium on Theory of computing , pages 371-380, 2009.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference , pages 265-284. Springer, 2006.

Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science , 9(3-4):211-407, 2014.

Úlfar Erlingsson, Vasyl Pihur, and Aleksandra Korolova. Rappor: Randomized aggregatable privacy-preserving ordinal response. In Proceedings of the 2014 ACM SIGSAC conference on computer and communications security , pages 1054-1067, 2014.

Giulia Fanti, Vasyl Pihur, and Úlfar Erlingsson. Building a rappor with the unknown: Privacypreserving learning of associations and data dictionaries. Proceedings on Privacy Enhancing Technologies , 2016(3):41-61, 2016.

Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimization: optimal rates in linear time. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing , pages 439-449, 2020.

James Foulds, Joseph Geumlek, Max Welling, and Kamalika Chaudhuri. On the theory and practice of privacy-preserving bayesian data analysis. arXiv preprint arXiv:1603.07294 , 2016.

Arun Ganesh, Mahdi Haghifam, Thomas Steinke, and Abhradeep Thakurta. Faster differentially private convex optimization via second-order methods. arXiv preprint arXiv:2305.13209 , 2023.

Arun Ganesh, Daogao Liu, Sewoong Oh, and Abhradeep Thakurta. Private (stochastic) nonconvex optimization revisited: Second-order stationary points and excess risks. arXiv preprint arXiv:2302.09699 , 2023.

Quan Geng, Peter Kairouz, Sewoong Oh, and Pramod Viswanath. The staircase mechanism in differential privacy. IEEE Journal of Selected Topics in Signal Processing , 9(7):1176-1184, 2015.

Antonious M Girgis, Deepesh Data, Suhas Diggavi, Ananda Theertha Suresh, and Peter Kairouz. On the renyi differential privacy of the shuffle model. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security , pages 2321-2341, 2021.

Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz. Numerical composition of differential privacy. Advances in Neural Information Processing Systems , 34:11631-11642, 2021.

Samuel B Hopkins, Gautam Kamath, and Mahbod Majid. Efficient mean estimation with pure differential privacy via a sum-of-squares exponential mechanism. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , pages 1406-1417, 2022.

Samuel B Hopkins, Gautam Kamath, Mahbod Majid, and Shyam Narayanan. Robustness implies privacy in statistical estimation. arXiv preprint arXiv:2212.05015 , 2022.

Peter J Huber. Robust estimation of a location parameter. In Breakthroughs in statistics , pages 492-518. Springer, 1992.

Arun Jambulapati, Jerry Li, and Kevin Tian. Robust sub-gaussian principal component analysis and width-independent schatten packing. Advances in Neural Information Processing Systems , 33, 2020.

Peter Kairouz, Sewoong Oh, and Pramod Viswanath. Extremal mechanisms for local differential privacy. Advances in neural information processing systems , 27, 2014.

Peter Kairouz, Sewoong Oh, and Pramod Viswanath. The composition theorem for differential privacy. In International conference on machine learning , pages 1376-1385. PMLR, 2015.

Gautam Kamath, Xingtu Liu, and Huanyu Zhang. Improved rates for differentially private stochastic convex optimization with heavy-tailed data. arXiv preprint arXiv:2106.01336 , 2021.

Vishesh Karwa and Salil Vadhan. Finite sample differentially private confidence intervals. arXiv preprint arXiv:1711.03908 , 2017.

Ashish Khetan, Zachary C Lipton, and Animashree Anandkumar. Learning from noisy singlylabeled data. In International Conference on Learning Representations , 2018.

Daniel Kifer, Adam Smith, and Abhradeep Thakurta. Private convex empirical risk minimization and high-dimensional regression. In Conference on Learning Theory , pages 25-1. JMLR Workshop and Conference Proceedings, 2012.

Adam Klivans, Pravesh K Kothari, and Raghu Meka. Efficient algorithms for outlier-robust regression. In Conference On Learning Theory , pages 1420-1430. PMLR, 2018.

Weihao Kong, Rajat Sen, Pranjal Awasthi, and Abhimanyu Das. Trimmed maximum likelihood estimation for robust learning in generalized linear models. arXiv preprint arXiv:2206.04777 , 2022.

Pravesh K Kothari, Pasin Manurangsi, and Ameya Velingker. Private robust estimation by stabilizing convex relaxations. arXiv preprint arXiv:2112.03548 , 2021.

Arun Kumar Kuchibhotla and Abhishek Chakrabortty. Moving beyond sub-gaussianity in high-dimensional statistics: Applications in covariance estimation and linear regression. arXiv preprint arXiv:1804.02605 , 2018.

Janardhan Kulkarni, Yin Tat Lee, and Daogao Liu. Private non-smooth empirical risk minimization and stochastic convex optimization in subquadratic steps. arXiv preprint arXiv:2103.15352 , 2021.

Xiyang Liu, Weihao Kong, Prateek Jain, and Sewoong Oh. DP-PCA: Statistically optimal and differentially private pca. In Advances in Neural Information Processing Systems , 2022.

Xiyang Liu, Weihao Kong, Sham Kakade, and Sewoong Oh. Robust and differentially private mean estimation. Advances in Neural Information Processing Systems , 34, 2021.

Xiyang Liu, Weihao Kong, and Sewoong Oh. Differential privacy and robust statistics in high dimensions. In Conference on Learning Theory , pages 1167-1246. PMLR, 2022.

Frank D McSherry. Privacy integrated queries: an extensible platform for privacy-preserving data analysis. In Proceedings of the 2009 ACM SIGMOD International Conference on Management of data , pages 19-30, 2009.

Jason Milionis, Alkis Kalavasis, Dimitris Fotakis, and Stratis Ioannidis. Differentially private regression with unbounded covariates. In International Conference on Artificial Intelligence and Statistics , pages 3242-3273. PMLR, 2022.

Kentaro Minami, HItomi Arai, Issei Sato, and Hiroshi Nakagawa. Differential privacy without sensitivity. In Advances in Neural Information Processing Systems , pages 956-964, 2016.

Darakhshan J Mir. Differential privacy: an exploration of the privacy-utility landscape . Rutgers The State University of New Jersey-New Brunswick, 2013.

Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF) , pages 263-275. IEEE, 2017.

Ankit Pensia, Varun Jog, and Po-Ling Loh. Robust regression with covariate filtering: Heavy tails and adversarial contamination. arXiv preprint arXiv:2009.12976 , 2020.

Adarsh Prasad, Arun Sai Suggala, Sivaraman Balakrishnan, and Pradeep Ravikumar. Robust estimation via robust gradient estimation. arXiv preprint arXiv:1802.06485 , 2018.

Holger Sambale. Some notes on concentration for α -subexponential random variables. arXiv preprint arXiv:2002.10761 , 2020.

Or Sheffet. Old techniques in differentially private linear regression. In Algorithmic Learning Theory , pages 789-827. PMLR, 2019.

Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with differentially private updates. In 2013 IEEE global conference on signal and information processing , pages 245-248. IEEE, 2013.

Shuang Song, Om Thakkar, and Abhradeep Thakurta. Characterizing private clipped gradient descent on convex generalized linear problems. arXiv preprint arXiv:2006.06783 , 2020.

Jacob Steinhardt, Moses Charikar, and Gregory Valiant. Resilience: A criterion for learning in the presence of arbitrary outliers. arXiv preprint arXiv:1703.04940 , 2017.

Thomas Steinke and Alexander Knop. Differentially Private Linear Regression via Medians. https://openreview.net/pdf?id=JSBgIaxAXk9 , 2022. [Online; accessed 28-April2023].

Arun Sai Suggala, Kush Bhatia, Pradeep Ravikumar, and Prateek Jain. Adaptive hard thresholding for near-optimal consistent robust regression. In Conference on Learning Theory , pages 2892-2897. PMLR, 2019.

Jun Tang, Aleksandra Korolova, Xiaolong Bai, Xueqiang Wang, and Xiaofeng Wang. Privacy loss in apple's implementation of differential privacy on macos 10.12. arXiv preprint arXiv:1709.02753 , 2017.

Henri Theil. A rank-invariant method of linear and polynomial regression analysis. Indagationes mathematicae , 12(85):173, 1950.

Kiran K Thekumparampil, Ashish Khetan, Zinan Lin, and Sewoong Oh. Robustness of conditional gans to noisy labels. Advances in neural information processing systems , 31, 2018.

John W Tukey. Mathematics and the picturing of data. In Proceedings of the International Congress of Mathematicians, Vancouver, 1975 , volume 2, pages 523-531, 1975.

John W Tukey and Donald H McLaughlin. Less vulnerable confidence and significance procedures for location based on a single sample: Trimming/winsorization 1. Sankhy¯ a: The Indian Journal of Statistics, Series A , pages 331-352, 1963.

Prateek Varshney, Abhradeep Thakurta, and Prateek Jain. (nearly) optimal private linear regression via adaptive clipping. arXiv preprint arXiv:2207.04686 , 2022.

Duy Vu and Aleksandra Slavkovic. Differential privacy for clinical trial data: Preliminary evaluations. In 2009 IEEE International Conference on Data Mining Workshops , pages 138-143. IEEE, 2009.

Yu-Xiang Wang. Revisiting differentially private linear regression: optimal and adaptive prediction & estimation in unbounded domain. arXiv preprint arXiv:1803.02596 , 2018.

Yu-Xiang Wang, Borja Balle, and Shiva Prasad Kasiviswanathan. Subsampled rényi differential privacy and analytical moments accountant. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1226-1235. PMLR, 2019.

Yu-Xiang Wang, Stephen Fienberg, and Alex Smola. Privacy for free: Posterior sampling and stochastic gradient monte carlo. In International Conference on Machine Learning , pages 2493-2502. PMLR, 2015.

Xi Wu, Fengan Li, Arun Kumar, Kamalika Chaudhuri, Somesh Jha, and Jeffrey Naughton. Bolton differential privacy for scalable stochastic gradient descent-based analytics. In Proceedings of the 2017 ACM International Conference on Management of Data , pages 1307-1322, 2017.

Liang Zhang, Kiran Koshy Thekumparampil, Sewoong Oh, and Niao He. Bring your own algorithm for optimal differentially private stochastic minimax optimization. arXiv preprint arXiv:2206.00363 , 2022.

Liang Zhang, Kiran Koshy Thekumparampil, Sewoong Oh, and Niao He. DPZero: dimension-independent and differentially private zeroth-order optimization. arXiv preprint arXiv:2310.09639 , 2023.

Banghua Zhu, Jiantao Jiao, and Jacob Steinhardt. Generalized resilience and robust statistics. arXiv preprint arXiv:1909.08755 , 2019.

Yuqing Zhu, Jinshuo Dong, and Yu-Xiang Wang. Optimal accounting of differential privacy via characteristic function. In International Conference on Artificial Intelligence and Statistics , pages 4782-4817. PMLR, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the problem of $(\\varepsilon, \\delta)$-differently private linear regression. The covariates are drawn from a Gaussian distribution $N(0,\\Sigma)$ with unknown covariance $\\Sigma$ and the random noise in the model is either Gaussian or heavier-tailed. Further, a small fraction of the labels can be corrupted. The error is measured as $\\lVert \\Sigma^{1/2} (\\hat{w} - w^*) \\rVert_2$ (when the variance of the random noise equals 1), where $w^*$ is the true regression vector.\n\nLet $\\kappa$ be the condition number of $\\Sigma$. For the case that the noise is Gaussian the authors propose an algorithm that achieves error $\\alpha$ in polynomial time using $\\tilde{O}(\\tfrac d {\\alpha^2} + \\sqrt{\\kappa} \\cdot \\tfrac d {\\varepsilon \\alpha})$ samples. Previous efficient algorithms for this task needed $\\tilde{O}(\\tfrac d {\\alpha^2} + \\kappa \\cdot \\tfrac d {\\varepsilon \\alpha} + \\kappa^2 \\tfrac d \\varepsilon)$. On the other hand, it is known that  $\\Omega(\\tfrac d {\\alpha^2} + \\tfrac d {\\varepsilon \\alpha})$ samples are necessary information-theoretically. This can be achieved by inefficient algorithms, i.e., there is no dependence on the condition number.\n\nTheir results for heavy-tailed noise are similar to the results for Gaussian noise. Previously there were no known results in this setting. Their algorithm is based on differentially private stochastic gradient descent (DP-SGD).\n\nSTRENGTHS:\nPrevious works have also used DP-SGD for this problem. The authors give a more careful analysis (and tweak it in some places) which yields an algorithm with an improved sample complexity. One of their tweaks allows them to also handle corruptions in the labels. It is nice to see that this algorithmic approach can be made robust to corruptions in the labels.\n\nOverall, the paper is well-written.\n\nWEAKNESSES:\nQualitatively, there is still a dependence on the condition number. So this work does not give new insights into whether this is necessary or not.\n\nQUESTIONS:\nSince both you and previous work analyze DP-SGD and incur (some) dependence on the condition number, is there any reason to believe that this approach might inherently have at least some dependence on this?\n\nLIMITATIONS:\nThe authors mention open questions that are note answered by their work in multiple places.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper introduces the first computationally and statistically efficient algorithm for differentially private and label-corruption robust linear regression. In particular, they improve over the best sample complexity achieved by non-robust algorithms in terms of the condition number of the covariance matrix. This is achieved through a variant of DP-SGD with a new adaptive clipping technique. The authors also show the empirical improvement of their proposed method over existing techniques on a synthetic dataset.

STRENGTHS:
* Technically solid paper on an important topic. Some of the analysis methods are original and interesting.
* The paper is well written and the contributions are clear.

WEAKNESSES:
### Weaknesses:

* It is clear that there is an improvement in terms of sample complexity, but my main concern is the significance of the improvement, especially that a polynomial dependence on the condition number remains, which may be extremely large. Do you think it is possible to suppress the dependence on the condition number, and how?

* Is there any novelty in the proof of the lower bound in Proposition 3.9 compared to [11]?

* Can you show the exact runtimes of your algorithm compared to that of [78]?

* While I understand that the paper is mainly theoretical, I fail to think of any practical scenarios where an adversary can only corrupt labels.



### Minor comments:

* Typos: line 178: proportional -> proportionally, line 363: increase -> increases
* Equation 3: $\nu_t$ -> $\nu_T$
* Definition 2.2: (suggestion) random variables are typically denoted with capital letters
* Lemma B.2: $\infty$ -> {$\infty$}

QUESTIONS:
Refer to Weaknesses section for questions.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper focuses on differentially private linear regression with potential label corruption. The proposed algorithm is based on DP-SGD and introduces new components such as full batch-size GD and adaptive norm clipping. The paper achieves the best sample complexity under the assumption of sub-Gaussian data distribution in the absence of corruption, with a polynomial running time. Additionally, it presents the first algorithm for DP linear regression in the presence of label corruption, supported by experimental validation.

STRENGTHS:
The paper is well-structured and easy to follow. Privacy and robustness are crucial in current machine learning research, making the studied topic highly relevant. The analysis of adaptive norm clipping introduces a novel technique that could have implications for future research. The results obtained in the paper are promising, and the proposed algorithm makes a substantial contribution to the field.

WEAKNESSES:
The paper's focus on a specific distribution requirement for the samples and only considering corruption in the labels limits its generalizability. It would be beneficial to explore more general cases.

QUESTIONS:
- Line 181: Should it be "$\Theta(\theta_t)$" instead?
- In Algorithm 1, why are the three subsets equally divided? Is it possible to improve performance by adjusting their sizes?

LIMITATIONS:
Yes, limitations are discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a variant of the Differentially Private Stochastic Gradient Descent (DP-SGD) algorithm aimed at achieving efficient linear regression under differential privacy, even in scenarios where a portion of response variables is adversarially corrupted. The introduced methodology involves a full-batch gradient descent to optimize sample complexity and an adaptive clipping mechanism for robustness. The algorithm, exhibiting linear time complexity with respect to input size, also matches the theoretically optimal sample complexity, considering a distribution-dependent condition number factor.\n\nSTRENGTHS:\nThe paper presents a focused yet intriguing investigation into the Label Robust and Differentially Private Linear Regression problem.\n\nThe proposed algorithm demonstrates improved sample complexity relative to existing polynomial-time algorithms.\n\nWEAKNESSES:\nThe presentation of experimental results appears overly simplistic, resulting in some difficulties in interpretation. The study would greatly benefit from a more comprehensive representation of data, rather than relying solely on a single figure (Figure 1). Additionally, only one value of $\\kappa$ is used.\n\nQUESTIONS:\nThe authors assert that their methodology represents \"the first efficient approach to achieve robustness and (ε, δ)-Differential Privacy (DP) simultaneously\". However, previous research, as indicated in references [1] and [2], appears to have discussed the interplay between DP and robustness. Could the authors clarify the novel aspects of their approach in the context of these prior works, if any?\n\nThe manuscript refers to a variant of Stochastic Gradient Descent (SGD) being utilized in the process. Is this variation generalizable, or is it specifically tailored for the application to Linear Regression models?\n\n\n[1] Phan, H., Thai, M. T., Hu, H., Jin, R., Sun, T., & Dou, D. (2020, November). Scalable differential privacy with certified robustness in adversarial learning. In International Conference on Machine Learning (pp. 7683-7694). PMLR.\n[2] Lecuyer, M., Atlidakis, V., Geambasu, R., Hsu, D., & Jana, S. (2019, May). Certified robustness to adversarial examples with differential privacy. In 2019 IEEE symposium on security and privacy (SP) (pp. 656-672). IEEE.\n\nLIMITATIONS:\nExperimental results are overly simplistic.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}