SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiyang Liu
Paul Allen School of Computer Science & Engineering
xiyang1@cs.washington.edu
Weihao Kong
weihaokong@google.com
University of Washington, and Google Research
sewoong@cs.washington.edu
We study the canonical problem of linear regression under  $(\varepsilon, \delta)$ -differential privacy when the datapoints are sampled i.i.d. from a distribution and a fraction of response variables are adversarially corrupted. We provide the first provably efficient - both computationally and statistically - method for this problem, assuming standard assumptions on the data distribution. Our algorithm is a variant of the popular differentially private stochastic gradient descent (DP-SGD) algorithm with two key innovations: a full-batch gradient descent to improve sample complexity and a novel adaptive clipping to guarantee robustness. Our method requires only linear time in input size, and still matches the information theoretical optimal sample complexity up to a data distribution dependent condition number factor. Interestingly, the same algorithm, when applied to a setting where there is no adversarial corruption, still improves upon the existing state-of-the-art and achieves a near optimal sample complexity.
Differential Privacy (DP) [33] is a standard notion of privacy widely adopted by both industry and government [76, 35, 36, 2]. With widespread usage of ML and statistical techniques, DP becomes even more critical to ensure private information of participating individuals is not revealed in any form via the learned model. An statistical estimator is said to be  $(\varepsilon, \delta)$ -differentially private if presence/absence of an individual's data point in the dataset does not significantly change the estimated output. Smaller  $\varepsilon > 0$  and  $\delta \in [0,1]$  imply stronger privacy guarantees.
While privacy preserving statistical estimators have been studied extensively in recent past, several critical questions remain open (see App. A for a survey). Consider the canonical statistical task of linear regression with  $n$  i.i.d. samples,  $\{(x_{i} \in \mathbb{R}^{d}, y_{i} \in \mathbb{R})\}_{i=1}^{n}$ , drawn from  $x_{i} \sim \mathcal{N}(0, \Sigma)$ ,  $y_{i} = x_{i}^{\top} w^{*} + z_{i}$ ,  $z_{i} \sim \mathcal{N}(0, \sigma^{2})$  and  $\mathbb{E}[x_{i}z_{i}] = 0$  for some true parameter  $w^{*} \in \mathbb{R}^{d}$ . The error is measured in  $(1 / \sigma) \| \hat{w} - w^{*} \|_{\Sigma} := (1 / \sigma) \| \Sigma^{1/2} (\hat{w} - w^{*}) \|$ , which correctly accounts for the signal-to-noise ratio in each direction; in the direction of large eigenvalue of  $\Sigma$ , we have larger signal in  $x_{i}$
but the noise  $z_{i}$  remains the same. We expect smaller errors in those directions, which is accounted for in the error measure  $(1 / \sigma)\| \hat{w} -w^{*}\|_{\Sigma}$ .
Minimax optimal sample complexity for estimating the optimal linear regression model with DP was recently established. For the lower bound, using recently introduced score attack technique, [16, Theorem 3.1] shows that  $n = \Omega (d / \alpha^2 + d / (\varepsilon \alpha))$  samples are necessary to achieve an error of  $(1 / \sigma)\| \hat{w} - w^{*}\|_{\Sigma} = \alpha$  (in expectation). For the matching upper bound, High-dimensional Propose-Test-Release (HPTR) in [61] and Robust-to-Private in [10] show that  $n = \tilde{O}(d / \alpha^2 + d / (\varepsilon \alpha))$  samples are also sufficient. The first term of  $d / \alpha^2$  is the fundamental sample complexity even if privacy is not required, and the second term of  $d / (\varepsilon \alpha)$  is the cost of privacy.
This implies that, statistically, the problem appears to be solved. However, computationally, the problem is still open despite multiple studies of the problem. That is, the statistical optimal algorithms still take exponential time.
After a series of efforts in computationally efficient approaches as surveyed in App. A, [81] achieves the best known sample complexity of  $n = \tilde{O}(d / \alpha^2 + \kappa d / (\varepsilon \alpha) + \kappa^2 d / \varepsilon)$ , where  $\kappa$  is the condition number of the covariance  $\Sigma$  of the covariates. Compared to HPTR, the cost of computational efficiency is factor of  $\kappa$  in the second term and the third term that is unnecessary. As the condition number can be quite large, improving the dependence on  $\kappa$  is of utmost importance. Furthermore, the technique of [81] strictly requires sampling without replacement, whose analysis relies on having an explicit form of the end-to-end update. In particular, their analysis technique is not applicable to the case with corrupted samples.
In contrast, we propose a novel method (Alg. 1) that builds upon full-batch gradient descent and applies a carefully chosen adaptive clipping which is a general technique used in practice as well [1]. Together with an intuitive but intricate analysis technique, we improve the sample complexity to  $n = \tilde{O}(d / \alpha^2 + \kappa^{1/2} d / (\varepsilon \alpha))$ .
Corollary 1.1 (Corollary of Thm. 3.1 for sub-Gaussian data). Alg. 1 is  $(\varepsilon, \delta)$ -DP. Let  $S = \{(x_i, y_i)\}_{i=1}^n$  be a dataset of i.i.d. samples with  $x_i \sim \mathcal{N}(0, \Sigma)$ ,  $y_i = x_i^\top w^* + z_i$  and  $z_i \sim \mathcal{N}(0, \sigma^2)$  for some unknown true parameter  $w^* = \Sigma^{-1}\mathbb{E}[y_i x_i] \in \mathbb{R}^d$  and unknown  $\Sigma$  and  $\sigma^2$ . Then  $n = \tilde{O}(d / \alpha^2 + \kappa^{1/2} d / (\varepsilon \alpha))$  samples are sufficient for Alg. 1 to achieve  $(1 / \sigma) \| \hat{w} - w^* \|_{\Sigma} = \tilde{O}(\alpha)$  with high probability, where  $\kappa := \lambda_{\max}(\Sigma) / \lambda_{\min}(\Sigma)$ .
Due to space constraints, we focus on sub-Gaussian distributions in the main text and provide comparisons to prior work in Tab. 1. Our analysis in App. H applies to a more general family of light-tailed distributions, called sub-Weibull. Next, when the noise in the samples is heavy-tailed, a similar algorithm can be applied with carefully chosen clipping thresholds to account for the heavier tail. Concretely, for  $k$ -th moment bounded distributions, the tail of the distribution gets increasingly heavier with smaller  $k$ . This would require larger numbers of samples to achieve the same accuracy, which is captured in our sample complexity of  $n = \tilde{O}(d / \alpha^{2k / (k - 1)} + \kappa^{1/2} d / (\varepsilon \alpha^{k / (k - 1)}))$ . We explain the heavy-tailed setting, provide a detailed analysis and a proof, and discuss the results in App. L. This is the first efficient algorithm with provable guarantees achieving  $(\varepsilon, \delta)$ -DP.
Corollary 1.2 (informal version of Coro. L.7 for heavy-tailed noise). Alg. 4 is  $(\varepsilon, \delta)$ -DP. Let  $S = \{(x_i, y_i)\}_{i=1}^n$  be a dataset of i.i.d. samples with  $x_i \sim \mathcal{N}(0, \Sigma)$ ,  $y_i = x_i^\top w^* + z_i$ , and the zero-mean, independent, and heavy-tailed noise  $z_i$  satisfies  $\mathbb{E}[|z / \sigma|^k] = O(1)$  for some unknown true parameter  $w^* \in \mathbb{R}^d$  and unknown  $\Sigma$  and  $\sigma^2$ . Then  $n = \tilde{O}(d / \alpha^{2k/(k-1)} + \kappa^{1/2} d / (\varepsilon \alpha^{k/(k-1)}))$  samples are sufficient for Alg. 4 in App. L to achieve an error rate of  $(1/\sigma) \| \hat{w} - w^* \|_\Sigma = \tilde{O}(\alpha)$  with high probability, where  $\kappa := \lambda_{\max}(\Sigma) / \lambda_{\min}(\Sigma)$ .
Perhaps surprisingly, we show that Alg. 1 is also robust against label-corruption, where an adversary selects an arbitrary  $\alpha_{\mathrm{corrupt}}$  fraction of the data points and changes their response variables arbitrarily. Ideally, we want a robust algorithm against a stronger adversary who can corrupt the covariates also. However, even for a simpler problem of private mean estimation, achieving robustness against such a strong adversary with  $O(d)$  samples requires heavy machinery (convex relaxations of sum-of-squares optimization) with significantly more computations (although polynomial) [45].
Our lower bound in Prop. 3.9, together with the lower bound in [16] on the uncorrupted case, shows that  $n = \Omega (d / \alpha^2 + d / (\varepsilon \alpha))$  samples are necessary to achieve an error rate of  $(1 / \sigma)\| \hat{w} - w^{*}\|_{\Sigma} = O(\alpha + \alpha_{\mathrm{corr}})$ . In particular, it is impossible to achieve an error below  $\alpha_{\mathrm{corr}}$  even if we have
Table 1: Suppose data is drawn from a linear model in  $d$ -dimensions from sub-Gaussian covariates with covariance  $\Sigma$  and sub-Gaussian noise with variance  $\sigma^2$ . To achieve an error rate of  $(1 / \sigma)\| \hat{w} - w^{*}\|_{\Sigma} = \alpha$  with  $(\varepsilon, \delta)$ -DP, DP-RobGD requires the least number of samples among computationally efficient algorithms. This improves over [81] by a factor of  $\kappa^{1/2}$  in the second term, where  $\kappa$  is the condition number of  $\Sigma$ . We hide polylogarithmic factors in  $d$ ,  $\kappa$  and  $1/\delta$ .  $\spadesuit$ DP-Theil-Sen is only analyzed when  $\kappa = 1$  and its dependence  $\kappa^c$  is unknown.
Algorithm	Runtime	Sample Complexity
TukeyEM [6]	poly	no guarantee
DP-Theil-Sen [74]♣	poly	d2/α2 + d/εακc
DP-AMBSSGD [81]	poly	d/α2 + d/εακ + κ2d/ε
DP-RobGD [Theorem 3.8]	poly	d/α2 + d/εακ1/2
HPTR [61], Robust-to-private [10]	exp	d/α2 + d/εα
Lower Bound [16]	d/α2 + d/εα
infinite samples (Prop. 3.9), and hence there is no need to aim for  $\alpha < \alpha_{\mathrm{corrupt}}$ . This lower bound is matched by exponential time approaches, HPTR in [61] and Robust-to-Private in [10], which also guarantee robustness. Currently, there is no efficient algorithm that can guarantee both privacy and robustness for linear regression. To this end, we provide the first efficient algorithm guaranteeing both, with a sample complexity that is optimal up to a  $\kappa^{1/2}$  factor.
Corollary 1.3 (Corollary of Thm. H.3 for sub-Gaussian data with adversarial label corruption). Under the hypotheses of Coro. 1.1, suppose  $\alpha_{\mathrm{corrupt}}$ -fraction of the labels are corrupted arbitrarily. Then  $n = \tilde{O}(d / \alpha^2 + \kappa^{1/2} d / (\varepsilon \alpha))$  samples are sufficient for Alg. 1 to achieve an error rate of  $(1 / \sigma) \| \hat{w} - w^* \|_{\Sigma} = \tilde{O}(\alpha + \alpha_{\mathrm{corrupt}})$  with high probability, where  $\kappa := \lambda_{\max}(\Sigma) / \lambda_{\min}(\Sigma)$ .
When  $\alpha_{\mathrm{corrupt}} = 0$ , this recovers the non-robust result from Coro. 1.1. A similar robustness guarantee also holds for heavy-tailed settings. We provide a formal statement in App. L
Contributions. For a canonical problem of private linear regression under sub-Gaussian distributions, the best known efficient algorithm [81] requires
$$
n = \tilde {O} \left(\frac {d}{\alpha^ {2}} + \frac {\kappa d}{\varepsilon \alpha} + \frac {\kappa^ {2} d}{\varepsilon}\right),
$$
to achieve  $(1 / \sigma)\| \hat{w} -w^{*}\|_{\Sigma} = \alpha$  . We provide the first efficient algorithm that improves this to
$$
n = \tilde {O} \left(\frac {d}{\alpha^ {2}} + \frac {\kappa^ {1 / 2} d}{\varepsilon \alpha}\right),
$$
which nearly matches the exponential-time algorithms [61, 10] and the lower bound [16] up to  $\kappa^{1/2}$  in the second term. For the same problem, we show that the same algorithm is the first to achieve robustness against adversarial corruption of the labels.
Under a heavy-tailed distribution of the noise, we provide the first computationally efficient algorithm, to the best of our knowledge, that achieves a sample complexity close to that of an exponential-time algorithm of [61]. There is no matching lower bound in the heavy-tailed setting. This is also the first efficient algorithm to achieve robustness against adversarial corruption of the labels under heavy-tailed noise.
When there is no adversary, we present our results under the standard linear model with sub-Gaussian covariates and noise. In App. H, we present a more general family of  $(K,a)$ -sub-Weibull distributions that recovers the standard sub-Gaussian family as a special case when  $a = 0.5$ . The necessity of such assumptions on the tail is explained in Sec. 3.4.
Assumption 2.1 (sub-Gaussian model). We have i.i.d. samples  $S = \{(x_{i} \in \mathbb{R}^{d}, y_{i} \in \mathbb{R})\}_{i=1}^{n}$  from a distribution  $\mathcal{P}_{\Sigma, w^{*}, \sigma^{2}}$  of a linear model  $y_{i} = \langle x_{i}, w^{*} \rangle + z_{i}$ , where the input vector  $x_{i}$  has zero
mean  $\mathbb{E}[x_i] = 0$  and a positive definite covariance  $\Sigma \coloneqq \mathbb{E}[x_i x_j^\top ]\succ 0$ , and the (input dependent) label noise  $z_{i}$  has zero mean  $\mathbb{E}[z_i] = 0$  and variance  $\sigma^2\coloneqq \mathbb{E}[z_i^2 ]$ . We further assume  $\mathbb{E}[x_i z_i] = 0$  which is equivalent to assuming that the true parameter  $w^{*} = \Sigma^{-1}\mathbb{E}[y_{i}x_{i}]$ . We assume the marginal distributions of  $x_{i}$  and  $z_{i}$  are  $K$ -sub-Gaussian with  $K = O(1)$ , as defined below.
Definition 2.2.  $x \in \mathbb{R}^d$  is  $K$ -sub-Gaussian if for all  $v \in \mathbb{R}^d$ ,  $\mathbb{E}\left[\exp \left(\frac{\langle v,x\rangle^2}{K^2\mathbb{E}[\langle v,x\rangle^2]}\right)\right] \leq 2$ .
Given a dataset  $S$  that is i.i.d. sampled from  $\mathcal{P}_{\Sigma^2,w^*,\sigma^2}$  satisfying Asmp. 2.1, our goal is to estimate  $w^{*}$  that minimizes  $(1 / \sigma)\| \hat{w} -w^{*}\|_{\Sigma}$  which is also equivalent to minimize the excess population risk, i.e.,  $\mathcal{L}(w^{*}) - \mathcal{L}(\hat{w})$  where  $\mathcal{L}(w)\coloneqq \mathbb{E}_{(x,y)\sim \mathcal{P}_{\Sigma ,w^{*},\sigma^{2}}}[(y - \langle w,x\rangle)^{2}]$ .
Notations. A vector  $x \in \mathbb{R}^d$  has the Euclidean norm  $\| x \|$ . For a matrix  $M$ , we use  $\| M \|_2$  to denote the spectral norm. The error is measured in  $\| \hat{w} - w^* \|_{\Sigma} \coloneqq \| \Sigma^{1/2} (\hat{w} - w^*) \|$  for some PSD matrix  $\Sigma$ . The identity matrix is denoted by  $\mathbf{I}_d \in \mathbb{R}^{d \times d}$ . Let  $[n] = \{1, 2, \dots, n\}$ .  $\tilde{O}(\cdot)$  hides some constants terms,  $K = \Theta(1)$ , and poly-logarithmic terms in  $n, d, 1/\varepsilon, \log(1/\delta), 1/\zeta,$  and  $1/\alpha_{\mathrm{corrupt}}$ . For a vector  $x \in \mathbb{R}^d$ , we define  $\operatorname{clip}_a(x) \coloneqq x \cdot \min \{1, a / \|x\|\}$ .
Background on DP. Differential Privacy is a standard measure of privacy leakage when data is accessed via queries, introduced by [33]. Two datasets  $S$  and  $S'$  are said to be neighbors if they differ at most by one entry, which is denoted by  $S \sim S'$ . A stochastic query  $q$  is said to be  $(\varepsilon, \delta)$ -differentially private for some  $\varepsilon > 0$  and  $\delta \in [0,1]$ , if  $\mathbb{P}(q(S) \in A) \leq e^{\varepsilon} \mathbb{P}(q(S) \in A) + \delta$ , for all neighboring datasets  $S \sim S'$  and all subset  $A$  of the range of the query. We build upon two widely used DP primitives, the Gaussian mechanism and the private histogram. A central concept in DP mechanism design is the sensitivity of a query, defined as  $\Delta_q := \sup_{S \sim S'} \| q(S) - q(S') \|$ . We describe Gaussian mechanism and private histogram in App. B.
2.1 Comparisons with the prior work
The state-of-the-art approach introduced by [81] is based on DP-SGD [71], where privacy is ensured by gradient norm clipping and the Gaussian mechanism. Two additional technical components are adaptive clipping and streaming SGD. Adaptive clipping with an appropriate threshold  $\theta_{t}$  ensures that no data point is clipped (under the sub-Gaussian assumption), while providing a bound on the sensitivity of the average mini-batch gradient (to ensure we do not add too much noise). The streaming approach, where each data point is only touched once and discarded, ensures independence between the past iterate  $w_{t}$  and the gradients at round  $t + 1$ , which the analysis critically relies on. For  $T = \tilde{\Theta} (\kappa)$  iterations where  $\kappa$  is the condition number of the covariance  $\Sigma$ , the dataset  $S = \{(x_i,y_i)\}_{i = 1}^n$  is partitioned into  $\{B_t\}_{t = 0}^{T - 1}$  subsets of equal size:  $|B_t| = \tilde{\Theta} (n / \kappa)$ . At each round  $t$ , the gradients are clipped and averaged with additive Gaussian noise chosen to satisfy  $(\varepsilon ,\delta)$ -DP:
$$
w _ {t + 1} \leftarrow w _ {t} - \eta \left(\frac {1}{\left| B _ {t} \right|} \sum_ {i \in B _ {t}} \operatorname {c l i p} _ {\theta_ {t}} \left(x _ {i} \left(w _ {t} ^ {\top} x _ {i} - y _ {i}\right)\right) + \frac {\theta_ {t} \sqrt {2 \log (1 . 2 5 / \delta)}}{\varepsilon \left| B _ {t} \right|} \nu_ {t}\right), \tag {1}
$$
where  $\nu_{t}\sim \mathcal{N}(0,\mathbf{I}_{d})$ . In [81], a slight variation of this streaming SGD is shown to achieve an error of  $(1 / \sigma)\| w_{T} - w^{*}\|_{\Sigma} = \alpha$  with  $n = \tilde{O} (d / \alpha^2 +\kappa d / (\varepsilon \alpha) + \kappa^2 d / \varepsilon)$  samples (Row 3 in Tab. 1).
Our technical innovations. Our approach builds upon such gradient based methods but makes several important innovations. First, we use full-batch gradient descent, as opposed to the streaming SGD above. Using all  $n$  samples reduces the sensitivity of the per-round gradient average by a  $\kappa$  factor, and thus decreases the privacy noise added in each iteration. This improves the second term of sample complexity from  $\kappa d / (\varepsilon \alpha)$  to  $\kappa^{1/2}d / (\varepsilon \alpha)$  and removes the third term completely. However, full-batch GD loses the independence that the streaming SGD enjoyed between  $w_{t}$  and the samples used in the round  $t + 1$ . This dependence makes the analysis more challenging. We instead propose using the resilience to precisely track the bias and variance of the (dependent) full-batch average gradient. Resilience is a central concept in robust statistics that links the tail-property of the distribution to the bias, which we explain in Sec. 5.
Next, one critical component in achieving this improved sample complexity is the new analysis technique we introduce for tracking the end-to-end gradient updates. Since our gradient descent algorithm is not guaranteed to make progress every step, we cannot use the vanilla one-step analysis.
Taking the full end-to-end analysis by expanding the whole gradient trajectory will introduce too many correlated cross-terms which are very hard to control. Therefore, we leverage an every  $\kappa$ -step analysis and show that the objective function at least decreases geometrically every  $\kappa$  steps. To be more specific, our analysis technique in App. H (steps 3 and 4) opens up the iterative updates from the beginning to the end, and exploits the fact that  $\lambda_{\max}((\eta \Sigma)^{1/2}(1 - \eta \Sigma)^i (\eta \Sigma)^{1/2})$  is upper bounded by  $1/(i + 1)$  when  $\|\eta \Sigma\| \leq 1$ . This technique is critical in achieving the near-optimal dependence in  $\kappa$ . This might be of independent interest to other analysis of gradient-based algorithms. We refer to the beginning of step 3 in App. H for a detailed explanation.
Finally, we propose a novel clipping that separately clips  $x_{i}$  and  $(w_{t}^{\top}x_{i} - y_{i})$  in the gradient,  $(w_{t}^{\top}x_{i} - y_{i})x_{i}$ . This is critical in achieving robustness to label-corruption, as we explain in Sec. 3.1.
3 Label-robust and private linear regression
We introduce a novel gradient descent approach. This achieves an improved sample complexity compared to the state-of-the-art algorithm and robustness against label corruption.
The skeleton of our approach in Alg. 1 is the general DP-SGD [1, 71] with adaptive clipping [7]. We partition the dataset into three equal-sized subsets:  $S_{1}, S_{2}, S_{3}$ .  $S_{1}$  and  $S_{2}$  are used in adaptively estimating the clipping thresholds, and  $S_{3}$  is re-used every step to compute the average gradient.
The standard adaptive clipping, e.g., [7, 81], is not robust against label-corruption. Under sub-Gaussian distribution, a positive fraction of the covariates,  $x_{i}$ 's, can be close to the origin. If the adversary chooses to corrupt those points with small norm,  $\| x_{i}\|$ , they can make large changes in the corrupted residual,  $(y_{i} - w_{t}^{\top}x_{i})$ , while evading the standard clipping by the norm of the gradient; the norm of the gradient,  $\| x_{i}(y_{i} - w_{t}^{\top}x_{i})\| = \| x_{i}\| |y_{i} - w_{t}^{\top}x_{i}|$ , can remain under the threshold. This is problematic, since the bias due to the corrupted samples in the gradient scales proportionally to the magnitude of the residual (after clipping). To this end, we propose clipping the norm and the residual separately:  $\mathrm{clip}_{\Theta}(x_i)\mathrm{clip}_{\theta_t}\left(w_t^\top x_i - y_i\right)$ . This keeps the sensitivity of gradient average bounded by  $\Theta (\theta_t)$ . The subsequent Gaussian mechanism in line 11 ensures  $(\varepsilon_0,\delta_0)$ -DP at each round. Applying advanced composition in Lemma B.5 of  $T$  rounds, this ensures end-to-end  $(\varepsilon ,\delta)$ -DP.
Novel adaptive clipping. When clipping with  $\mathrm{clip}_{\Theta}(x_i)$ , the only purpose of clipping the covariate by its norm,  $\| x_{i}\|$ , is to bound the sensitivity of the resulting clipped gradient. In particular, we do not need to make it robust as there is no corruption in the covariates. Ideally, we want to select the smallest threshold  $\Theta$  that does not clip any of the covariates. Since the norm of a covariate is upper bounded by  $\| x_{i}\|^{2} \leq K^{2}\mathrm{Tr}(\Sigma)\log (1 / \zeta)$  with probability  $1 - \zeta$  (Lemma J.3), we estimate the unknown  $\operatorname{Tr}(\Sigma)$  using Private Norm Estimator in Alg. 3 in App. F and set the norm threshold  $\Theta = K\sqrt{2\Gamma\log(n / \zeta)}$  (Alg. 1 line 4). The  $n$  in the logarithm ensures that the union bound holds.
When clipping with  $\mathrm{clip}_{\theta_t}(w_t^\top x_i - y_i)$ , the purpose of clipping the residual by its magnitude,  $|y_{i} - w_{t}^{\top}x_{i}| = |(w^{*} - w_{t})^{\top}x_{i} + z_{i}|$ , is to bound the sensitivity of the gradient and also to provide robustness against label-corruption. We want to choose a threshold that only clips corrupt data points and at most a few clean data points. In order to achieve an error  $(1 / \sigma)\| w_{T} - w^{*}\|_{\Sigma} = \alpha$ , we know that any set of  $(1 - \alpha)$  fraction of the clean data points is sufficient to get a good estimate of the average gradient. By clipping at  $|(w^{*} - w_{t})^{\top}x_{i} + z_{i}|^{2}\leq (\| w_{t} - w^{*}\|_{\Sigma}^{2} + \sigma^{2})CK^{2}\log (1 / (2\alpha))$ , Lemma J.3 guarantees that the unclipped subset will be large enough, i.e.,  $(1 - \alpha)n$ . At the same time, this threshold on the residual is small enough to guarantee robustness against the label-corrupted samples. We introduce the robust and DP Distance Estimator in Alg. 2 to estimate the unknown (squared and shifted) distance,  $\| w_{t} - w^{*}\|_{\Sigma}^{2} + \sigma^{2}$ , and set the distance threshold  $\theta_t = 2\sqrt{2\gamma_t}\sqrt{9C_2K^2\log(1 / (2\alpha))}$  (Alg. 1 line 7). Both norm and distance estimation rely on DP histogram (Lemma B.2), but over a set of statistics computed on partitioned datasets, which we explain in detail in App. C.