SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jongho Park
KRAFTON
jongho PARK@krafton.com
Sushrut Karmalkar
skarmalkar@wisc.edu
Christos Tzamos
University of Athens
tzamos@wisc.edu
We initiate the study of stochastic optimization with oblivious noise, broadly generalizing the standard heavy-tailed noise setup. In our setting, in addition to random observation noise, the stochastic gradient may be subject to independent oblivious noise, which may not have bounded moments and is not necessarily centered. Specifically, we assume access to a noisy oracle for the stochastic gradient of  $f$  at  $x$ , which returns a vector  $\nabla f(\gamma, x) + \xi$ , where  $\gamma$  is the bounded variance observation noise and  $\xi$  is the oblivious noise that is independent of  $\gamma$  and  $x$ . The only assumption we make on the oblivious noise  $\xi$  is that  $\mathbf{Pr}[\xi = 0] \geq \alpha$  for some  $\alpha \in (0,1)$ . In this setting, it is not information-theoretically possible to recover a single solution close to the target when the fraction of inliers  $\alpha$  is less than  $1/2$ . Our main result is an efficient list-decodable learner that recovers a small list of candidates, at least one of which is close to the true solution. On the other hand, if  $\alpha = 1 - \epsilon$ , where  $0 < \epsilon < 1/2$  is sufficiently small constant, the algorithm recovers a single solution. Along the way, we develop a rejection-sampling-based algorithm to perform noisy location estimation, which may be of independent interest.
A major challenge in modern machine learning systems is to perform inference in the presence of outliers. Such problems appear in various contexts, such as analysis of real datasets with natural outliers, e.g., in biology [Rosenberg et al., 2002, Paschou et al., 2010, Li et al., 2008], or neural network training, where heavy-tailed behavior arises from stochastic gradient descent when the batch size is small and when the step size is large [Hodgkinson and Mahoney, 2021, Gurbuzbalaban et al., 2021]. In particular, when optimizing various neural networks, there exists strong empirical evidence indicating that gradient noise frequently displays heavy-tailed behavior, often showing characteristics of unbounded variance. This phenomenon has been observed for fully connected and convolutional neural networks [Simsekli et al., 2019, Gurbuzbalaban and Hu, 2021] as well as attention models [Zhang et al., 2020]. Hence, it is imperative to develop robust optimization methods for machine learning in terms of both performance and security.
In this paper, we study robust first-order stochastic optimization under heavy-tailed noise, which may not have any bounded moments. Specifically, given access to a noisy gradient oracle for  $f(x)$
(which we describe later), the goal is to find a stationary point of the objective function  $f: \mathbb{R}^d \to \mathbb{R}$ ,  $f(x) \coloneqq \mathbf{E}_{\gamma}[f(\gamma, x)]$ , where  $\gamma$  is drawn with respect to an arbitrary distribution on  $\mathbb{R}^d$ .
Previous work on  $p$ -heavy-tailed noise often considers the case where one has access to  $\nabla f(\gamma, x)$  such that  $\mathbf{E}_{\gamma}[\nabla f(\gamma, x)] = \nabla f(x)$  and  $\mathbf{E}_{\gamma}[\| \nabla f(\gamma, x) - \nabla f(x) \|^{p}] \leq \sigma^{p}$  for  $p = 2$  [Nazin et al., 2019, Gorbunov et al., 2020], and recently for  $p \in (1, 2]$  [Cutkosky and Mehta, 2021, Sadiev et al., 2023]. In contrast, Diakonikolas et al. [2019] investigates a more challenging noise model in which a small proportion of the sampled functions  $f(\gamma, x)$  are arbitrary adversarial outliers. In their scenario, they have full access to the specific functions being sampled and can eventually eliminate all but the least harmful outlier functions. Their algorithm employs robust mean estimation techniques to estimate the gradient and filter out the outlier functions. Note that their adversarial noise model requires that at least half of the samples are inliers.
This work aims to relax the distributional assumption of heavy-tailed noise such that the fraction of outliers can approach one, and do not have any bounded moment constraints while keeping optimization still computationally tractable. Thus, we ask the following natural question:
What is the weakest noise model for optimization in which efficient learning is possible, while allowing for strong corruption of almost all samples?
In turn, we move beyond prior heavy-tailed noise models with bounded moments and define a new noise model for first-order optimization, inspired by the oblivious noise model studied in the context of regression [Bhatia et al., 2015, d'Orsi et al., 2021b]. We consider the setting in which the gradient oracle returns noisy gradients whose mean may not be  $\nabla f(x)$ , and in fact, may not even exist. The only condition on our additive noise distribution of  $\xi$  is that with probability at least  $\alpha$ , oblivious noise takes the value zero. Without this, the optimization problem would become intractable. Notably, oblivious noise can capture any tail behavior that arises from independent  $p$ -heavy-tailed noise.
We now formally define the oblivious noise oracle below.
Definition 1.1 (Oblivious Noise Oracle). We say that an oracle  $\mathcal{O}_{\alpha, \sigma, f}(x)$  is an oblivious-noise oracle, if, when queried on  $x$ , the oracle returns  $\nabla f(\gamma, x) + \xi$  where  $\gamma$  is drawn from some arbitrary distribution  $Q$  and  $\xi$  is drawn from  $D_{\xi}$  independent of  $x$  and  $\gamma$ , satisfying  $\operatorname*{Pr}_{\xi \sim D_{\xi}}[\xi = 0] \geq \alpha$ . The distribution  $Q$  satisfies  $\mathbf{E}_{\gamma \sim Q}[\nabla f(\gamma, x)] = \nabla f(x)$  and  $\mathbf{E}_{\gamma \sim Q}[\| \nabla f(\gamma, x) - \nabla f(x) \|^{2}] \leq \sigma^{2}$ .
Unfortunately, providing an approximate stationary point  $\widehat{x}$  such that  $\| \nabla f(\widehat{x})\| \leq \epsilon$ , for some small  $\epsilon > 0$ , is information-theoretically impossible when  $\alpha \leq 1/2$ . To see this, consider the case where  $\operatorname{Pr}_{\gamma}[\gamma = 0] = 1$  and the function  $f(0,x) = x^2$ . If  $\xi$  follows a uniform distribution over the set  $\{-2,0,2\}$ , when we query the oracle for a gradient at  $x$ , we will observe a uniform distribution over  $\{2x - 2,2x,2x + 2\}$ . This scenario cannot be distinguished from a similar situation where the objective is to minimize  $f(0,x) = (x - 1)^2$ , and  $\xi$  follows a uniform distribution over  $\{0,2,4\}$ .
To address this challenge, we allow our algorithm to work in the list-decodable setting, implying that it may return a small list of candidate solutions such that one of its elements  $\widehat{x}$  satisfies  $\| \nabla f(\widehat{x})\| \leq \epsilon$ . We now define our main problem of interest.
Definition 1.2 (List-Decodable Stochastic Optimization). The problem of List-Decodable Stochastic Optimization with oblivious noise is defined as follows: For an  $f$  that is  $L$ -smooth and has a global minimum, given access to  $\mathcal{O}_{\alpha,\sigma,f}(\cdot)$  as defined in Definition 1.1, the goal is to output a list  $\mathcal{L}$  of size  $s$  such that  $\min_{\widehat{x} \in \mathcal{L}} \| \nabla f(\widehat{x}) \| \leq \epsilon$ .
As our main result, we demonstrate the equivalence between list-decodable stochastic optimization with oblivious noise and list-decodable mean estimation (LDME), which we define below.
Definition 1.3 (List-Decodable Mean Estimation). Algorithm  $\mathcal{A}$  is an  $(\alpha, \beta, s)$ -LDME algorithm for  $\mathcal{D}$  (a set of candidate inlier distributions) if with probability  $1 - \delta_{\mathcal{A}}$ , it returns a list  $\mathcal{L}$  of size  $s$  such that  $\min_{\hat{\mu} \in \mathcal{L}} \|\hat{\mu} - \mathbf{E}_{x \sim D}[x]\| \leq \beta$  for  $D \in \mathcal{D}$  when given  $m_{\mathcal{A}}$  samples  $\{z_i + \xi_i\}_{i=1}^{m_{\mathcal{A}}} \text{ for } z_i \sim D$  and  $\xi_i \sim D_{\xi}$  where  $\operatorname*{Pr}_{\xi \sim D_{\xi}}[\xi = 0] \geq \alpha$ . If  $1 - \alpha$  is a sufficiently small constant less than  $1/2$ , then  $s = 1$ .
We define  $\mathcal{D}_{\sigma}$  to be the following set of distributions over  $\mathbb{R}^d$ :  $\mathcal{D}_{\sigma} := \{D \mid \mathbf{E}_D[||x - \mathbf{E}_D[x]||^2] \leq \sigma^2\}$ . We also use  $\tilde{O}(\cdot)$  to hide all log factors in  $d, 1/\alpha, 1/\eta, 1/\delta$ , where  $\delta$  denotes the failure probability and  $\eta$  is a multiplicative parameter we use in our algorithm.
Our first theorem shows that if there exists an efficient algorithm for the problem of list-decodable mean estimation and an inexact learner (an optimization method using inexact gradients), there is an efficient algorithm for list-decodable stochastic optimization.
Theorem 1.4 (List-Decodable Stochastic Optimization  $\rightarrow$  List-Decodable Mean Estimation). Suppose that for any  $f$  having a global minimum and being  $L$ -smooth, the algorithm  $\mathcal{A}_G$ , given access to  $g_{x}$  satisfying  $\| g_{x} - \nabla f(x)\| \leq O(\eta \sigma)$ , recovers  $\hat{x}$  satisfying  $\| \nabla f(\hat{x})\| \leq O(\eta \sigma) + \epsilon$  in time  $T_{G}$ . Let  $\mathcal{A}_{ME}$  be an  $(\alpha, O(\eta \sigma), s)$ -LDME algorithm for  $\mathcal{D}_{\sigma}$ . Then, there exists an algorithm (Algorithm 2) which uses  $\mathcal{A}_{ME}$  and  $\mathcal{A}_G$ , makes  $m = m_{\mathcal{A}_{ME}} + \tilde{O}(T_G \cdot (O(1) / \alpha)^{2 / \eta} / \eta^5)$  queries to  $\mathcal{O}_{\alpha,\sigma,f}$ , runs in time  $T_{G} \cdot \mathrm{poly}(d, 1 / \eta, (O(1) / \alpha)^{1 / \eta}, \log(1 / \delta \eta \alpha))$ , and returns a list  $\mathcal{L}$  of  $s$  candidates satisfying  $\min_{x \in \mathcal{L}} \| \nabla f(x)\| \leq O(\eta \sigma) + \epsilon$  with probability  $1 - \delta_{\mathcal{A}_{ME}}$ .
To achieve this reduction, we develop an algorithm capable of determining the translation between two instances of a distribution by utilizing samples that are subject to additive noise. It is important to note that the additive noise can differ for each set of samples. As far as we know, this is the first guarantee of its kind for estimating the location of a distribution, considering the presence of noise, sample access, and the absence of density information.
Theorem 1.5 (Noisy Location Estimation). Let  $\eta \in (0,1)$  and let  $D_{\xi}, D_z, D_{z'}$  be distributions such that  $\operatorname{Pr}_{\xi \sim D_{\xi}}[\xi = 0] \geq \alpha$  and  $D_z, D_{z'}$  are possibly distinct mean zero distributions with variance bounded by  $\sigma^2$ . Then there is an algorithm (Algorithm 5) which, for unknown  $t \in \mathbb{R}^d$ , takes  $m = \tilde{O}((1/\eta^5)(O(1)/\alpha)^{2/\eta})$  samples  $\{\xi_i + z_i + t\}_{i=1}^m$  and  $\{\tilde{\xi}_i + z_i'\}_{i=1}^m$ , where  $\xi_i$  and  $\tilde{\xi}_i$  are drawn independently from  $D_{\xi}$  and  $z_i$  and  $z_i'$  are drawn from  $D_z$  and  $D_{z'}$  respectively, runs in time  $\mathrm{poly}(d,1/\eta, (O(1)/\alpha)^{1/\eta}, \log(1/\delta \eta \alpha))$ , and with probability  $1 - \delta$  recovers  $t'$  such that  $|t - t'| \leq O(\eta \sigma)$ .
The fact that there exist algorithms for list-decodable mean estimation with the inliers coming from  $\mathcal{D}_{\sigma}$  allows us to get concrete results for list-decodable stochastic optimization.
Conversely, we show that if we have an algorithm for list-decodable stochastic optimization, then we also get an algorithm for list-decodable mean-estimation. This in turn implies, via Fact 2.1, that an exponential dependence on  $1 / \eta$  is necessary in the list-size if we want to estimate the correct gradient up to an error of  $O(\eta \sigma)$  when the inliers are drawn from some distribution in  $\mathcal{D}_{\sigma}$ .
Theorem 1.6 (List-Decodable Mean Estimation  $\rightarrow$  List-Decodable Stochastic Optimization). Assume there is an algorithm for List-Decodable Stochastic Optimization with oblivious noise that runs in time  $T$  and makes  $m$  queries to  $\mathcal{O}_{\alpha, \sigma, f}$ , and returns a list  $\mathcal{L}$  of size  $s$  containing  $\hat{x}$  satisfying  $\|f(\hat{x})\| \leq \epsilon$ . Then, there is an  $(\alpha, \epsilon, s)$ -LDME algorithm for  $\mathcal{D}$  that runs in time  $T$ , queries  $m$  samples, and returns a list of size  $s$ .
If  $\alpha = 1 - \epsilon$ , where  $0 < \epsilon < 1/2$  is at most a sufficiently small constant, the above theorems hold for the same problems, but with the constraint that the list is singleton.
Given the extensive robust optimization and estimation literature, we focus on the most relevant work.
Optimization with Heavy-tailed Noise There is a wealth of literature on both the theoretical and empirical convergence behavior of stochastic gradient descent (SGD) for both convex and non-convex problems, under various assumptions on the stochastic gradient (see, e.g., Hardt et al. [2016], Wang et al. [2021] and references within). However, the noisy gradients have shown to be problematic when training ML models [Shen and Sanghavi, 2019, Zhang et al., 2020], hence necessitating robust optimization algorithms.
From a theoretical point of view, several noise models have been proposed to account for inexact gradients. A line of work [d'Aspremont, 2008, So, 2013, Devolder et al., 2014, Cohen et al., 2018] studies the effects of inexact gradients to optimization methods in terms of error accumulation. For instance, Devolder et al. [2014] demonstrates that, given an oracle that outputs an arbitrary
perturbation of the true gradient, the noise can be determined adversially to encode non-smooth problems. Alternatively, many recent works [Lan, 2012, Gorbunov et al., 2020, Cutkosky and Mehta, 2021, Mai and Johansson, 2021, Sadiev et al., 2023] have studied  $p$ -heavy-tailed noise, an additive stochastic noise to the true gradient where one has access to  $\nabla f(\gamma, x)$  such that  $\mathbf{E}_{\gamma}[\nabla f(\gamma, x)] = \nabla f(x)$  and  $\mathbf{E}_{\gamma}[\| \nabla f(\gamma, x) - \nabla f(x) \|^{p}] \leq \sigma^{p}$ . For instance, Sadiev et al. [2023] propose and analyze a variant of clipped-SGD to provide convergence guarantees for when  $p \in (1, 2]$ . However, these noisy oracles, whether deterministic or stochastic, assume bounded norm or moments on the noise, an assumption that is not present in the oblivious noise oracle.
Robust Estimation Our oblivious noise oracle for optimization is motivated by the recent work on regression under oblivious outliers. In the case of linear regression, the oblivious noise model can be seen as the weakest possible noise model that allows almost all points to be arbitrarily corrupted, while still allowing for recovery of the true function with vanishing error [Bhatia et al., 2015, Suggala et al., 2019]. This also captures heavy-tailed noise that may not have any moments. The setting has been studied for various problems, including online regression [Pesme and Flammarion, 2020], PCA [d'Orsi et al., 2021a], and sparse signal recovery [d'Orsi et al., 2022].
On the other hand, there has been a flurry of work on robust estimation in regards to worst-case adversarial outliers (see, e.g., Diakonikolas et al. [2016], Lai et al. [2016], Charikar et al. [2017], Diakonikolas et al. [2019]). Robust mean estimation aims to develop an efficient mean estimator when  $(1 - \alpha)$ -fraction of the samples is arbitrary. In contrast, list-decodable mean estimation generates a small list of candidates such that one of these candidates is a good estimator. While robust mean estimation becomes information-theoretically impossible when  $\alpha \leq 1/2$ , the relaxation to output a list allows the problem to become tractable for any  $\alpha \in (0,1]$  [Charikar et al., 2017, Diakonikolas et al., 2022]. See Diakonikolas and Kane [2019, 2023] for in-depth treatments of the subject.
In the past, robust mean estimators have been used to perform robust gradient estimation (see, e.g., Charikar et al. [2017], Diakonikolas et al. [2019]), which is similar to what we do in our paper. However, these results assume access to the entire function set, which allows them to discard outlier functions. In contrast, in our setting, we only have access to a noisy gradient oracle, so at each step, we get a fresh sample set and, hence, a different set of outliers. This introduces further difficulties, which we resolve via location estimation.
We note a subtle difference in the standard list-decodable mean estimation setting and the setting considered in this work. The standard list-decodable mean estimation setting draws an inlier with probability  $\alpha$  and an outlier with the remaining probability. In contrast, our model gets samples of the kind  $\xi + z$  where  $z$  is drawn from the inlier distribution, and  $\operatorname{Pr}[\xi = 0] > \alpha$ , and is arbitrary otherwise. The algorithms for the mixture setting continue to work in our setting as well. Another difference is that the standard setting requires that the distribution have bounded variance in every direction. On the other hand, in the optimization literature, the assumption is that the stochastic gradient has bounded expected squared norm from the expectation.
Location estimation Location estimation has been extensively studied since the 1960s. Traditional approaches to location estimation have focused on achieving optimal estimators in the asymptotic regime. The asymptotic theory of location estimation is discussed in detail in [Van der Vaart, 2000].
Recent research has attempted to develop the finite-sample theory of location estimation. These efforts aim to estimate the location of a Gaussian-smoothed distribution with a sample complexity that matches the optimal sample complexity up a constant (see [Gupta et al., 2022] and [Gupta et al., 2023]). However, these results assume prior knowledge of the likelihood function of the distribution up to translation.
Another closely related work initiates the study of robust location estimation for the case where the underlying high-dimensional distribution is symmetric and an  $0 < \epsilon \ll 1/2$  fraction of the samples are adversarially corrupted Novikov et al. [2023]. This follows the line of work on robust mean estimation discussed above.
We present a finite-sample guarantee for the setting with noisy access to samples drawn from a distribution and its translation. Our assumption on the distribution is that it places an  $\alpha$  mass at some point, where  $\alpha \in (0,1)$  and, for instance, could be as small as  $1 / d^{c}$  for some constant  $c$ . The noise is constrained to have mean zero and bounded variance, but crucially the noise added to the samples coming from the distribution and its translation might be drawn from different distributions. To the
best of our knowledge, this is the first result that has noisy access, does not have prior knowledge of the probability density of the distribution and achieves a finite-sample guarantee.
For ease of exposition, we will make the simplifying assumption that the observation noise is bounded between  $[- \sigma, \sigma]$ . Let  $y$  and  $y'$  be two distinct mean-zero noise distributions, both bounded within the range  $[- \sigma, \sigma]$ . Define  $\xi$  to be the oblivious noise drawn from  $D_{\xi}$ , satisfying  $\operatorname{Pr}[\xi = 0] \geq \alpha$ . Assume we have access to the distributions (i.e., we have infinite samples).
Stochastic Optimization reduces to Mean Estimation. In Theorem 1.4, we show how we can leverage a list-decodable mean estimator to address the challenge of list-decodable stochastic optimization with oblivious noise (see Algorithm 2). The key idea is to recognize that we can generate list of gradient estimates, and update this list such that one of the elements always closely approximates the true gradient in  $\ell_2$  norm at the desired point.
The algorithmic idea is as follows: First, run a list-decodable mean estimation algorithm on the noisy gradients at  $x_0 = 0$  to retrieve a list  $\mathcal{L}$  consisting of  $s$  potential gradient candidates. This set of candidates contains at least one element which closely approximates  $\nabla f(0)$ .
One natural approach at this stage would be to perform a gradient descent step for each of the  $s$  gradients, and run the list-decodable mean estimation algorithm again to explore all potential paths that arise from these gradients. However, this naive approach would accumulate an exponentially large number of candidate solutions. We use a location-estimation algorithm to tackle this issue.
We can express  $f(\gamma, x)$  as  $f(x) + e(\gamma, x)$  where  $\mathbf{E}_{\gamma}[e(\gamma, x)] = 0$  and  $\mathbf{E}_{\gamma}[\|e(\gamma, x)\|^2] < \sigma^2$ . When we query  $\mathcal{O}_{\alpha, \sigma, f}(\cdot)$  at  $x$ , we obtain samples of the form  $\nabla f(x) + \xi + \nabla e(\gamma, x)$ , i.e., samples from a translated copy of the oblivious noise distribution convolved with the distribution of  $\nabla e(\gamma, x)$ . We treat the distribution of  $\nabla e(\gamma, x)$  as observation noise. The translation between the distributions of our queries to  $\mathcal{O}_{\alpha, \sigma, f}(\cdot)$  at  $x$  and 0 correspond to  $\nabla f(x) - \nabla f(0)$ . To update the gradient, it is sufficient to recover this translation accurately. By doing so, we can adjust  $\mathcal{L}$  by translating each element while maintaining the accuracy of the estimate.
Finally, we run a first-order learner for stochastic optimization using these gradient estimates. We select and explore  $s$  distinct paths, with each path corresponding to an element of  $\mathcal{L}$  obtained earlier. Since one of the paths always has approximately correct gradients, this path will converge to a stationary point in the time that it takes for the first-order learner to converge.
Noisy Location Estimation in 1-D. The objective of noisy location estimation is to retrieve the translation between two instances of a distribution by utilizing samples that are subject to additive mean-zero and bounded-variance noise.
In Lemma 3.1, we establish that if Algorithm 1 is provided with a parameter  $\eta \in (0,1)$  and samples from the distributions of  $\xi +y$  and  $\tilde{\xi} +y^{\prime} + t$ , it can recover the value of  $t$  within an error of  $O(\eta \sigma)$ . Here  $\xi$  and  $\tilde{\xi}$  are independent draws from the same distribution.
Observe that in the absence of additive noise  $y$  and  $y'$ ,  $t$  can be estimated exactly by simply taking the median of a sufficiently large number of pairwise differences between independent samples from  $\xi$  and  $\tilde{\xi} + t$ . This is because the distribution of  $\tilde{\xi} - \xi$  is symmetric at zero and  $\operatorname{Pr}[\tilde{\xi} - \xi = 0] = \alpha^2$ . However, with unknown  $y$  and  $y'$ , this estimator is no longer consistent since the median of zero-mean and  $\sigma$ -variance random variables can be as large as  $O(\sigma)$  in magnitude. In fact, Claim 3.2 demonstrates that for the setting we consider the median of  $\xi + y$  can as far as  $O(\sigma / \sqrt{\alpha})$  far from the mean of  $\xi$ . However, it does recover a rough estimate  $t_r'$  such that  $|t - t_r'| \leq O(\sigma / \sqrt{\alpha})$ .
Consequently, our approach starts by using  $t_r'$  as the estimate for  $t$ , and then iteratively improving the estimate by mitigating the heavy-tail influence of  $\xi$ . Note that if  $\xi$  did not have heavy tails, then we could try to directly compute  $t = \mathbf{E}[\xi + y' + t] - \mathbf{E}[\xi + y]$ . Unfortunately, due to  $\xi$ 's completely unconstrained tail,  $\xi$  may not even possess a well-defined mean. Nonetheless, if we are able to condition  $\xi$  to a bounded interval, we can essentially perform the same calculation described above. In what follows, we describe one step of an iterative process to improve our estimate of  $t$ .