Title: First Order Stochastic Optimization with Oblivious Noise
Venue: NeurIPS 2023
Accepted: yes
Keywords: Oblivious noise

Robust Statistics

Heavy-tailed Stochastic Optimization

Approximate Gradients

Inexact Gradients
Abstract: We initiate the study of stochastic optimization with oblivious noise, broadly generalizing the standard heavy-tailed noise setup.
In our setting, in addition to random observation noise, the stochastic gradient 
may be subject to independent \emph{oblivious noise}, 
which may not have bounded moments and is not necessarily centered. 
Specifically, we assume access to a noisy oracle for the stochastic gradient of $f$ 
at $x$,  which returns a vector $\nabla f(\gamma, x) + \xi$, where $\gamma$ is 
the  bounded variance observation noise 
and $\xi$ is the oblivious noise that is independent of $\gamma$ and $x$. 
The only assumption we make on the oblivious noise $\xi$ 
is that $\Pr[\xi = 0] \ge \alpha$, for some $\alpha \in (0, 1)$.
In this setting, it is not information-theoretically possible to recover a single solution 
close to the target when the fraction of inliers $\alpha$ is less than $1/2$. 
Our main result is an efficient {\em list-decodable} learner that recovers 
a small list of candidates at least one of which is close to the true solution. 
On the other hand, if $\alpha = 1-\epsilon$, where $0< \epsilon < 1/2$ is sufficiently small
constant, the algorithm recovers a single solution.

Along the way, we develop a rejection-sampling-based algorithm to perform noisy location estimation, 
which may be of independent interest.

First Order Stochastic Optimization with Oblivious Noise

Abstract

We initiate the study of stochastic optimization with oblivious noise, broadly generalizing the standard heavy-tailed noise setup. In our setting, in addition to random observation noise, the stochastic gradient may be subject to independent oblivious noise , which may not have bounded moments and is not necessarily centered. Specifically, we assume access to a noisy oracle for the stochastic gradient of f at x , which returns a vector ∇ f ( γ, x ) + ξ , where γ is the bounded variance observation noise and ξ is the oblivious noise that is independent of γ and x . The only assumption we make on the oblivious noise ξ is that Pr [ ξ = 0] ≥ α for some α ∈ (0 , 1) . In this setting, it is not information-theoretically possible to recover a single solution close to the target when the fraction of inliers α is less than 1 / 2 . Our main result is an efficient list-decodable learner that recovers a small list of candidates, at least one of which is close to the true solution. On the other hand, if α = 1 -ϵ , where 0 < ϵ < 1 / 2 is sufficiently small constant, the algorithm recovers a single solution. Along the way, we develop a rejection-sampling-based algorithm to perform noisy location estimation, which may be of independent interest.

1 Introduction

A major challenge in modern machine learning systems is to perform inference in the presence of outliers. Such problems appear in various contexts, such as analysis of real datasets with natural outliers, e.g., in biology [Rosenberg et al., 2002, Paschou et al., 2010, Li et al., 2008], or neural network training, where heavy-tailed behavior arises from stochastic gradient descent when the batch size is small and when the step size is large [Hodgkinson and Mahoney, 2021, Gurbuzbalaban et al., 2021]. In particular, when optimizing various neural networks, there exists strong empirical evidence indicating that gradient noise frequently displays heavy-tailed behavior, often showing characteristics of unbounded variance. This phenomenon has been observed for fully connected and convolutional neural networks [Simsekli et al., 2019, Gurbuzbalaban and Hu, 2021] as well as attention models [Zhang et al., 2020]. Hence, it is imperative to develop robust optimization methods for machine learning in terms of both performance and security.

In this paper, we study robust first-order stochastic optimization under heavy-tailed noise, which may not have any bounded moments. Specifically, given access to a noisy gradient oracle for f ( x )

(which we describe later), the goal is to find a stationary point of the objective function f : R d → R , f ( x ) := E γ [ f ( γ, x )] , where γ is drawn with respect to an arbitrary distribution on R d .

Previous work on p -heavy-tailed noise often considers the case where one has access to ∇ f ( γ, x ) such that E γ [ ∇ f ( γ, x )] = ∇ f ( x ) and E γ [ ∥∇ f ( γ, x ) - ∇ f ( x ) ∥ p ] ≤ σ p for p = 2 [Nazin et al., 2019, Gorbunov et al., 2020], and recently for p ∈ (1 , 2] [Cutkosky and Mehta, 2021, Sadiev et al., 2023]. In contrast, Diakonikolas et al. [2019] investigates a more challenging noise model in which a small proportion of the sampled functions f ( γ, x ) are arbitrary adversarial outliers. In their scenario, they have full access to the specific functions being sampled and can eventually eliminate all but the least harmful outlier functions. Their algorithm employs robust mean estimation techniques to estimate the gradient and filter out the outlier functions. Note that their adversarial noise model requires that at least half of the samples are inliers.

This work aims to relax the distributional assumption of heavy-tailed noise such that the fraction of outliers can approach one, and do not have any bounded moment constraints while keeping optimization still computationally tractable. Thus, we ask the following natural question:

What is the weakest noise model for optimization in which efficient learning is possible, while allowing for strong corruption of almost all samples?

In turn, we move beyond prior heavy-tailed noise models with bounded moments and define a new noise model for first-order optimization, inspired by the oblivious noise model studied in the context of regression [Bhatia et al., 2015, d'Orsi et al., 2021b]. We consider the setting in which the gradient oracle returns noisy gradients whose mean may not be ∇ f ( x ) , and in fact, may not even exist. The only condition on our additive noise distribution of ξ is that with probability at least α , oblivious noise takes the value zero. Without this, the optimization problem would become intractable. Notably, oblivious noise can capture any tail behavior that arises from independent p -heavy-tailed noise.

We now formally define the oblivious noise oracle below.

Definition 1.1 (Oblivious Noise Oracle) . We say that an oracle O α,σ,f ( x ) is an oblivious-noise oracle, if, when queried on x , the oracle returns ∇ f ( γ, x ) + ξ where γ is drawn from some arbitrary distribution Q and ξ is drawn from D ξ independent of x and γ , satisfying Pr ξ ∼ D ξ [ ξ = 0] ≥ α . The distribution Q satisfies E γ ∼ Q [ ∇ f ( γ, x )] = ∇ f ( x ) and E γ ∼ Q [ ∥∇ f ( γ, x ) -∇ f ( x ) ∥ 2 ] ≤ σ 2 .

Unfortunately, providing an approximate stationary point ̂ x such that ∥∇ f ( ̂ x ) ∥ ≤ ϵ , for some small ϵ > 0 , is information-theoretically impossible when α ≤ 1 / 2 . To see this, consider the case where Pr γ [ γ = 0] = 1 and the function f (0 , x ) = x 2 . If ξ follows a uniform distribution over the set {-2 , 0 , 2 } , when we query the oracle for a gradient at x , we will observe a uniform distribution over { 2 x -2 , 2 x, 2 x +2 } . This scenario cannot be distinguished from a similar situation where the objective is to minimize f (0 , x ) = ( x -1) 2 , and ξ follows a uniform distribution over { 0 , 2 , 4 } .

To address this challenge, we allow our algorithm to work in the list-decodable setting, implying that it may return a small list of candidate solutions such that one of its elements ̂ x satisfies ∥∇ f ( ̂ x ) ∥ ≤ ϵ . We now define our main problem of interest.

Definition 1.2 (List-Decodable Stochastic Optimization) . The problem of List-Decodable Stochastic Optimization with oblivious noise is defined as follows: For an f that is L -smooth and has a global minimum, given access to O α,σ,f ( · ) as defined in Definition 1.1, the goal is to output a list L of size s such that min ̂ x ∈L ∥∇ f ( ̂ x ) ∥ ≤ ϵ .

1.1 Our Contributions

As our main result, we demonstrate the equivalence between list-decodable stochastic optimization with oblivious noise and list-decodable mean estimation (LDME), which we define below.

Definition 1.3 (List-Decodable Mean Estimation) . Algorithm A is an ( α, β, s ) -LDME algorithm for D (a set of candidate inlier distributions) if with probability 1 -δ A , it returns a list L of size s such that min ˆ µ ∈L ∥ ˆ µ -E x ∼ D [ x ] ∥ ≤ β for D ∈ D when given m A samples { z i + ξ i } m A i =1 for z i ∼ D and ξ i ∼ D ξ where Pr ξ ∼ D ξ [ ξ = 0] ≥ α . If 1 -α is a sufficiently small constant less than 1 / 2 , then s = 1 .

Wedefine D σ to be the following set of distributions over R d : D σ := { D | E D [ ∥ x -E D [ x ] ∥ 2 ] ≤ σ 2 } . We also use ˜ O ( · ) to hide all log factors in d, 1 /α, 1 /η, 1 /δ , where δ denotes the failure probability and η is a multiplicative parameter we use in our algorithm.

Our first theorem shows that if there exists an efficient algorithm for the problem of list-decodable mean estimation and an inexact learner (an optimization method using inexact gradients), there is an efficient algorithm for list-decodable stochastic optimization.

Theorem 1.4 (List-Decodable Stochastic Optimization → List-Decodable Mean Estimation) . Suppose that for any f having a global minimum and being L -smooth, the algorithm A G , given access to g x satisfying ∥ g x -∇ f ( x ) ∥ ≤ O ( ησ ) , recovers ˆ x satisfying ∥∇ f (ˆ x ) ∥ ≤ O ( ησ ) + ϵ in time T G . Let A ME be an ( α, O ( ησ ) , s ) -LDME algorithm for D σ . Then, there exists an algorithm (Algorithm 2) which uses A ME and A G , makes m = m A ME + ˜ O ( T G · ( O (1) /α ) 2 /η /η 5 ) queries to O α,σ,f , runs in time T G · poly( d, 1 /η, ( O (1) /α ) 1 /η , log(1 /δηα )) , and returns a list L of s candidates satisfying min x ∈L ∥∇ f ( x ) ∥ ≤ O ( ησ ) + ϵ with probability 1 -δ A ME .

To achieve this reduction, we develop an algorithm capable of determining the translation between two instances of a distribution by utilizing samples that are subject to additive noise. It is important to note that the additive noise can differ for each set of samples. As far as we know, this is the first guarantee of its kind for estimating the location of a distribution, considering the presence of noise, sample access, and the absence of density information.

Theorem 1.5 (Noisy Location Estimation) . Let η ∈ (0 , 1) and let D ξ , D z , D z ′ be distributions such that Pr ξ ∼ D ξ [ ξ = 0] ≥ α and D z , D z ′ are possibly distinct mean zero distributions with variance bounded by σ 2 . Then there is an algorithm (Algorithm 5) which, for unknown t ∈ R d , takes m = ˜ O ((1 /η 5 )( O (1) /α ) 2 /η ) samples { ξ i + z i + t } i m =1 and { ˜ ξ i + z ′ i } i m =1 , where ξ i and ˜ ξ i are drawn independently from D ξ and z i and z ′ i are drawn from D z and D z ′ respectively, runs in time poly( d, 1 /η, ( O (1) /α ) 1 /η , log(1 /δηα )) , and with probability 1 -δ recovers t ′ such that | t -t ′ | ≤ O ( ησ ) .

The fact that there exist algorithms for list-decodable mean estimation with the inliers coming from D σ allows us to get concrete results for list-decodable stochastic optimization.

Conversely, we show that if we have an algorithm for list-decodable stochastic optimization, then we also get an algorithm for list-decodable mean-estimation. This in turn implies, via Fact 2.1, that an exponential dependence on 1 /η is necessary in the list-size if we want to estimate the correct gradient up to an error of O ( ησ ) when the inliers are drawn from some distribution in D σ .

Theorem 1.6 (List-Decodable Mean Estimation → List-Decodable Stochastic Optimization) . Assume there is an algorithm for List-Decodable Stochastic Optimization with oblivious noise that runs in time T and makes m queries to O α,σ,f , and returns a list L of size s containing ̂ x satisfying ∥ f ( ̂ x ) ∥ ≤ ϵ . Then, there is an ( α, ϵ, s ) -LDME algorithm for D that runs in time T , queries m samples, and returns a list of size s .

If α = 1 -ϵ , where 0 < ϵ < 1 / 2 is at most a sufficiently small constant, the above theorems hold for the same problems, but with the constraint that the list is singleton.

1.2 Related Work

Given the extensive robust optimization and estimation literature, we focus on the most relevant work.

Optimization with Heavy-tailed Noise There is a wealth of literature on both the theoretical and empirical convergence behavior of stochastic gradient descent (SGD) for both convex and non-convex problems, under various assumptions on the stochastic gradient (see, e.g., Hardt et al. [2016], Wang et al. [2021] and references within). However, the noisy gradients have shown to be problematic when training ML models [Shen and Sanghavi, 2019, Zhang et al., 2020], hence necessitating robust optimization algorithms.

From a theoretical point of view, several noise models have been proposed to account for inexact gradients. A line of work [d'Aspremont, 2008, So, 2013, Devolder et al., 2014, Cohen et al., 2018] studies the effects of inexact gradients to optimization methods in terms of error accumulation. For instance, Devolder et al. [2014] demonstrates that, given an oracle that outputs an arbitrary

perturbation of the true gradient, the noise can be determined adversarially to encode non-smooth problems. Alternatively, many recent works [Lan, 2012, Gorbunov et al., 2020, Cutkosky and Mehta, 2021, Mai and Johansson, 2021, Sadiev et al., 2023] have studied p -heavy-tailed noise, an additive stochastic noise to the true gradient where one has access to ∇ f ( γ, x ) such that E γ [ ∇ f ( γ, x )] = ∇ f ( x ) and E γ [ ∥∇ f ( γ, x ) -∇ f ( x ) ∥ p ] ≤ σ p . For instance, Sadiev et al. [2023] propose and analyze a variant of clipped-SGD to provide convergence guarantees for when p ∈ (1 , 2] . However, these noisy oracles, whether deterministic or stochastic, assume bounded norm or moments on the noise, an assumption that is not present in the oblivious noise oracle.

Robust Estimation Our oblivious noise oracle for optimization is motivated by the recent work on regression under oblivious outliers. In the case of linear regression, the oblivious noise model can be seen as the weakest possible noise model that allows almost all points to be arbitrarily corrupted, while still allowing for recovery of the true function with vanishing error [Bhatia et al., 2015, Suggala et al., 2019]. This also captures heavy-tailed noise that may not have any moments. The setting has been studied for various problems, including online regression [Pesme and Flammarion, 2020], PCA [d'Orsi et al., 2021a], and sparse signal recovery [d'Orsi et al., 2022].

On the other hand, there has been a flurry of work on robust estimation in regards to worst-case adversarial outliers (see, e.g., Diakonikolas et al. [2016], Lai et al. [2016], Charikar et al. [2017], Diakonikolas et al. [2019]). Robust mean estimation aims to develop an efficient mean estimator when (1 -α ) -fraction of the samples is arbitrary. In contrast, list-decodable mean estimation generates a small list of candidates such that one of these candidates is a good estimator. While robust mean estimation becomes information-theoretically impossible when α ≤ 1 / 2 , the relaxation to output a list allows the problem to become tractable for any α ∈ (0 , 1] [Charikar et al., 2017, Diakonikolas et al., 2022]. See Diakonikolas and Kane [2019, 2023] for in-depth treatments of the subject.

In the past, robust mean estimators have been used to perform robust gradient estimation (see, e.g., Charikar et al. [2017], Diakonikolas et al. [2019]), which is similar to what we do in our paper. However, these results assume access to the entire function set, which allows them to discard outlier functions. In contrast, in our setting, we only have access to a noisy gradient oracle, so at each step, we get a fresh sample set and, hence, a different set of outliers. This introduces further difficulties , which we resolve via location estimation.

We note a subtle difference in the standard list-decodable mean estimation setting and the setting considered in this work. The standard list-decodable mean estimation setting draws an inlier with probability α and an outlier with the remaining probability. In contrast, our model gets samples of the kind ξ + z where z is drawn from the inlier distribution, and Pr[ ξ = 0] > α , and is arbitrary otherwise. The algorithms for the mixture setting continue to work in our setting as well. Another difference is that the standard setting requires that the distribution have bounded variance in every direction . On the other hand, in the optimization literature, the assumption is that the stochastic gradient has bounded expected squared norm from the expectation.

Location estimation Location estimation has been extensively studied since the 1960s. Traditional approaches to location estimation have focused on achieving optimal estimators in the asymptotic regime. The asymptotic theory of location estimation is discussed in detail in [Van der Vaart, 2000].

Recent research has attempted to develop the finite-sample theory of location estimation. These efforts aim to estimate the location of a Gaussian-smoothed distribution with a sample complexity that matches the optimal sample complexity up a constant (see [Gupta et al., 2022] and [Gupta et al., 2023]). However, these results assume prior knowledge of the likelihood function of the distribution up to translation.

Another closely related work initiates the study of robust location estimation for the case where the underlying high-dimensional distribution is symmetric and an 0 < ϵ ≪ 1 / 2 fraction of the samples are adversarially corrupted Novikov et al. [2023]. This follows the line of work on robust mean estimation discussed above.

We present a finite-sample guarantee for the setting with noisy access to samples drawn from a distribution and its translation. Our assumption on the distribution is that it places an α mass at some point, where α ∈ (0 , 1) and, for instance, could be as small as 1 /d c for some constant c . The noise is constrained to have mean zero and bounded variance, but crucially the noise added to the samples coming from the distribution and its translation might be drawn from different distributions . To the

best of our knowledge, this is the first result that has noisy access, does not have prior knowledge of the probability density of the distribution and achieves a finite-sample guarantee.

1.3 Technical Overview

For ease of exposition, we will make the simplifying assumption that the observation noise is bounded between [ -σ, σ ] . Let y and y ′ be two distinct mean-zero noise distributions, both bounded within the range [ -σ, σ ] . Define ξ to be the oblivious noise drawn from D ξ , satisfying Pr[ ξ = 0] ≥ α . Assume we have access to the distributions (i.e., we have infinite samples).

Stochastic Optimization reduces to Mean Estimation. In Theorem 1.4, we show how we can leverage a list-decodable mean estimator to address the challenge of list-decodable stochastic optimization with oblivious noise (see Algorithm 2). The key idea is to recognize that we can a generate list of gradient estimates, and update this list such that one of the elements always closely approximates the true gradient in ℓ 2 norm at the desired point.

The algorithmic idea is as follows: First, run a list-decodable mean estimation algorithm on the noisy gradients at x 0 = 0 to retrieve a list L consisting of s potential gradient candidates. This set of candidates contains at least one element which closely approximates ∇ f (0) .

One natural approach at this stage would be to perform a gradient descent step for each of the s gradients, and run the list-decodable mean estimation algorithm again to explore all potential paths that arise from these gradients. However, this naive approach would accumulate an exponentially large number of candidate solutions. We use a location-estimation algorithm to tackle this issue.

We can express f ( γ, x ) as f ( x ) + e ( γ, x ) where E γ [ e ( γ, x )] = 0 and E γ [ ∥ e ( γ, x ) ∥ 2 ] < σ 2 . When we query O α,σ,f ( · ) at x , we obtain samples of the form ∇ f ( x ) + ξ + ∇ e ( γ, x ) , i.e., samples from a translated copy of the oblivious noise distribution convolved with the distribution of ∇ e ( γ, x ) . We treat the distribution of ∇ e ( γ, x ) as observation noise. The translation between the distributions of our queries to O α,σ,f ( · ) at x and 0 correspond to ∇ f ( x ) -∇ f (0) . To update the gradient, it is sufficient to recover this translation accurately. By doing so, we can adjust L by translating each element while maintaining the accuracy of the estimate.

Finally, we run a first-order learner for stochastic optimization using these gradient estimates. We select and explore s distinct paths, with each path corresponding to an element of L obtained earlier. Since one of the paths always has approximately correct gradients, this path will converge to a stationary point in the time that it takes for the first-order learner to converge.

Noisy Location Estimation in 1-D. The objective of noisy location estimation is to retrieve the translation between two instances of a distribution by utilizing samples that are subject to additive mean-zero and bounded-variance noise.

In Lemma 3.1, we establish that if Algorithm 1 is provided with a parameter η ∈ (0 , 1) and samples from the distributions of ξ + y and ˜ ξ + y ′ + t , it can recover the value of t within an error of O ( ησ ) . Here ξ and ˜ ξ are independent draws from the same distribution.

Observe that in the absence of additive noise y and y ′ , t can be estimated exactly by simply taking the median of a sufficiently large number of pairwise differences between independent samples from ξ and ˜ ξ + t . This is because the distribution of ˜ ξ -ξ is symmetric at zero and Pr[ ˜ ξ -ξ = 0] = α 2 . However, with unknown y and y ′ , this estimator is no longer consistent since the median of zeromean and σ -variance random variables can be as large as O ( σ ) in magnitude. In fact, Claim 3.2 demonstrates that for the setting we consider the median of ξ + y can as far as O ( σ/ √ α ) far from the mean of ξ . However, it does recover a rough estimate t ′ r such that | t -t ′ r | ≤ O ( σ/ √ α ) .

Consequently, our approach starts by using t ′ r as the estimate for t , and then iteratively improving the estimate by mitigating the heavy-tail influence of ξ . Note that if ξ did not have heavy tails, then we could try to directly compute t = E [ ξ + y ′ + t ] -E [ ξ + y ] . Unfortunately, due to ξ 's completely unconstrained tail, ξ may not even possess a well-defined mean. Nonetheless, if we are able to condition ξ to a bounded interval, we can essentially perform the same calculation described above. In what follows, we describe one step of an iterative process to improve our estimate of t .

Rejection Sampling. Suppose we have an initial rough estimate of t , given by t ′ r such that | t ′ r -t | < Aσ . We can then re-center the distributions to get ξ + z and ξ + z ′ , where z and z ′ have means of magnitude at most Aσ , and have variance that is O ( σ ) . Claim 3.4 then shows that we can refine our estimate of t . It does so by identifying an interval I = [ -kσ, kσ ] around 0 such that the addition of either z or z ′ to ξ does not significantly alter the distribution's mass within or outside of I . We show that since z, z ′ , and ξ are independent, the I that we choose satisfies E [ ξ + z ′ | ξ + z ′ ∈ I ] -E [ ξ + z | ξ + z ∈ I ] = E [ ξ + z ′ | ξ ∈ I ] -E [ ξ + z | ξ ∈ I ] ± ηA = E [ z ′ ] -E [ z ] ± ηA for some η < 1 .

To see that such an interval I exists, we will show that for some i , there are pairs of intervals ( Aσi, ( i +1) Aσ ] and [ -( i +1) Aσ, -iAσ ) which contain a negligible mass with respect to ξ . Since z and z ′ can move mass by at most ( A +1) σ with high probability, it is sufficient to condition on the interval around 0 which is contained in this pair of intervals. Since we do not have access to ξ , we instead search for pairs of intervals of length O ( Aσ ) which have negligible mass with respect to both ξ + z and ξ + z ′ , which suffices.

To do this, we will demonstrate an upper bound ˜ P ( i ) on the mass crossing the i th pair of intervals described above. This will satisfy ∑ i ˜ P ( i ) = C ′ for some constant C ′ .

To show that this implies there is an interval of negligible mass, we aim to demonstrate the existence of a k ∈ N such that k ˜ P ( k ) ≤ η . We will do this through a contradiction. Suppose this is not the case for all i ∈ [0 , k ] , then ∑ k i =0 ˜ P ( i ) ≥ η ∑ k i =0 (1 /i ) . If k ≥ exp(10 C ′ /η ) , we arrive at a contradiction because the right-hand side is at least η ∑ k i =0 (1 /i ) ≥ η log(exp(10 C/η )) > 10 C ′ , while the left-hand side is bounded above by C .

Claim C.4 demonstrates, via a more involved and finer analysis, that for the bounded-variance setting where intervals very far away can contribute to mass crossing Aσi , there exists an k such that k ˜ P ( k ) ≤ η Pr[ | ξ | < Aσk ] , and that taking the conditional mean restricted to the interval [ -kAσ,kAσ ] allows us to improve our estimate of t . Here ˜ P ( k ) denotes an upper bound on the total probability mass that crosses intervals described above.

Extension to Higher Dimensions. In order to extend the algorithm to higher dimensions, we apply the one-dimensional algorithm coordinate-wise, but in a randomly chosen coordinate-basis. Lemma 3.5 uses the fact that representing the distributions in a randomly rotated basis ensures that, with high probability, the inlier distribution will project down to each coordinate with a variance of O ( σ √ log( d ) / √ d ) to extend Lemma 3.1 to higher dimensions.

Mean Estimation reduces to Stochastic Optimization. In Theorem 1.6 we show that the problem of list-decodable mean estimation can be solved by using list-decodable stochastic optimization for the oblivious noise setting. This establishes the opposite direction of the reduction to show the equivalence of list-decodable mean estimation and list-decodable stochastic optimization.

The reduction uses samples from the list-decodable mean estimation problem to simulate responses from the oblivious oracle to queries at x . Let the mean of the inlier distribution be µ . If the first-order stochastic learner queries x , we return x + s where s is a sample drawn from the listdecodable mean-estimation problem. These correspond to possible responses to the queries when f ( x ) = (1 / 2) ∥ x + µ ∥ 2 , where µ is the true mean. The first-order stochastic learner learns a ̂ µ from a list L such that ∥∇ f ( ̂ µ ) ∥ = ∥ ̂ µ + µ ∥ ≤ ϵ ; then the final guarantee of the list-decodable mean estimator follows by returning -L which contains -̂ µ .

2 Preliminaries

Basic Notation For a random variable X , we use E [ X ] for its expectation and Pr [ X ∈ E ] for the probability of the random variable belonging to the set E . We use N ( µ, σ 2 ) to denote the Gaussian distribution with mean µ and variance matrix σ 2 . When D is a distribution, we use X ∼ D to denote that the random variable X is distributed according to D . When S is a set, we let E X ∼ S [ · ] denote the expectation under the uniform distribution over S . When clear from context, we denote the empirical expectation and probability by ̂ E and ̂ Pr . We denote ∥ · ∥ as the ℓ 2 -norm and assume f : R d → R is differentiable and L -smooth, i.e., ∥∇ f ( x ) -∇ f ( x ′ ) ∥ ≤ L ∥ x -x ′ ∥ for all x, x ′ ∈ R d .

ξ will always denote the oblivious noise drawn from a distribution Q satisfying Pr ξ ∼ Q [ ξ = 0] ≥ α . y, y ′ will be used to denote mean-zero and variance at-most σ 2 random variables. Also define e ( γ, x ) := f ( γ, x ) -f ( x ) and the interval σ ( a, b ] := ( σa, σb ] .

Facts We use these algorithmic facts in the following sections. We use a list-decodable robust mean estimation subroutine in a black-box manner. The proof for list-decodable mean estimation can be found in Appendix D, while such algorithms can be found in prior work, see, e.g., Charikar et al. [2017], Diakonikolas et al. [2020a]. We also define a ( β, ϵ ) -inexact-learner for f .

Fact 2.1 (List-decoding algorithm) . There is an ( α, ση, ˜ O ((1 /α ) 1 /η 2 )) -LDME algorithm for the an inlier distribution belonging to D σ which runs in time ˜ O ( d (1 /α ) 1 /η 2 ) and succeeds with probability 1 -δ . Conversely, any algorithm which returns a list, one of which makes an error of at most O ( ησ ) in ℓ 2 norm to the true mean, must have a list whose size grows exponentially in 1 /η .

If 1 -α is a sufficiently small constant less than half, then the list size is 1 to get an error of O ( √ 1 -α σ ) .

Wenowdefine the notion of a robust-learner, which is an algorithm which, given access to approximate gradients is able to recover a point at which the gradient norm is small.

Definition 2.2 (Robust Inexact Learner) . Let f be a L s -smooth function with a global minimum and O grad β,f ( · ) be an oracle which when queried on x returns a vector g x satisfying ∥ g x -∇ f ( x ) ∥ ≤ β . We say that an algorithm A G is an ( β, ϵ ) -inexact-learner for f if, given access to O grad β,f , the algorithm A G recovers ˆ x satisfying ∥∇ f (ˆ x ) ∥ ≤ β + ϵ . We will assume A G runs in time T G .

Several algorithms for convex optimization amount to there being an existence of a robust learner in the convex setting. More recently, for smooth nonconvex optimization, the convergence result for SGD under inexact gradients due to Ajalloeian and Stich [2020] doubles as a ( β, ϵ ) -inexact-learner running in T G = O ( LF/ ( β + ϵ ) 2 ) iterations, where F = f ( x 0 ) -min x f ( x ) and x 0 is the initial point. This follows by an application of Theorem 4 from Ajalloeian and Stich [2020] and by taking the point in the set of iterates that minimizes the gradient norm.

3 Location Estimation

To estimate how the gradient changes between iterations, we will need to estimate the shift between a distribution and its translation. In this section, we give an algorithm which, given access to samples from a distribution and its noisy translation returns an estimate of t accurate up to an error of O ( ησ ) in ℓ 2 -norm. We show a proof sketch here, and a more detailed proof of all the claims involved in Appendix C.

Lemma 3.1 (One-dimensional location-estimation) . There is an algorithm (Algorithm 1) which, for m = (1 /η 5 )( O (1) /α ) 2 /η log(1 /ηαδ ) , given samples { ξ i + y i + t } i m =1 and { ˜ ξ i + y ′ i } i m =1 where ξ i and ˜ ξ i are both drawn from D ξ , y i and y ′ i are drawn from distinct distributions with meanzero with variance bounded above by σ and t ∈ R is an unknown translation, runs in time ˜ poly(1 /η, ( O (1) /α ) 1 /η , log(1 /δηα )) and recovers t ′ such that | t -t ′ | ≤ O ( ησ ) .

Proof. We first identify t up to an error of O ( σ/ √ α ) with the following claim.

Claim 3.2 (Rough Estimate) . There is an algorithm which, for m = O ((1 /α 4 ) log(1 /δ )) given samples { ξ i + y i + t } i m =1 and { ˜ ξ i + y ′ i } i m =1 where ξ i and ˜ ξ i are both drawn from D ξ , y i and y ′ i are drawn from distinct distributions with mean-zero with variance bounded above by σ and t ∈ R is an unknown translation, returns | t ′ r -t | ≤ O ( σα -1 / 2 ) .

We use t ′ r to center the two distributions up to an error of σα -1 / 2 . Let the centered distributions be given by ξ + z and ξ + z ′ where z, z ′ are independent, have means that are at most σα -1 / 2 in magnitude and have variance that is bounded by 4 σ 2 .

Let A ∆ B denote the symmetric difference of the two events A,B . To improve our estimate further, we restrict our attention to an interval I = σ ( -i, i ) such that Pr [( ξ + z ∈ I ) ∆ ( ξ ∈ I )] <

Algorithm 1 One-dimensional Location Estimation: Shift1D ( S 1 , S 2 , η, σ, α )

Input: Sample sets S 1 , S 2 ⊂ R d of size m , α, η ∈ (0 , 1) , σ > 0

Let T = O (log 1 /η (1 /α )) . For j ∈ { 1 , 2 } , partition S j into T equal pieces, S ( i ) j for i ∈ [ T ] .

D = { a -b | a ∈ S (1) 1 , b ∈ S (1) 2 } .

Set A = O (1 / α ) .

t ′ (1) := median( D ) . √

Repeat steps 6 to 12, for i going from 2 to T :

S ( i ) 1 := S ( i ) 1 -t ′ r ( i -1) .

For j ∈ { 1 , 2 }

Let ˆ P ( i ) = ˆ P 1 ( i ) + ˆ P 2 ( i ) .

Identify an integer k ∈ [(1 /αη ) , ( C/α +1 / ( αη ) η ) 1 /η ] such that

11. A := ηA .

12. Return t ′ ( T )

O ( η Pr [ ξ ∈ I ]) and Pr [( ξ + z ′ ∈ I ) ∆ ( ξ ∈ I )] < O ( η Pr [ ξ ∈ I ]) , i.e., neither z nor z ′ moves more than a negligible mass of ξ either into, or out of I .

To detect such an interval (if it exists), we control the total mass contained in, and moved across the intervals σ ( -i -1 , -i ) and σ ( i, i +1) when z or z ′ are added to ξ . We denote this mass by P ′′ ( i ) . Claim C.4 shows that there is a function ˜ P ( · ) which we can compute using our samples from the distributions of ξ + z and ξ + z ′ , which serves as a good upper bound on P ′′ ( i ) .

Claim 3.3. There exists a function ˜ P : N → R + which satisfies the following:

For all i ∈ N , ˜ P ( i ) ≥ P ( i, z ) + P ( i, z ′ ) which can be computed using samples from ξ + z and ξ + z ′ .

There is a k ∈ [(1 /αη ) , ( C/α +1 / ( αη ) η ) 1 /η ] such that k ˜ P ( k ) < η ∑ k j =0 ˜ P ( k ) .

∑ k j =0 ˜ P ( k ) = O ( Pr [ | ξ + z | ≤ Aσk ] + Pr [ | ξ + z ′ | ≤ Aσk ]) .

With probability 1 -δ , for all i < ( O (1) /α ) 1 /η /η , ˜ P ( i ) can be estimated to an accuracy of O ( η/i ) by using ( O (1) /α ) 2 /η log(1 /ηαδ ) /η 5 samples from the distributions of ξ + z and ξ + z ′ .

Claim 3.4 shows that if such a ˜ P ( · ) exists, then the difference of the conditional expectations suffices to get a good estimate. This is because conditioning on | ξ + z | < 10 kσ satisfying conditions (2) and (3) above is almost the same as conditioning on | ξ | < 10 kσ .

Claim 3.4. Suppose i > ( ηα ) -1 and ˜ P ( · ) satisfies the conclusions of Claim C.4. If z, z ′ have Aσ bounded means and 4 σ 2 bounded variances for some A , then E [ ξ + z | | ξ + z | ≤ Aσi ] -E [ ξ + z ′ | | ξ + z ′ | ≤ Aσi ] = E [ z ] -E [ z ′ ] ± O ( Aησ )

The conclusion of the one-dimensional location estimation theorem then follows by putting these together, and iterating the above steps after re-centering the means. The sample complexity is governed by the following considerations:

(1) We require the E [ ξ + z | | ξ + z | ≤ σα -1 / 2 k ] -E [ ξ + z ′ | | ξ + z ′ | ≤ σα -1 / 2 k ] to concentrate around its mean for some i ≤ σ ( C/α +1 / ( αη ) η ) 1 /η . (2) We require Pr [ ξ + z ∈ σ ( i, i +1)] to concentrate for all integer i of magnitude at most σ ( C/α +1 / ( αη ) η ) 1 /η .

An application of Hoeffding's inequality (Lemma A.1) and a union bound over all the events prove that it suffices to ensure sample complexity to be m ≥ poly(1 /η, ( O (1) /α ) 1 /η , log(1 /δηα )) .

The following Lemma 3.5 uses the algorithm for one dimension to derive a higher-dimensional guarantee. The idea is to perform a random rotation, which, with high probability, ensures that the variance of the distribution is O ( σ √ log( d ) / √ d ) along each basis and then apply the one-dimensional algorithm coordinate-wise according to the random bases. We defer the proof to Appendix C. The algorithm referenced in this lemma is named 'ShiftHighD", which we use later.

Lemma 3.5 (Location Estimation) . Let D y i for i ∈ { 1 , 2 } be the distributions of ξ + z i for i ∈ { 1 , 2 } where ξ is drawn from D ξ and Pr ξ ∼ D ξ [ ξ = 0] ≥ α and z i ∼ D i are distributions over R d satisfying E D i [ x ] = 0 and E D i [ ∥ x ∥ 2 ] ≤ σ 2 . Let v ∈ R d be an unknown shift, and D y 2 ,v denote the distribution of y 2 shifted by v . There is an algorithm (Algorithm 5), which draws poly(1 /η, ( O (1) /α ) 1 /η , log(1 /δηα )) samples each from D y 1 and D y 2 ,v runs in time poly( d, 1 /η, ( O (1) /α ) 1 /η , log(1 /δηα )) and returns v ′ satisfying ∥ v ′ -v ∥ ≤ O ( ησ ) with probability 1 -δ .

4 Algorithmic Results for Stochastic Optimization

Our first main result demonstrates that if we have access to an algorithm that recovers a list of size s such that one of the means is O ( ησ ) -close to the true mean in ℓ 2 norm, then there is an algorithm which is able to recover ˆ x such that ∥∇ f (ˆ x ) ∥ ≤ O ( ησ ) + ϵ given access to gradients that are close to the true gradient up to an ℓ 2 error of O ( ησ ) .

4.1 Proof of Theorem 1.4: List-decodable Stochastic Optimization → LDME

Proof. The key to our proof is to recognize that at every step, we effectively have an oracle for an inexact gradient of f in the sense of O grad O ( ησ ) ,f as defined in Definition 2.2.

Claim 4.1. Given an initial list L 0 := { g 1 , . . . , g s } such that there is some fixed i ∈ [ s ] for which ∥ g i -∇ f (0) ∥ ≤ O ( ησ ) , a point x , access to O α,σ,f ( · ) , Algorithm 3 returns a list L := { g ′ 1 , . . . , g ′ s } such that ∥ g ′ i -∇ f ( x ) ∥ ≤ O ( ησ ) for the same i .

Proof. O α,σ,f ( y ) returns samples of the kind { ξ i + e ( γ, x ) i + ∇ f ( x ) } i m =1 and { ˜ ξ i + e ( γ, 0) i + ∇ f (0) } j m =1 for y = x and y = 0 , with e ( γ, y ) i being drawn from a distribution with 0 and variance bounded by σ 2 and ξ i and ˜ ξ i being drawn from D ξ where Pr ξ ∼ D ξ [ ξ = 0] ≥ α . Hence, one can interpret the samples drawn as being in the setting of Lemma 3.5 with the shift v = ∇ f (0) -∇ f ( x ) .

Let S 1 and S 2 be drawn from O α,σ,f ( y ) with y = 0 and y = x . Running Algorithm 5 on S 1 , S 2 , we recover v satisfying ∥ v -( ∇ f ( x ) -∇ f (0)) ∥ ≤ O ( ησ ) . A triangle inequality now tells us that if we set g ′ i := g i + v , we get ∥ g ′ i - ∇ f ( x ) ∥ = ∥ g ′ i -g i + g i - ∇ f (0) + ∇ f (0) - ∇ f ( x ) ∥ ≤ O ( ησ ) + O ( ησ ) = O ( ησ ) .

g ′ i can be interpreted as the output of O grad O ( ησ ) ,f ( x ) , since ∥ g ′ i -∇ f ( x ) ∥ ≤ O ( ησ ) . The result then follows from the guarantees of A G , since for at least one of the sequences of gradients, Algorithm 2 replicates every step of A G .

Substituting the guarentees of Fact 2.1 for the list-decoding algorithm in the above theorem then gives us the following corollary, the proof of which we defer to Appendix E.

Corollary 4.2. Given access to oblivious noise oracle O α,σ,f , a ( O ( ησ ) , ϵ ) -inexact-learner A G running in time T G , there exists an algorithm which takes poly(( O (1) /α ) 1 /η 2 , log( T G /δηα )) samples, runs in time T G · poly( d, ( O (1) /α ) 1 /η 2 , log(1 /ηαδ )) , and with probability 1 -δ returns a list L of size ˜ O ((1 /α ) 1 /η 2 ) such that min x ∈L ∥∇ f ( x ) ∥ ≤ O ( ησ ) + ϵ . Additionally, the exponential dependence on 1 /η in the size of the list is necessary.

4.2 Proof of Theorem 1.6: LDME → List-Decodable Stochastic Optimization

In this subsection, we show the converse of the results from the previous subsection, i.e. that list-decodable stochastic optimization can be used to perform list-decodable mean estimation.

Proof. Assume there exists an algorithm A that can recover a s -sized list containing ̂ x such that ∥∇ f ( ̂ x ) ∥ ≤ ϵ when given access to O o α,σ,f ( · ) . From the list-decodable mean estimation setting, denote D to be the distribution of ξ + z where E [ z ] = µ and E [ ∥ z -µ ∥ 2 ] < σ 2 , and Pr [ ξ = 0] ≥ α .

The goal of LDME is to recover µ from samples from D . We will show that we can recover an s -sized list that contains a ̂ µ satisfying ∥ ̂ µ -µ ∥ ≤ ϵ .

We do this by simulating the oracle O α,σ, (1 / 2) ∥ x + µ ∥ 2 ( · ) for A . To do this, whenever A asks for a query at x , we return x + p where we sample p ∼ D of the list-decodable setting. This effectively simulates the oracle for the objective function f ( x ) = (1 / 2) ∥ x + µ ∥ 2 , where the oblivious noise ξ is the same, and the observation noise is ( z -µ ) . Hence A will return a list L of size s containing ̂ x satisfying ∥ ̂ x + µ ∥ ≤ ϵ . Finally, return -L . This is the solution to list-decodable mean estimation because -L contains ̂ µ := -̂ x , which satisfies ∥ ̂ µ -µ ∥ ≤ ϵ .

5 Conclusion

In this paper, we have initiated the study of stochastic optimization in the presence of oblivious noise, which extends the traditional heavy-tailed noise framework. In our setting, the stochastic gradient is additionally affected by independent oblivious noise that lacks bounded moments and may not be centered. We have also designed an algorithm for finite-sample noisy location estimation based on taking conditional expectations, which we believe is of independent interest. We note that while the exponential dependence on 1 /η for the size of the list is unavoidable, but it is an open problem to show that this is the case for the problem of noisy location estimation.

References

Ajalloeian and S. U. Stich. Analysis of sgd with biased gradient estimators. arXiv preprint arXiv:2008.00051 , 2020.

Bhatia, P. Jain, and P. Kar. Robust regression via hard thresholding. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015 , pages 721-729, 2015.

Charikar, J. Steinhardt, and G. Valiant. Learning from untrusted data. In Proc. 49th Annual ACM Symposium on Theory of Computing , pages 47-60, 2017.

Cohen, J. Diakonikolas, and L. Orecchia. On acceleration with noise-corrupted gradients. In International Conference on Machine Learning , pages 1019-1028. PMLR, 2018.

Cutkosky and H. Mehta. High-probability bounds for non-convex stochastic optimization with heavy tails. Advances in Neural Information Processing Systems , 34:4883-4895, 2021.

d'Aspremont. Smooth optimization with approximate gradient. SIAM Journal on Optimization , 19(3):1171-1183, 2008.

Devolder, F. Glineur, and Y. Nesterov. First-order methods of smooth convex optimization with inexact oracle. Mathematical Programming , 146:37-75, 2014.

Diakonikolas and D. M. Kane. Recent advances in algorithmic high-dimensional robust statistics. CoRR , abs/1911.05911, 2019. URL http://arxiv.org/abs/1911.05911 .

Diakonikolas and D. M. Kane. Algorithmic High-Dimensional Robust Statistics . Cambridge university press, 2023. Full version available at https://sites.google.com/view/ars-book.

Diakonikolas, G. Kamath, D. M. Kane, J. Li, A. Moitra, and A. Stewart. Robust estimators in high dimensions without the computational intractability. In Proc. 57th IEEE Symposium on Foundations of Computer Science (FOCS) , pages 655-664, 2016.

Diakonikolas, G. Kamath, D. M. Kane, J. Li, J. Steinhardt, and A. Stewart. SEVER: A robust meta-algorithm for stochastic optimization. In Proc. 36th International Conference on Machine Learning (ICML) , pages 1596-1606, 2019.

Diakonikolas, D. Kane, and D. Kongsgaard. List-decodable mean estimation via iterative multifiltering. Advances in Neural Information Processing Systems , 33:9312-9323, 2020a.

Diakonikolas, D. M. Kane, and A. Pensia. Outlier robust mean estimation with subgaussian rates via stability. Advances in Neural Information Processing Systems , 33:1830-1840, 2020b.

Diakonikolas, D. Kane, S. Karmalkar, A. Pensia, and T. Pittas. List-decodable sparse mean estimation via difference-of-pairs filtering. Advances in Neural Information Processing Systems , 35:13947-13960, 2022.

d'Orsi, C. H. Liu, R. Nasser, G. Novikov, D. Steurer, and S. Tiegel. Consistent estimation for pca and sparse regression with oblivious outliers. Advances in Neural Information Processing Systems , 34:25427-25438, 2021a.

d'Orsi, G. Novikov, and D. Steurer. Consistent regression when oblivious outliers overwhelm. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 2297-2306. PMLR, 18-24 Jul 2021b. URL http://proceedings.mlr.press/v139/d-orsi21a.html .

d'Orsi, R. Nasser, G. Novikov, and D. Steurer. Higher degree sum-of-squares relaxations robust against oblivious outliers. CoRR , abs/2211.07327, 2022. doi: 10.48550/arXiv.2211.07327. URL https://doi.org/10.48550/arXiv.2211.07327 .

Gorbunov, M. Danilova, and A. Gasnikov. Stochastic optimization with heavy-tailed noise via accelerated gradient clipping. Advances in Neural Information Processing Systems , 33:1504215053, 2020.

Gupta, J. Lee, E. Price, and P. Valiant. Finite-sample maximum likelihood estimation of location. Advances in Neural Information Processing Systems , 35:30139-30149, 2022.

Gupta, J. C. Lee, and E. Price. High-dimensional location estimation via norm concentration for subgamma vectors. arXiv preprint arXiv:2302.02497 , 2023.

Gurbuzbalaban and Y. Hu. Fractional moment-preserving initialization schemes for training deep neural networks. In International Conference on Artificial Intelligence and Statistics , pages 2233-2241. PMLR, 2021.

Gurbuzbalaban, U. Simsekli, and L. Zhu. The heavy-tail phenomenon in sgd. In International Conference on Machine Learning , pages 3964-3975. PMLR, 2021.

Hardt, B. Recht, and Y. Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning , pages 1225-1234. PMLR, 2016.

Hodgkinson and M. Mahoney. Multiplicative noise and heavy tails in stochastic optimization. In International Conference on Machine Learning , pages 4262-4274. PMLR, 2021.

A. Lai, A. B. Rao, and S. Vempala. Agnostic estimation of mean and covariance. In Proc. 57th IEEE Symposium on Foundations of Computer Science (FOCS) , pages 665-674, 2016.

Lan. An optimal method for stochastic composite optimization. Mathematical Programming , 133 (1-2):365-397, 2012.

Z. Li, D. M. Absher, H. Tang, A. M. Southwick, A. M. Casto, S. Ramachandran, H. M. Cann, G. S. Barsh, M. Feldman, L. L. Cavalli-Sforza, and R. M. Myers. Worldwide human relationships inferred from genome-wide patterns of variation. Science , 319:1100-1104, 2008.

V. Mai and M. Johansson. Stability and convergence of stochastic gradient clipping: Beyond lipschitz continuity and smoothness. In International Conference on Machine Learning , pages 7325-7335. PMLR, 2021.

V. Nazin, A. S. Nemirovsky, A. B. Tsybakov, and A. B. Juditsky. Algorithms of robust stochastic optimization based on mirror descent method. Automation and Remote Control , 80:1607-1627, 2019.

Novikov, D. Steurer, and S. Tiegel. Robust mean estimation without a mean: Dimensionindependent error in polynomial time for symmetric distributions, 2023.

Paschou, J. Lewis, A. Javed, and P. Drineas. Ancestry informative markers for fine-scale individual assignment to worldwide populations. Journal of Medical Genetics , 47:835-847, 2010.

Pesme and N. Flammarion. Online robust regression via sgd on the l1 loss. Advances in Neural Information Processing Systems , 33:2540-2552, 2020.

Rosenberg, J. Pritchard, J. Weber, H. Cann, K. Kidd, L. Zhivotovsky, and M. Feldman. Genetic structure of human populations. Science , 298:2381-2385, 2002.

Sadiev, M. Danilova, E. Gorbunov, S. Horváth, G. Gidel, P. Dvurechensky, A. Gasnikov, and P. Richtárik. High-probability bounds for stochastic optimization and variational inequalities: the case of unbounded variance. arXiv preprint arXiv:2302.00999 , 2023.

Shen and S. Sanghavi. Learning with bad training data via iterative trimmed loss minimization. In International Conference on Machine Learning , pages 5739-5748. PMLR, 2019.

Simsekli, L. Sagun, and M. Gurbuzbalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning , pages 5827-5837. PMLR, 2019.

M.-C. So. Non-asymptotic convergence analysis of inexact gradient methods for machine learning without strong convexity, 2013.

S. Suggala, K. Bhatia, P. Ravikumar, and P. Jain. Adaptive hard thresholding for near-optimal consistent robust regression. In Conference on Learning Theory , pages 2892-2897. PMLR, 2019.

A. W. Van der Vaart. Asymptotic statistics , volume 3. Cambridge university press, 2000.

Wang, M. Gurbuzbalaban, L. Zhu, U. Simsekli, and M. A. Erdogdu. Convergence rates of stochastic gradient descent under infinite noise variance. Advances in Neural Information Processing Systems , 34:18866-18877, 2021.

Zhang, S. P. Karimireddy, A. Veit, S. Kim, S. Reddi, S. Kumar, and S. Sra. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems , 33: 15383-15393, 2020.

Supplementary Material

Organization In Appendix A, we state some elementary probabilistic facts. The next two sections focus on proving our lemma on noisy location estimation. In Appendix B, we prove some critical lemmas used in the proof, and in Appendix C, we present the complete version of our location estimation algorithm, while addressing some typos in the main text. We mention some typos in footnotes and correct other minor typos without a mention.

Moving forward, in Appendix D, we introduce an algorithm and prove a hardness result for the specific version of list-decodable mean estimation we consider, which differs from prior work. Finally, in Appendix E, we state the final guarantees we can get for the problem of list-decodable stochastic optimization, incorporating our lemma from Appendix D.

A Elementary Probability Facts

In this section, we recall some elementary lemmas from probability theory.

Lemma A.1 (Hoeffding) . Let X 1 , . . . X n be independent random variables such that X i ∈ [ a i , b i ] . Let S n := 1 n ∑ n i =1 X i , then for all t > 0

Lemma A.2 (Multivariate Chebyshev) . Let X 1 , . . . , X m be independent random variables drawn from D where D is a distribution over R d such that E X ∼ D [ X ] = 0 and E X ∼ D [ ∥ X ∥ 2 ] ≤ σ 2 . Let S m := 1 m ∑ m i =1 X i , then for all t > 0

Proof. We first prove the following upper bound,

Since the inequality holds for all unit v , it also holds for the unit v in the direction of S m , completing the proof.

Fact A.3 (Inflation via conditional probability) . Let y be a random variable with mean µ and variance σ 2 and let ξ be an arbitrary random variable independent of y , then

Proof. To do this, we inflate the intervals and use conditional probabilities.

The second inequality above follows from observing that we are simply lengthening the interval.

We will often use the second version for ease of analysis.

B Useful Lemmas

In this section, we present some helpful lemmas for the algorithm on noisy one-dimensional location estimation.

To recap the setting: We can access samples from distributions ξ + y and ξ + y ′ + t . Here, Pr [ ξ = 0] > α , y and y ′ are distributions with zero mean and bounded variance, and t ∈ R is an unknown translation. Our objective is to estimate the value of t .

B.1 Useful Lemma for Rough Estimation

Our algorithm for one-dimensional location estimation consists of two steps. In the first step, we obtain an initial estimate of the shift between the two distributions by computing pairwise differences of samples drawn from each distribution. This involves taking the median of the distribution of x + y , where x is symmetric and y has mean 0 and bounded variance.

The following lemma demonstrates that the median of this distribution is at most O ( σα -1 / 2 ) , where σ is the standard deviation of y . Furthermore, this guarantee cannot be improved.

Fact B.1 (Median of Symmetric + Bounded-variance Distribution) . Let x be a random variable symmetric around 0 such that α ∈ (0 , 1) , Pr [ x = 0] ≥ α . Let y be a random variable with mean 0 and variance σ 2 . If S is a set of O (1 /α 2 log(1 /δ )) samples drawn from the distribution of x + y , | median( S ) | ≤ O ( σα -1 / 2 ) .

This guarantee is tight in the sense that there exist distributions for x and y satisfying the above constraints, such that median( x + y ) can be as large as Ω( σα -1 / 2 ) .

Proof. We show that Pr [ x + y < -O ( σ/ √ α )] < 0 . 5 and Pr [ x + y > O ( σ/ √ α )] < 0 . 5 , as a result, | median( x + y ) | < O ( σ/ √ α ) . We will later transfer this guarantee to the uniform distribution over the samples.

Applying Fact A.3 to the first probability, we see that Pr [ x + y < -O ( σ/ √ α )] < (1 + α ) Pr [ x < -O ( σ/ √ α )] .

Since Pr [ x = 0] ≥ α , we see that Pr [ x < 0] ≤ 1 / 2 -α ,

The upper bound follows similarly.

Since Pr [ x = 0] ≥ α and Pr [ | y | < O ( σ/ √ α )] ≥ 1 -α , we see Pr [ | x + y | < O ( σ/ √ α )] ≥ α/ 2 . Hoeffding's inequality (Lemma A.1) now implies that the empirical median also satisfies the above upper bound as long as the number of samples is greater than O (1) /α 2 log(1 /δ ) .

To see that this is tight, consider the distribution centered at 0 , whose density function is 2 / ( y +2) 3 in the range [1 , ∞ ) , and is 0 otherwise.

Let x be a symmetric distribution whose distribution takes the value 0 with probability α and takes the values ± α -1 / 2 100 C 1 / 2 +10 with probability 0 . 5(1 -α ) .

We show that the median of the distribution of x + y where y is drawn from D y -3 , is larger than Ω( α -1 / 2 ) .

To see this, we show that the probability that x + y takes a value smaller than 100 α -1 / 2 C 1 / 2 is less than half, implying that the median has to be larger than this quantity.

x takes three values. Note that y + 100 α -1 / 2 C 1 / 2 + 12 places no mass in the region ( -∞ , 100 α -1 / 2 C 1 / 2 + 10] . So to estimate the probability that x + y takes a value smaller than 100 α -1 / 2 C 1 / 2 +10 , we only need to consider contributions from the other two possible values. By

choosing α small enough, so that 100 α -1 / 2 C 1 / 2 > 10 , we see

We are done when 0 . 5 α -α 1 / 2 / 8 · (400 2 ) < 0 , this happens for α 1 / 2 < 2 / (8 · (400 2 )) .

B.2 Useful Lemma for Finer Estimation

In the second step of our location-estimation lemma, we refine the estimate of t . To do this, we first re-center the distributions based on our rough estimate, so that the shift after re-centering is bounded. Then, we identify an interval I centered around 0 such that, when conditioning on ξ + z falling within this interval, the expected value of ξ + z remains the same as when conditioning on ξ falling within the same interval. This expectation will help us get an improved estimate, which we use to get an improved re-centering of our original distributions, and repeat the process.

To identify such an interval, we search for a pair of bounded-length intervals equidistant from the origin (for e.g. ( -10 σ, -5 σ ) and (5 σ, 10 σ ) ) that contain very little probability mass. By doing so, when z is added to ξ , the amount of probability mass shifted into the interval ( -5 σ, 5 σ ) z remains small.

In this subsection, we prove Lemma B.2, which states that any positive sequence which has a finite sum must eventually have one small element. The lemma also gives a concrete upper bound on which element of the sequence satisfies this property.

Lemma B.2. a i ≥ 0 for all i and ∑ ∞ i =0 a i < C for some constant C . Also, suppose we have η ∈ (0 , 1) and L ∈ R such that L ≥ 1 . Then there is an integer i such that L ≤ i ≤ ( C/a 0 + L η ) 1 /η and ia i < η ∑ i j =0 a j .

Consider a partition of the reals into length L intervals. Here, L simply denotes the length of the intervals and the lower bound to i , not the smoothness parameter.

In our proof, we will use Lemma B.2 on the sequence a i , where a i corresponds to an upper bound on the mass of ξ contained in the i -th intervals equidistant from the origin on either side, and the mass that crosses them (i.e., the mass of ξ that is moved either inside or out of the interval when z is added to it).

We need the following calculation to prove Lemma B.2.

Notation: For integer i ≥ 1 and η ∈ (0 , 1) , define ( i -η )! := Π i j =1 ( j -η ) .

Proof. We prove this by induction. By definition, our hypothesis holds for A 2 because A 2 = 1 + η/ (1 -η ) = 1 / (1 -η ) = (2 -1)! / (2 -1 -η )! . Suppose it holds for all 2 ≤ t ≤ k . We then show that it holds for t = k +1 .

Proof of Lemma B.2 Let U = ( C/a 0 + L η ) 1 /η and suppose towards a contradiction that there is no such i that satisfies the lemma. That means, we will assume that ia i ≥ η ∑ i j =0 a j for all integers i ∈ [1 , U ] ; or equivalently, we assume a i ≥ η i -η ∑ i -1 j =0 a j . We then show that this implies i 1 -η a i ≥ ηa 0 for all i in the range.

Consider the inductive hypothesis on t given by a t ≥ η ( t -1)! ( t -η )! · a 0 . The base case when t = 1 is true since a 1 ≥ ηa 0 / (1 -η ) by our assumption. Suppose the inductive hypothesis holds for integers t ∈ [1 , k -1] . We show this for t = k below. By starting with our assumption then applying the inductive hypothesis, we have that

The final equality follows from Fact B.3 which states that ( k -1)! / ( k -1 -η )! = 1 + ∑ k -1 t =1 η ( t -1)! ( t -η )! . Simplifying this further, we see that since ( i -η ) ≥ i exp( -η/i ) for all i ∈ [1 , k ] ,

By definition, U = ( C/a 0 + L η ) 1 /η so the above inequality cannot be true and thus we have arrived at a contradiction. Then it follows that there exists an i such that 1 ≤ L ≤ i ≤ ( C/a 0 + L η ) 1 /η and ia i < η ∑ i j =0 a j .

C Noisy Location Estimation

In this section, we state and prove the guarantees of our algorithms for noisy location estimation (Lemma 3.1 and Lemma 3.5).

C.1 One-dimensional Noisy Location Estimation

Throughout the technical summary and some parts of the proof, we make the assumption that the variables y and y ′ were bounded. Extending this assumption to bounded-variance distributions requires significant effort.

Finally, observe that

Our algorithm for one-dimensional noisy location estimation (Algorithm 1) can be thought of as a two-step process. The first step involves a rough initial estimation algorithm, while the second step employs an iterative algorithm that progressively refines the estimate by a factor of η in each iteration.

Due to space limitations and for ease of exposition, the algorithm we present in the main body is a sketch of the refinement procedure.

In this Algorithm 1 , we introduce the definition of ˆ P (the empirical estimate of ˜ P ( · ) ), which is an upper bound on the probability mentioned earlier. This probability can be calculated using samples from ξ + z and ξ + z ′ . Additionally, we incorporate the iterative refinement process within the algorithm.

Algorithm 4 One-dimensional Location Estimation: Shift1D ( S 1 , S 2 , η, σ, α )

Input: Sample sets S 1 , S 2 ⊂ R d of size m , α, η ∈ (0 , 1) , σ > 0

Let T = O (log 1 /η (1 /α )) . For j ∈ { 1 , 2 } , partition S j into T equal pieces, S ( i ) j for i ∈ [ T ] .

D = { a -b | a ∈ S (1) 1 , b ∈ S (1) 2 } .

Set A = O (1 / α ) .

t ′ (1) := median( D ) . √

Repeat steps 6 to 12, for i going from 2 to T :

S ( i ) 1 := S ( i ) 1 -t ′ r ( i -1) .

For j ∈ { 1 , 2 }

Let ˆ P ( i ) = ˆ P 1 ( i ) + ˆ P 2 ( i ) .

Identify an integer k ∈ [(1 /αη ) , ( C/α +1 / ( αη ) η ) 1 /η ] such that

z

∼

S

(

i

1

)

[

z

|

|

z

Aσk

]

-

E

z

∼

S

(

i

2

)

[

z

|

|

z

Aσk

10.

t

′

(

i

) :=

t

′

(

i

-

11. A := ηA .

12. Return t ′ ( T )

LemmaC.1 (One-dimensional location-estimation) . There is an algorithm (Algorithm 1) which, given (1 /η 5 )( O (1) /α ) 2 /η log(1 /ηαδ ) samples { ξ i + y i + t } i m =1 and { ˜ ξ i + y ′ i } i m =1 where ξ i and ˜ ξ i are both drawn from D ξ , y i and y ′ i are drawn from distinct distributions with mean-zero with variance bounded above by σ and t ∈ R is an unknown translation, runs in time ˜ poly(1 /η, ( O (1) /α ) 1 /η , log(1 /δηα )) and recovers t ′ such that | t -t ′ | ≤ O ( ησ ) .

Proof. Our proof is based on the following claims:

Claim C.2 (Rough Estimate) . There is an algorithm which, given m = O ((1 /α 4 ) log(1 /δ )) samples of the kind ξ + y + t and ξ + y ′ , where t ∈ R is an unknown translation, returns t ′ r satisfying | t ′ r -t | < O ( σα -1 / 2 ) .

Claim C.3 (Fine Estimate) . Suppose z, z ′ have means bounded from above by Aσ and variances at most σ 2 and suppose α ∈ (0 , 1) and η ∈ (0 , 1 / 2) . Then in poly(( O (1) /αη ) 1 /η , log(1 /δηα )) samples and poly(( O (1) /αη ) 1 /η , log(1 /δηα )) time, it is possible to recover k ∈ [1 /ηα, ( O (1) /ηα ) O (1 /η ) ] such that

1) +

E

]

.

Using Claim 3.2, we first identify a rough estimate t ′ r satisfying | t ′ r -t | < O ( σα -1 / 2 ) . This allows us to re-center y ′ . Let the re-centered distribution be denoted by z ′ = y ′ and z = y + t -t ′ r . Then z and z ′ are such that E [ z ] and E [ z ′ ] are both at most O ( σα -1 / 2 ) in magnitude, and have variance at most σ 2 .

Claim 3.4 then allows us to estimate t ′ f such that | ( E [ z ] -E [ z ′ ]) -t ′ f | = | t -t ′ r -t ′ f | ≤ η O ( σα -1 / 2 ) .

Setting t ′ = t ′ r + t ′ f , we see that our estimate t ′ is now η times closer to t compared to t ′ r .

To refine this estimate further, we can obtain fresh samples and re-center using t ′ instead of t ′ r . Repeating this process O (log 1 /η (1 /α )) = O (log η ( α )) times is sufficient to obtain an estimate that incurs an error of η · η log η ( α 1 / 2 ) · O ( σα -1 / 2 ) ≤ O ( ησ ) .

This results in a runtime and sample complexity that is only O (log 1 /η (1 /α )) times the runtime and sample complexity required by Claim 3.4. This amounts to the final runtime and sample complexity being poly(( O (1) /αη ) 1 /η , log(1 /δηα )) .

We now prove Claim 3.2 and Claim 3.4.

Claim 3.2 shows that the median of the distribution of pairwise differences of ξ + y + t and ξ + y ′ estimates the mean up to an error of σα -1 / 2 .

Proof of Claim 3.2 Let ˜ ξ be a random variable with the same distribution as ξ and independently drawn. We have independent samples from the distributions of ξ + y + t and ξ + y ′ . Applying Fact B.1 to these distributions, we see that if we have at least O (1 /α 4 ) log(1 /δ ) samples from the distribution of ( ξ -˜ ξ ) + ( y -y ′ ) + t , these samples will have a median of t ± O ( σ/ √ α ) .

Proof of Claim 3.4 To identify such a k , the idea is to ensure that E [ ξ + z | | ξ + z | ≤ Aσk ] = E [ ξ + z | | ξ | ≤ Aσk ] ± O ( Aησ ) = E [ ξ | | ξ | ≤ Aσk ] + E [ z ] ± O ( Aησ ) , and similarly for z ′ . The theorem follows by taking the difference of these equations.

Before we proceed, we will need the following definitions: let P ( i, z ) be defined as follows:

This will help us bound the final error terms that arise in the calculation. We will need the following upper bound on P ( i, z ) + P ( i, z ′ ) .

Claim C.4. There exists a function ˜ P : N → R + satisfying:

For all i ∈ N , ˜ P ( i ) ≥ P ( i, z ) + P ( i, z ′ ) which can be computed using samples from ξ + z and ξ + z ′ .

There is a k ∈ [(1 /αη ) , ( C/α +1 / ( αη ) η ) 1 /η ] such that k ˜ P ( k ) < η ∑ k j =0 ˜ P ( k ) .

∑ k j =0 ˜ P ( k ) = O ( Pr [ | ξ + z | ≤ Aσk ] + Pr [ | ξ + z ′ | ≤ Aσk ]) .

With probability 1 -δ , for all i < ( O (1) /α ) 1 /η /η , ˜ P ( i ) can be estimated to an accuracy of O ( η/i ) by using ( O (1) /α ) 2 /η log(1 /ηαδ ) /η 5 samples from ξ + z and ξ + z ′ .

We defer the proof of Claim C.4 to Appendix C.2, and continue with our proof showing that E [ ξ + z | | ξ + z | ≤ Aσk ] ≈ E [ ξ + z | | ξ | ≤ Aσk ] for k satisfying the conclusions of Claim C.4. To this end, observe the following for f ( ξ, z ) being either 1 or ξ + z .

By setting f ( ξ, z ) := 1 and considering the case where i = k satisfies the conclusions of Claim C.4, we can bound the 'error terms"

Pr [ | ξ + z | ≤ Aσk and | ξ | > Aσk ] and Pr [ | ξ + z | > Aσk and | ξ | ≤ Aσk ] in terms of ˜ P ( k ) .

Furthermore, ˜ P ( k ) itself is upper bounded by O ( η/k )( Pr [ | ξ + z | ≤ Aσk ] + Pr [ | ξ + z ′ | ≤ Aσk ]) as per Item 2 and Item 3. Putting these facts together, we have that

A similar claim holds for the distribution over z ′ . An application of the triangle inequality now implies

If | A -B | < τ ( A + B ) it follows that (1 -τ ) / (1 + τ ) < A/B < (1 + τ ) / (1 -τ ) . For τ ∈ (0 , 1 / 2] , this means A = Θ( B ) . Applying this to our case, we can conclude that Pr [ | ξ + z ′ | ≤ Aσk ] = Θ( Pr [ | ξ + z | ≤ Aσk ]) . Substituting this equivalence back into the previous expression, we obtain:

Similarly, when f ( ξ, z ) := ξ + z , we need to control the error terms: E [( ξ + z ) 1 ( | ξ + z | ≤ Aσk ) 1 ( | ξ | > Aσk )] and E [( ξ + z ) 1 ( | ξ + z | > Aσk ) 1 ( | ξ | ≤ Aσk )] .

Observe that ( ξ + z ) 1 ( | ξ + z | ≤ Aσk ) 1 ( | ξ | > Aσk ) has a nonzero value with probability at most E [ 1 ( | ξ + z | ≤ Aσk ) 1 ( | ξ | > Aσk )] < ˜ P ( k ) . Also, the magnitude of ( ξ + z ) in this event is at most Aσk . Putting these together, we get that

Unfortunately, we cannot use the same argument to bound E [( ξ + z ) 1 ( | ξ + z | > Aσk ) 1 ( | ξ | ≤ Aσk )] , since | ξ + z | is no longer bounded by Aσk in this event. However, we can break the sum ξ + z as follows: ξ + z = ξ + z 1 ( | z | > Aσk ) + z 1 ( | z | ≤ Aσk ) . This allows us to get the following bound:

where the third inequality follows by an application of Chebyshev's inequality, and the final inequality follows by choosing k ≥ 1 / ( ηα ) .

Putting everything together, we see

To finally compute the conditional probability, we use Equation (1) and Equation (2) to get

where the second inequality is a consequence of Item 3, and the last is due to the fact that Pr [ | ξ + z | ≤ Aσk ] ≥ α/ 2 whenever k > 2 , which follows from an application of Fact A.3 while noting the fact that Pr [ ξ = 0] ≥ α .

Taking a difference for the above calculations for z and z ′ , we see that,

Consider this final error, and let O ( Aησ ) < CAησ for some constant C . Repeating the above argument initially setting η = η ′ /C , where C is the constant gives us the guarantee we need.

Finally, we estimate the runtime and sample complexity of our algorithm. The main bottleneck in our algorithm is the repeated estimation of ˜ P ( i ) and estimation of E [( ξ + z ) | | ξ + z | ≤ Aσk ] .

According to Item 4, each time we estimate ˜ P ( i ) to the desired accuracy, we draw poly(1 /η, ( O (1) /α ) 1 /η , log(1 /δηα )) samples.

An application of Hoeffding's inequality (Lemma A.1) then allows us to estimate the conditional expectation E [( ξ + z ) | | ξ + z | ≤ Aσk ] to an accuracy of ηAσ by drawing poly(1 /η, ( O (1) /α ) 1 /η , log(1 /δηα )) samples as well. The exponential dependence here comes from the exponential upper bound on k .

C.2 Proof of Claim C.4

In this section, we prove the existence of ˜ P ( · ) which is an upper bound on P ( i, z ) + P ( i, z ′ ) , which we can estimate using samples from ξ + z and ξ + z ′ .

Proof of Claim C.4

Proof of Item 1 :

Recall the definition of P ( i, z ) . P ( i, z ) := Pr [ | ξ | ∈ Aσ ( i -1 , i +1)] + Pr [ | ξ | < Aiσ, | ξ + z | > Aiσ ] + Pr [ | ξ | > Aiσ, | ξ + z | < Aiσ ] .

For Item 1 to hold, we need to define ˜ P ( i ) to be an upper bound on P ( i, z ) + P ( i, z ′ ) which can be computed using samples from ξ + z and ξ + z ′ . To this end, we bound P ( i, z ) as follows. First, note that we can adjust the endpoints of the intervals to get

Then, we partition the ranges in the definition above into intervals of length Aσ to get:

Next, an application of the triangle inequality to | ξ | ∈ Ajσ +[0 , Aσ ) and | ξ + z | > Aiσ implies that | z | ≥ A ( i -j -1) σ . Similarly, the same kind of argument when | ξ | > A ( i + 1) σ and | ξ + z | ∈ Ajσ +[0 , Aσ ) demonstrates that |-z | = | ξ + z -ξ | ≥ A ( i -j ) σ . We then use Fact A.3 to move from | ξ + z | to | ξ | in the third term.

An application of Chebyshev's inequality to z , using the independence of z and ξ , gives that

Another application of Fact A.3 applied to ( ξ + z ) -z then gives us

Finally, extending all intervals so that they match, and observing that ∑ i -2 (1 / ( i -j -1) 2 ) Pr [ | ξ +

We now let ˜ P ( i, z ) denote the final upper bound on P ( i, z ) . The value of having ˜ P ( i, z ) is that it can be computed using samples from ξ + z .

We defined ˜ P ( i ) = ˜ P ( i, z ) + ˜ P ( i, z ′ ) .

Proof of Item 2:

First observe that ∑ ∞ i =0 ˜ P ( i ) < C for some constant C . It is clear that this is true of the first term, since every interval will get over-counted at most 10 times. To see that the second term can be bounded, observe that

The first inequality follows by extending the limits of summation.

The final inequality follows from the fact that the total probability is at most 1 , every interval of size σ gets over-counted at most finitely many times, and the fact that ∑ ∞ k =1 1 /k 2 = O (1) .

Item 2 now follows from the fact that ˜ P ( i ) , i ≥ 0 is a non-negative sequence that sums to a finite quantity, and ˜ P (0) ≥ α/ 2 , since the interval ˜ P (0) upper bounds is contains at least a constant fraction of the mass of ξ at 0 that is moved by z, z ′ , and Pr [ ξ = 0] ≥ α .

Applying Lemma B.2, we get our result.

Proof of Item 3:

Let k be such that Item 2 holds, i.e. k ˜ P ( k ) < η ∑ k j =1 ˜ P ( k ) , then the goal is to show ∑ k j =1 ˜ P ( k ) = O ( Pr [ | ξ + z | < Aσk ] + Pr [ | ξ + z ′ | < Aσk ]) .

We first consider the sum over i , of ˜ P ( i, z ) . It is easy to see that this is

The first term on the RHS is almost what we want. We now show how to bound the second term,

̸

̸

The first inequality holds since any pair of ( i, j ) that has a nonzero term in the first sum will also occur in the second sum, and all terms are non-negative.

The second equality is just pulling the common j term out.

The third inequality follows from the fact that ∑ ∞ i =1 1 /i 2 = O (1) .

The fourth inequality follows from the fact that each σ -length interval is overcounted at most a constant number of times.

This allows us to bound ∑ k i =1 ˜ P ( i, z ) by O ( Pr [ | ξ + z | < Aσ ( k + 5)]) overall. Similarly for ∑ k i =1 ˜ P ( i, z ′ ) , we can obtain a bound of O ( Pr [ | ξ + z ′ | < Aσ ( k +5)]) . Putting these together, we see ∑ k i =1 ˜ P ( i ) ≤ O ( Pr [ | ξ + z | < Aσ ( k +5)] + Pr [ | ξ + z ′ | < Aσ ( k +5)]) . Finally, to get the upper bound claimed in Item 3, observe that

Rearranging the inequality and by scaling η such that that O ( η/k ) ≤ 1 / 2 , we see that Pr [ | ξ + z | < Aσ ( k +5)]+ Pr [ | ξ + z ′ | < Aσ ( k +5)] = O ( Pr [ | ξ + z | < Aσ ( k -4)]+ Pr [ | ξ + z ′ | < Aσ ( k -4)]) = O ( Pr [ | ξ + z | < Aσk ] + Pr [ | ξ + z ′ | < Aσk ]) , completing our proof of Item 3.

Proof of Item 4:

Finally, to see Item 4 holds, observe that 0 < ˜ P ( i ) < O (1) . Let B = (1 /η )( O (1) /α ) 1 /η denote the maximum index before which we can find a k such that k ˜ P ( k ) ≤ η ∑ k i =0 ˜ P ( i ) . Now, To estimate ˜ P ( i ) empirically, we partition the interval ( -BAσ,BAσ ) into B intervals of length Aσ each, and estimate the probability of ξ + z falling in each interval. If we estimate each of these probabilities to an accuracy of η/ (100 B ) , we can estimate ˜ P ( i ) to an accuracy of O ( η/i ) .

An application of Hoeffding's inequality (Lemma A.1) tells us that each estimate will require O ( B 2 /η 2 log(1 /δ )) samples. Taking a union bound over all these intervals, we see that we will require O ( B 2 /η 2 log( B/δ )) samples.

Finally, another union bound over each i ∈ [0 , B ] implies that we will need O ( B 2 /η 2 log( B 2 /δ )) samples. Substituting the value of B back in, we see that this amounts to requiring (1 /η 5 )( O (1) /α ) 2 /η log(1 /ηαδ ) samples.

Estimating ˜ P ( i ) will take time polynomial in the number of samples, and so we take time poly(( O (1) /α ) 1 /η , 1 /η, log(1 /ηαδ )) .

C.3 High-dimensional Noisy Location Estimation

In this section, we explain how to use our one-dimensional location estimation algorithm to get an algorithm for noisy location estimation in d dimensions.

The algorithm performs one-dimensional location estimation coordinate-wise, after a random rotation.

We need to perform such a rotation to ensure that every coordinate has a known variance bound of σ/ √ d .

Algorithm 5 High-dimensional Location Estimation: ShiftHighD ( S , S , η, σ, α )

Lemma C.5 (Location Estimation) . Let D y i for i ∈ { 1 , 2 } be the distributions of ξ + z i for i ∈ { 1 , 2 } where ξ is drawn from D ξ and Pr ξ ∼ D ξ [ ξ = 0] ≥ α and z i ∼ D i are distributions over R d satisfying E D i [ x ] = 0 and E D i [ ∥ x ∥ 2 ] ≤ σ 2 . Let v ∈ R d be an unknown shift, and D y 2 ,v denote the distribution of y 2 shifted by v . There is an algorithm (Algorithm 5), which draws poly(1 /η, ( O (1) /α ) 1 /η , log(1 /δηα )) samples each from D y 1 and D y 2 ,v runs in time poly( d, 1 /η, ( O (1) /α ) 1 /η , log(1 /δηα )) and returns v ′ satisfying ∥ v ′ -v ∥ ≤ O ( ησ ) with probability 1 -δ .

Proof. Consider a matrix R whose entries R i,j are independently drawn from the uniform distribution over ± 1 / √ d . and whose diagonals are 1 / √ d .

Our goal is to show that with probability at least 99% , the standard deviation of each coordinate of Rz is bounded by O ( σ/ √ d ) , i.e., the standard deviation of Rz · e i is at most O ( σ/ √ d ) for all integer i in [ d ] .

We can then amplify this probability to ensure that the algorithm fails with a probability that is exponentially small.

To see this, observe that Rz · e i = r i · z , and so E z [ r i · z ] = 0 .

̸

We now bound the second term with probability 99% via applying Chebyshev's inequality. Observe that E [ z p z q ] ≤ √ E [ z 2 p ] E [ z 2 q ] ≤ σ 2 . Since R i,p and R i,q are drawn independently and p = q , we see that the variables R i,p R i,q and R i,l R i,m pairwise independent for pairs ( p, q ) = ( l, m ) , this implies Pr [ | ∑ p,q ∈ [ d ] ,p<q R i,p R i,q E [ z p z q ] | > T ] ≤ O ( σ 4 ) d d 2 T 2 . By choosing T = O ( σ 2 /d ) , we see that the right-hand side above is at most 0 . 001 /d .

̸

A union bound over all the coordinates then tells us that with probability 99% , the variance of each coordinate is at most O ( σ 2 /d ) .

√

Then, for each coordinate i , we can identify v ′ i = v i ± O ( ησ/ d ) through an application of Lemma 3.1. Putting these together with probability at least 99% , we find v ′ satisfying ∥ v ′ -v ∥ 2 ≤ O ( η 2 σ 2 ) .

Changing between these basis representations maintains the quality of our estimate since the new basis contains unit vectors nearly orthogonal to each other. With high probability, the inner products between these are around O (1 / √ d ) for every pairwise comparison, so R approximates a random rotation.

Probability Amplification: The current guarantee ensures that we obtain a good candidate with a constant probability of success. However, for the final algorithmic guarantee, we need a higher probability of success. To achieve this, we modify the algorithm as follows:

Run the algorithm T times, each time returning a candidate v ′ i that is, with probability 99% , within O ( ησ ) distance from the true solution.

Construct a list of candidates C = { v ′ 1 , . . . , v ′ T } .

Identify a ball of radius O ( ση ) centered at one of the v ′ i that contains at least 90% of the remaining points.

Return the corresponding v ′ i as the final output.

If no such v ′ i exists, return any vector from C .

Let E denote the event that a point is within O ( ησ ) to the true solution.

This will succeed with probability 1 -exp( -T ) . To see why, observe the chance that we recover (2 / 3) T vectors outside the event E is less than (0 . 01) 2 / 3 T ( T 2 / 3 T ) < (0 . 047) T ( T T/ 2 ) < (0 . 047) T (2 T / √ T ) < (0 . 095) T .

D List-Decodable Mean Estimation

This section presents an algorithm for list-decodable mean estimation when the inlier distribution follows D σ . Here, D σ represents a set of distributions over R d defined as D σ := { D | E D [ | x -E D [ x ] | 2 ] ≤ σ 2 } . In our setting, we receive samples from ξ + z , where Pr [ ξ = 0] > α , where α can be close to 0 . Our objective is to estimate the mean with a high degree of precision.

Note that the guarantees provided by prior work do not directly apply to our setting. Prior work examines a more aggressive setting where arbitrary outliers are drawn with a probability of 1 -α . These outliers might not have the additive structure we have.

Recall the definition of an ( α, β, s ) -LDME algorithm:

Definition D.1 (Algorithm for List-Decodable Mean Estimation) . Algorithm A is an ( α, β, s ) -LDME algorithm for D (a set of candidate inlier distributions) if with probability 1 -δ A , it returns a list L of size s such that min ˆ µ ∈L ∥ ˆ µ -E x ∼ D [ x ] ∥ ≤ β for D ∈ D when given m A samples of the kind z + ξ for z ∼ D and Pr [ ξ = 0] ≥ α . If 1 -α is a sufficiently small constant less than 1 / 2 , then s = 1 .

We now prove Fact 2.1 which we restate below for convenience.

Fact D.2 (List-decoding algorithm) . There is an ( α, ησ, ˜ O ((1 /α ) 2 /η 2 )) -LDME algorithm for the inlier distribution belonging to D σ which runs in time ˜ O ( d (1 /α ) 2 /η 2 ) and succeeds with probability 1 -δ . Conversely, any algorithm which returns a list, one of which makes an error of at most O ( ησ ) in ℓ 2 norm to the true mean, must have a list whose size grows exponentially in 1 /η .

If 1 -α is a sufficiently small constant less than half, then the list size is 1 to get an error of O ( √ 1 -α σ ) .

Proof. Algorithm: Consider the following algorithm:

If α < c and η > √ α : Run any stability-based robust mean estimation algorithm from Diakonikolas et al. [2020b] and return a singleton list containing the output of the algorithm.

Otherwise, for integer each i ∈ [1 , 100(1 /α ) 2 /η 2 log(1 /δ ) 2 ] sample 1 /η 2 samples and let their mean be µ i .

Return the list L = { µ i | i ∈ [1 , 100(1 /α ) 2 /η 2 log(1 /δ ) 2 ] } .

If the algorithm returns in the first step, then the guarantees follow from the guarantees of the algorithm for robust mean estimation from Diakonikolas et al. [2020b] (Proposition 1.5 on page 4).

Otherwise, observe that the probability that every one of 1 /η 2 samples drawn is an inlier, is α 1 /η 2 .

Hence, with probability 1 -δ we see that if we draw 1 /η 2 samples O ((1 /α ) 2 /η 2 log(1 /δ ) 2 ) times, there are at least O (log(1 /δ )) sets of samples containing only inliers. Then, the mean of one of these concentrates to an error of O ( ησ ) by an application of Lemma A.2. More precisely, Lemma A.2 ensures that with probability 99% , the mean of a set of 1 /η 2 inliers concentrates up to an error of O ( ησ ) . Repeating this log(1 /δ ) times, we get our result.

Hardness: To see that the list size must be at least exp(1 /η ) , consider the set of inlier distributions given by { D s | s ∈ {± 1 } d } where each D s is defined as follows: D s is a distribution over R d such that each coordinate independently takes the value s i with probability 1 /d , and 0 otherwise.

Each D s defined above belongs to D √ 1 -1 /d since E x ∼ D s [ x ] = s/d and

We will set the oblivious noise distribution for each D s to be -D s . Our objective is to demonstrate that the distribution of D s -D s is the same for all s and is independent of s . This means that we cannot identify s by seeing samples from D s -D s .

Then, since the means of D s and D ′ s for any distinct pair s, s ′ ∈ {± 1 } d differ by at least 1 /d , if we set d = 1 /η we see that there are 2 1 /η possible different values of the original mean, each pair being at least η far apart, which is larger than ησ 2 = η (1 -η ) .

We can assume, without loss of generality, that s = 1 , where 1 represents the all-ones vector. Each coordinate of D s can be viewed as a random coin flip, taking the value 0 with probability 1 -1 /d and 1 with probability 1 /d .

The probability of obtaining the all-zeros vector is given by (1 -1 /d ) d , which approaches a constant value for sufficiently large d , and so, Pr x ∼ D s [ x = 0] ≥ 0 . 001 , i.e., the α for the oblivious noise is at least a constant. In fact, it can be as large as 1 /e > 0 . 35 for large enough d .

Let the oblivious noise be -D . Now, consider the distribution of x + y , where x follows the distribution D and y follows the distribution -D . If we focus on the first coordinate, x 1 + y 1 , we observe that it follows a symmetric distribution over {-1 , 0 , 1 } which does not depend on s 1 . Also, each coordinate exhibits the same distribution, and they are drawn independently of one another. Hence, the final distribution is independent of s , so we get our result.

E Proof of Corollary 4.2

Below, we restate Corollary 4.2 for convenience.

Corollary E.1. Given access to oblivious noise oracle O α,σ,f , a ( O ( ησ ) , ϵ ) -inexact-learner A G running in time T G , there exists an algorithm that takes poly((1 /α ) 1 /η 2 , ( O (1) /η ) 1 /η , log( T G /δηα )) samples, runs in time T G · poly( d, (1 /α ) 1 /η 2 , ( O (1) /η ) 1 /η , log(1 /ηαδ )) , and with probability 1 -δ returns a list L of size ˜ O ((1 /α ) 1 /η 2 ) such that min x ∈L ∥∇ f ( x ) ∥ ≤ O ( ησ ) + ϵ . Additionally, the exponential dependence on 1 /η in the list size is necessary.

Proof. This follows by substituting the guarantees of Fact 2.1 for the algorithm A ME

F Varying Noise Assumptions

In this section, we demonstrate that changing some distributional assumptions of the oblivious noise ξ can yield better guarantees or simpler algorithms.

F.1 Stronger Concentration on ξ Yields Polynomial Dependence on 1 /η

We show that if the oblivious noise has bounded tails we can avoid the exponential dependence on 1 /η , which we have shown in Appendix D to be necessary under our standard assumptions.

For the sake of demonstration, we will assume the standard assumptions of heavy-tailed stochastic optimization where the mean of ξ is equal to zero and the variance of ξ is upper bounded by σ , as is done in Gorbunov et al. [2020]. In this specific setting, our algorithm remains the same but our analysis becomes tighter in Item 2 of Claim C.4. Our original analysis yields an exponential dependence on 1 /η due to the upper bound on k , yet with concentration on the noise, we can show that k = poly(1 /α, 1 /η ) .

Due to our specific construction of ˜ P ( k ) in the proof of Claim C.4 of Item 1, we have that k ˜ P ( k ) ≤ O ( k -1 / 3 ) and α/ 2 ≤ ˜ P (1) ≤ ∑ k j =1 ˜ P ( j ) . Thus, when k ≥ O (1 / ( αη ) 3 ) , the inequality of Item 2 is satisfied. The fact that α/ 2 ≤ ˜ P (1) is given, and thus, what remains is to show ˜ P ( k ) ≤ k -4 / 3 , which follows by splitting the summation in ˜ P ( i, z ) over j = 1 , ..., k -1 into j ∈ [1 , k 2 / 3 ] and j ∈ ( k 2 / 3 , k -1] . The sum over [1 , k 2 / 3 ] is upper bounded by O ( k -4 / 3 ) and the sum over ( k 2 / 3 , k -1] is upper bounded by O ( A -2 k -4 / 3 ) = O ( αk -4 / 3 ) using Chebyshev's inequality. Therefore, ˜ P ( k ) ≤ k -4 / 3 and k = poly(1 /α, 1 /η ) in Claim 3.3.

Ultimately, it would be an interesting direction to identify conditions necessary and sufficient for a polynomial dependence on 1 /η .

F.2 Small Fraction of Adversarial Corruptions

If, on the other hand, at each point you have access to gradients v ( x ) such that with probability 1 -ϵ you see ∇ f ( x, γ ) , which concentrates around ∇ f ( x, γ ) and with probability ϵ you see an arbitrary point, then one can use robust mean estimation instead of our location estimation algorithm and recover guarentees which are correct upto an error of O ( √ ϵ ) .

In this case, there is no need for list-decoding or robust location estimation.

in Theorem 1.4.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper initiates the study of stochastic optimization with oblivious noise that might be biased and have unbounded variance, which is a generalization of the heavy-tailed noise setup. The key assumption regarding the oblivious noise is that it assumes a value of $0$ with a probability within the range of $0 \\leq \\alpha \\leq 1$, which can be interpreted as the fraction of inliers. Notably, when $\\alpha \\leq 1/2$, it is proven to be information-theoretically impossible to find an approximate stationary point of the function. To address this challenge, the authors incorporate the concept of list-decoding into the framework of stochastic optimization, and focus instead on identifying a list of points where at least one of them is an approximate stationary point.\n\nTechnically, this paper presents an equivalence between list-decodable stochastic optimization with oblivious noise and list-decodable mean estimation problem leveraging a technique known as noisy location estimation. The analysis of list-decodable stochastic optimization with oblivious noise is conducted by examining the list-decodable mean estimation problem.\n\nSTRENGTHS:\nThis paper investigates an important setting of stochastic optimization introduces a fresh perspective by introducing the concept of list-decodable stochastic optimization. The definition of this new framework is not only intuitive and well-motivated by practical problems but also exhibits elegance from a theoretical standpoint, given how weak the assumptions on the noise model are. The algorithms presented in the paper, along with their corresponding proofs, are intricate and highly nontrivial from a technical standpoint. Nevertheless, the authors have succeeded in presenting the analyses in a well-organized manner, ensuring that they are generally not hard to follow.\n\nWEAKNESSES:\n1. As pointed out in the paper, an exponential dependence on $1/\\eta$ is necessary in the list-size, which can mildly impact the overall appeal of the results. This dependency may introduce some considerations regarding scalability and practicality.\n\n2. The framework presented in this paper is inherently abstract, and there is a lack of clarity concerning the algorithm's performance in concrete examples, including scenarios with more specific theoretical settings that incorporate additional assumptions, as well as practical problem domains.\n\nQUESTIONS:\nCorrespondingly, I have the following questions that could possibly make the results even stronger if addressed:\n1. Can the exponential dependence on $\\eta$ be mitigated by making slight adjustments to the original definition of list-decodable stochastic optimization? For instance, are there additional assumptions that can be incorporated or specific parameter regimes that can be adjusted to reduce this exponential dependency?\n\n2. Are there more concrete applications of list-decodable stochastic optimization methods? \n\n3. Minor comment: I saw the term \"convex\" in the caption of Algorithm 2. I assume this is a typo and convexity is not needed in the proof, right?\n\nLIMITATIONS:\nNot relevant in my opinion.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an algorithm for first-order stochastic optimization where the algorithm has access to an oracle that returns a noisy version of the gradient of the objective function. The considered noise model includes two components: A bounded-variance observation noise (which is the typical well-studied type of noise), and oblivious outliers noise $\\xi$ satisfying $Pr[\\xi = 0] >= \\alpha$. Furthermore, the distribution of the oblivious noise \\xi does not need to be symmetric.\n\nIt is shown that if the fraction of inliers is below 1/2, it is information-theoretically impossible to give a unique solution. This is why the authors consider a list-decodable learner where the learner returns a list of solutions, one of which is guaranteed to be good. The authors show that if the fraction of inliers is sufficiently close to 1, then the algorithm can recover a single solution.\n\nSTRENGTHS:\nDesigning learning algorithms which are robust against adversarial or semi-adversarial type of noise is very important. The setup that is considered in this paper is original (as far as I can tell).\n\nWEAKNESSES:\nI found the paper to be generally not very well written. The notation is a bit confusing in several places (e.g., check the question regarding line 203 below), and the writing style can be sometimes too informal.\n\nOne thing that I found crucially missing is the clear and formal statement of the problem and the clear statements of the assumptions. For example, what are the properties of the function $f(\\gamma,x)$? The only property that I found is that $f(x) = E_{\\gamma}[f(\\gamma,x)]$ must be $L$-smooth. However, this is clearly not enough to even guarantee the existence of a stationary point. For example, consider $x\\in \\mathbb{R}$ (i.e., one dimension) and define $f(\\gamma,x) = x$. In this case, we have $f(x) = x$ and hence $\\nabla f(x) = 1$ fo all $x$ and there is no stationary point.\n\nTypos:\n- Page 4, line 175: \"we can a generate list\" -> \"we can generate a list\"\n\nQUESTIONS:\n- What are the properties of the function $f(\\gamma,x)$ which are needed for the main result to hold?\n- Page 4, line 203: Is $\\xi$ in $\\xi + y' + t$ the same as the $\\xi$ in $\\xi + y$, or is it an independent instance? It seems from the following discussion that the authors consider an independent instance. If this is the case, please write $\\xi' + y' + t$.\n- Page 7, line 309: What is $L$? Is it the same as the $L$ of Section 2? But in Section 3 we don't have a parameter $L$ for location estimation.\n- There doesn't seem to be a proof for Claim 3.3 (even in the appendices).\n\nLIMITATIONS:\nNo concerns regarding potential societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of stochastic optimization with oblivious noise. Here, one receives noisy gradients of a non-convex function $f(x) = \\mathbb{E} [f(x, \\gamma)]$ (where $\\gamma$ is bounded variance observational noise) and the noisy gradient samples are generated as follows $\\nabla_x f(x, \\gamma) + \\xi$ where $\\xi$ is generated independently of $\\gamma$ and $x$ with the only restriction that $\\mathbb{P} (\\xi = 0) \\geq \\alpha$. The paper specifically focuses on the setting where $\\alpha \\ll 1 / 2$. Under such mild restrictions on $\\xi$, it is impossible to recover a single point which is guaranteed to be near-stationary. However, in line with recent results on list-decodable robust estimation, the paper shows that one can recover a list of estimates one of which is approximately stationary. \n\nTechnically, the paper builds on two recent results on robust estimation. The first is SEVER which is a robust stochastic estimation algorithm focusing on the setting when $\\alpha \\to 1$. In this setting, it is possible to leverage recent robust estimation algorithms to clean the observed gradients and recover a good approximation to the true gradient (up to the degree determined by $1 - \\alpha$) and then utilize this approximate gradient to find a stationary point. The second is a recent line of work on list-decodable mean estimation. Here, one receives corrupted samples from a high-dimensional distribution where $\\alpha$ fraction of points are from the true distribution and the goal is to estimate its mean. While producing a single estimate is impossible, these algorithms return a list of size $1 / \\alpha$, one of which is guaranteed to be accurate. In this paper, the authors essentially extend the SEVER framework to the list-decodable setting. However, this requires some novel technical contributions. A naive implementation would lead to exponential growth in the number of estimates (a list of size $l$ would have $l / \\alpha$ many elements in the next iteration if each of its elements were queried and updated with the $1 / \\alpha$ resulting gradients). Instead the authors introduce a novel technical tool that they term location estimation which when given samples from $z + \\xi$ and $z' + t + \\xi$ for some unknown $t$ (and $z$ and $z'$ have bounded covariance) can estimate $t$. With this tool, the algorithm starts by first generating $1/\\alpha$ gradients at $0$ and initializing $1 / \\alpha$ candidates each corresponding to one of the estimates. Then, each element of the list, $x$, is queried to produce gradient estimates. The location estimate procedure is then run on gradient estimates from $x$ and $0$ to essentially estimate $\\nabla f(x) - \\nabla f(0)$. Finally, from this each candidate is updated by estimating $\\nabla f(x)$ using the particular estimate of $\\nabla f(0)$ it corresponds to.\n\nOverall, this is a really nice paper studying an interesting problem. My one concern is in the assumptions made in the paper. For instance, the assumptions don't capture the canonical estimation problem of list-decodable mean estimation. Here, one assumes that the true distribution has covariance bounded in spectral norm whereas this paper essentially assumes a bound on the expected squared length of a data point which could be larger by a factor of $d$. It would be interesting to see if these results could be extended to setting with weaker assumptions. Can we obtain similar results with $\\mathbb{E} [(\\nabla f(x, \\gamma) - \\nabla f(x)) (\\nabla f(x, \\gamma) - \\nabla f(x))^\\top] \\prec \\sigma^2 I$ as opposed to the stronger assumption in this paper of $\\mathbb{E} [\\|\\nabla f(x, \\gamma) - \\nabla f(x)\\|^2] \\prec \\sigma^2$ used here.\n\nSTRENGTHS:\nSee main review\n\nWEAKNESSES:\nSee main review\n\nQUESTIONS:\nSee main review\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies robust first order optimization in a challenging setting where noise may be unbounded, a setting that arises often for real world optimization problems. Because this problem is intractable in general, one needs to make plausible assumptions on the noise, that are realistic on one hand but allow for efficient analysis.\n\nThe noise model proposed here allows for noise to be unbounded, and introduces two simple constraints:\n\n1. The unbounded noise when computing a gradient is *oblivious* in the following sense: there are two noise components, one that is well-behaved (zero mean and bounded variation), and one that is unbounded but oblivious/indpendent of both the location in which gradient is computed and the value of the well-behaved part of the noise.</li>\n\n2. We assume the unbounded noise has probability bounded away from 0 to be equal to zero (i.e., to not exist at all).\n\nIt turns out that these two relatively weak conditions allow for efficient robust first order optimization. Specifically, these conditions allow for list-decodable robust optimization, where the goal is to output a list of candidate outputs where at least one should be a good approximation of the correct optimization outcome. The main technical result shows how to solve this problem by reducing it to list-decodable mean estimation, a problem that enjoyed substantial progress in recent years. The authors also show a reduction in the opposite direction. A substantial component in the technical analysis is a procedure that the authors develop for location estimation in an appropriate noisy setting.\n\nSTRENGTHS:\n1. Interesting and important goal, of better understanding the beyond worst case landscape for (first order) optimization.\n\n2. Writing is very clear and relatively easy to follow for me (a non-expert outsider). \n\n3. The assumptions required for the analysis are weak and seemingly realistic.\n\nWEAKNESSES:\n1. The technical novelty is perhaps somewhat limited, the work relies heavily on reductions to existing results in robust mean estimation.\n\nQUESTIONS:\nComment: my review is a low-confidence one (as a non expert in the field) and I may have missed central points in the paper, so may update the score after subsequent reviewers and authors discussions.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new setup for stochastic optimization, where in addition to random observation noise, the stochastic gradient may be subject to independent oblivious noise. This noise might not have bounded moments and isn't necessarily centered. The authors propose a Noisy Location Estimation approach that estimates the gradient difference between two points, specifically \\nabla f(x_t)-\\nabla f(x_0). As such, they maintain robust estimations of the gradient at all points {x_t} as long as there is a reliable estimation of \\nabla f(x_0).\n\nSTRENGTHS:\nThe new setup for oblivious noise introduced in the work is plausible, and the authors effectively discuss its relation to existing research. The Noisy Location Estimation proposed by the authors provides an innovative way to estimate gradient differences accurately, reducing the stochastic optimization problem to a mean estimation problem, which seems simpler in the setting. Also, as shown by the authors, the reverse of the reduction holds by simple arguments.\n\nWEAKNESSES:\nThe paper's presentation, particularly in the technical sections, lacks clarity.\n\n1. Definitions should be more precise and self-contained. For instance, the work seems to require that oblivious noise be independent of the noisy gradient, but Definition 1.1 doesn't explicitly state this. In Definition 1.3, phrases like \"sufficiently small constant\" are too vague.\n2. The methodology for mean estimation (Fact 2.1), isn't discussed in the main body. A brief discussion may be helpful.\n3. The \"Rejection Sampling\" discussion on page 5 is difficult to follow and potentially misleading. From my understanding, the core intuition is to identify a large enough domain of size $i$, such that $i$ times the conditional expectation is robust and stable upon shifting the domain.\n\nQUESTIONS:\n1. In Line 227, it appears that [i- 4 · 12, i + 4 · 12) almost fully contains [i - 4 · 12, -i + 4 · 12)]. If that's the case, why is there a need for a \\cup operation?\n2. In Section 5, why is the exponential dependence on 1/\\eta for the size of the list unavoidable?\n3. Is it a requirement for the Noisy Location Estimation that alpha > 0? I didn't delve into the proof details, but it seems that if you're considering the conditional expectation, it might not require alpha > 0. Can you clarify this?\n\nLIMITATIONS:\nAs discussed in Weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}