SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rejection Sampling. Suppose we have an initial rough estimate of  $t$ , given by  $t_r'$  such that  $|t_r' - t| < A\sigma$ . We can then re-center the distributions to get  $\xi + z$  and  $\xi + z'$ , where  $z$  and  $z'$  have means of magnitude at most  $A\sigma$ , and have variance that is  $O(\sigma)$ . Claim 3.4 then shows that we can refine our estimate of  $t$ . It does so by identifying an interval  $I = [-k\sigma, k\sigma]$  around 0 such that the addition of either  $z$  or  $z'$  to  $\xi$  does not significantly alter the distribution's mass within or outside of  $I$ . We show that since  $z, z'$ , and  $\xi$  are independent, the  $I$  that we choose satisfies  $\mathbf{E}[\xi + z' \mid \xi + z' \in I] - \mathbf{E}[\xi + z \mid \xi + z \in I] = \mathbf{E}[\xi + z' \mid \xi \in I] - \mathbf{E}[\xi + z \mid \xi \in I] \pm \eta A = \mathbf{E}[z'] - \mathbf{E}[z] \pm \eta A$  for some  $\eta < 1$ .
To see that such an interval  $I$  exists, we will show that for some  $i$ , there are pairs of intervals  $(A\sigma i, (i + 1)A\sigma]$  and  $[-(i + 1)A\sigma, -iA\sigma)$  which contain a negligible mass with respect to  $\xi$ . Since  $z$  and  $z'$  can move mass by at most  $(A + 1)\sigma$  with high probability, it is sufficient to condition on the interval around 0 which is contained in this pair of intervals. Since we do not have access to  $\xi$ , we instead search for pairs of intervals of length  $O(A\sigma)$  which have negligible mass with respect to both  $\xi + z$  and  $\xi + z'$ , which suffices.
To do this, we will demonstrate an upper bound  $\tilde{P}(i)$  on the mass crossing the  $i^{th}$  pair of intervals described above. This will satisfy  $\sum_{i} \tilde{P}(i) = C'$  for some constant  $C'$ .
To show that this implies there is an interval of negligible mass, we aim to demonstrate the existence of a  $k \in \mathbb{N}$  such that  $k\tilde{P}(k) \leq \eta$ . We will do this through a contradiction. Suppose this is not the case for all  $i \in [0,k]$ , then  $\sum_{i=0}^{k} \tilde{P}(i) \geq \eta \sum_{i=0}^{k} (1/i)$ . If  $k \geq \exp(10C' / \eta)$ , we arrive at a contradiction because the right-hand side is at least  $\eta \sum_{i=0}^{k} (1/i) \geq \eta \log(\exp(10C / \eta)) > 10C'$ , while the left-hand side is bounded above by  $C$ .
Claim C.4 demonstrates, via a more involved and finer analysis, that for the bounded-variance setting where intervals very far away can contribute to mass crossing  $A\sigma i$ , there exists an  $k$  such that  $k\tilde{P}(k) \leq \eta \operatorname*{Pr}[|\xi| < A\sigma k]$ , and that taking the conditional mean restricted to the interval  $[-kA\sigma, kA\sigma]$  allows us to improve our estimate of  $t$ . Here  $\tilde{P}(k)$  denotes an upper bound on the total probability mass that crosses intervals described above.
Extension to Higher Dimensions. In order to extend the algorithm to higher dimensions, we apply the one-dimensional algorithm coordinate-wise, but in a randomly chosen coordinate-basis. Lemma 3.5 uses the fact that representing the distributions in a randomly rotated basis ensures that, with high probability, the inlier distribution will project down to each coordinate with a variance of  $O(\sigma \sqrt{\log(d)} / \sqrt{d})$  to extend Lemma 3.1 to higher dimensions.
Mean Estimation reduces to Stochastic Optimization. In Theorem 1.6 we show that the problem of list-decodable mean estimation can be solved by using list-decodable stochastic optimization for the oblivious noise setting. This establishes the opposite direction of the reduction to show the equivalence of list-decodable mean estimation and list-decodable stochastic optimization.
The reduction uses samples from the list-decodable mean estimation problem to simulate responses from the oblivious oracle to queries at  $x$ . Let the mean of the inlier distribution be  $\mu$ . If the first-order stochastic learner queries  $x$ , we return  $x + s$  where  $s$  is a sample drawn from the list-decodable mean-estimation problem. These correspond to possible responses to the queries when  $f(x) = (1/2)\|x + \mu\|^2$ , where  $\mu$  is the true mean. The first-order stochastic learner learns a  $\widehat{\mu}$  from a list  $\mathcal{L}$  such that  $\| \nabla f(\widehat{\mu}) \| = \| \widehat{\mu} + \mu \| \leq \epsilon$ ; then the final guarantee of the list-decodable mean estimator follows by returning  $-\mathcal{L}$  which contains  $-\widehat{\mu}$ .
Basic Notation For a random variable  $X$ , we use  $\mathbf{E}[X]$  for its expectation and  $\mathbf{Pr}[X \in E]$  for the probability of the random variable belonging to the set  $E$ . We use  $\mathcal{N}(\mu, \sigma^2)$  to denote the Gaussian distribution with mean  $\mu$  and variance matrix  $\sigma^2$ . When  $D$  is a distribution, we use  $X \sim D$  to denote that the random variable  $X$  is distributed according to  $D$ . When  $S$  is a set, we let  $\mathbf{E}_{X \sim S}[\cdot]$  denote the expectation under the uniform distribution over  $S$ . When clear from context, we denote the empirical expectation and probability by  $\widehat{\mathbf{E}}$  and  $\widehat{\mathbf{Pr}}$ . We denote  $\|\cdot\|$  as the  $\ell_2$ -norm and assume  $f: \mathbb{R}^d \to \mathbb{R}$  is differentiable and  $L$ -smooth, i.e.,  $\| \nabla f(x) - \nabla f(x') \| \leq L \| x - x' \|$  for all  $x, x' \in \mathbb{R}^d$ .
$\xi$  will always denote the oblivious noise drawn from a distribution  $Q$  satisfying  $\mathbf{Pr}_{\xi \sim Q}[\xi = 0] \geq \alpha$ .  $y, y'$  will be used to denote mean-zero and variance at-most  $\sigma^2$  random variables. Also define  $e(\gamma, x) \coloneqq f(\gamma, x) - f(x)$  and the interval  $\sigma(a, b] \coloneqq (\sigma a, \sigma b]$ .
Facts We use these algorithmic facts in the following sections. We use a list-decodable robust mean estimation subroutine in a black-box manner. The proof for list-decodable mean estimation can be found in Appendix D, while such algorithms can be found in prior work, see, e.g., Charikar et al. [2017], Diakonikolas et al. [2020a]. We also define a  $(\beta, \epsilon)$ -inexact-learner for  $f$ .
Fact 2.1 (List-decoding algorithm). There is an  $(\alpha, \sigma \eta, \tilde{O}((1 / \alpha)^{1 / \eta^2}))$ -LDME algorithm for the an inlier distribution belonging to  $\mathcal{D}_{\sigma}$  which runs in time  $\tilde{O}(d(1 / \alpha)^{1 / \eta^2})$  and succeeds with probability  $1 - \delta$ . Conversely, any algorithm which returns a list, one of which makes an error of at most  $O(\eta \sigma)$  in  $\ell_2$  norm to the true mean, must have a list whose size grows exponentially in  $1 / \eta$ .
If  $1 - \alpha$  is a sufficiently small constant less than half, then the list size is 1 to get an error of  $O(\sqrt{1 - \alpha}\sigma)$ .
We now define the notion of a robust-learner, which is an algorithm which, given access to approximate gradients is able to recover a point at which the gradient norm is small.
Definition 2.2 (Robust Inexact Learner). Let  $f$  be a  $L_{s}$ -smooth function with a global minimum and  $\mathcal{O}_{\beta, f}^{\mathrm{grad}}(\cdot)$  be an oracle which when queried on  $x$  returns a vector  $g_{x}$  satisfying  $\| g_{x} - \nabla f(x) \| \leq \beta$ . We say that an algorithm  $\mathcal{A}_{G}$  is an  $(\beta, \epsilon)$ -inexact-learner for  $f$  if, given access to  $\mathcal{O}_{\beta, f}^{\mathrm{grad}}$ , the algorithm  $\mathcal{A}_{G}$  recovers  $\hat{x}$  satisfying  $\| \nabla f(\hat{x}) \| \leq \beta + \epsilon$ . We will assume  $\mathcal{A}_{G}$  runs in time  $T_{G}$ .
Several algorithms for convex optimization amount to there being an existence of a robust learner in the convex setting. More recently, for smooth nonconvex optimization, the convergence result for SGD under inexact gradients due to Ajalloeian and Stich [2020] doubles as a  $(\beta ,\epsilon)$ -inexact-learner running in  $T_{G} = O(LF / (\beta +\epsilon)^{2})$  iterations, where  $F = f(x_0) - \min_xf(x)$  and  $x_0$  is the initial point. This follows by an application of Theorem 4 from Ajalloeian and Stich [2020] and by taking the point in the set of iterates that minimizes the gradient norm.
3 Location Estimation
To estimate how the gradient changes between iterations, we will need to estimate the shift between a distribution and its translation. In this section, we give an algorithm which, given access to samples from a distribution and its noisy translation returns an estimate of  $t$  accurate up to an error of  $O(\eta \sigma)$  in  $\ell_2$ -norm. We show a proof sketch here, and a more detailed proof of all the claims involved in Appendix C.
Lemma 3.1 (One-dimensional location-estimation). There is an algorithm (Algorithm 1) which, for  $m = (1 / \eta^5)(O(1) / \alpha)^{2 / \eta}\log (1 / \eta \alpha \delta)$ , given samples  $\{\xi_i + y_i + t\}_{i = 1}^m$  and  $\{\xi_i + y_i'\}_{i = 1}^m$  where  $\xi_{i}$  and  $\tilde{\xi}_i$  are both drawn from  $D_{\xi}$ ,  $y_{i}$  and  $y_{i}'$  are drawn from distinct distributions with mean-zero with variance bounded above by  $\sigma$  and  $t\in \mathbb{R}$  is an unknown translation, runs in time  $\tilde{\mathrm{poly}} (1 / \eta ,(O(1) / \alpha)^{1 / \eta},\log (1 / \delta \eta \alpha))$  and recovers  $t^\prime$  such that  $|t - t^{\prime}|\leq O(\eta \sigma)$ .
Proof. We first identify  $t$  up to an error of  $O(\sigma / \sqrt{\alpha})$  with the following claim.
Claim 3.2 (Rough Estimate). There is an algorithm which, for  $m = O((1 / \alpha^4)\log (1 / \delta))$  given samples  $\{\xi_i + y_i + t\}_{i = 1}^{m}$  and  $\{\tilde{\xi}_i + y_i'\}_{i = 1}^m$  where  $\xi_{i}$  and  $\tilde{\xi}_i$  are both drawn from  $D_{\xi}$ ,  $y_{i}$  and  $y_{i}'$  are drawn from distinct distributions with mean-zero with variance bounded above by  $\sigma$  and  $t\in \mathbb{R}$  is an unknown translation, returns  $|t_r^\prime -t|\leq O(\sigma \alpha^{-1 / 2})$ .
We use  $t_r'$  to center the two distributions up to an error of  $\sigma \alpha^{-1/2}$ . Let the centered distributions be given by  $\xi + z$  and  $\xi + z'$  where  $z, z'$  are independent, have means that are at most  $\sigma \alpha^{-1/2}$  in magnitude and have variance that is bounded by  $4\sigma^2$ .
Let \(A\Delta B\) denote the symmetric difference of the two events \(A, B\). To improve our estimate further, we restrict our attention to an interval \(I = \sigma(-i,i)\) such that \(\mathbf{Pr}[(\xi + z \in I) \Delta (\xi \in I)] <
Algorithm 1 One-dimensional Location Estimation: Shift1D(S1,S2,η,σ,α)
Input: Sample sets  $S_{1}, S_{2} \subset \mathbb{R}^{d}$  of size  $m, \alpha, \eta \in (0,1), \sigma > 0$
1. Let  $T = O(\log_{1/\eta}(1/\alpha))$ . For  $j \in \{1,2\}$ , partition  $S_j$  into  $T$  equal pieces,  $S_j^{(i)}$  for  $i \in [T]$ .
2.  $D = \{a - b\mid a\in S_1^{(1)},b\in S_2^{(1)}\}$
3.  $t^{\prime}(1)\coloneqq \mathrm{median}(D)$
4. Set  $A = O(1 / \sqrt{\alpha})$
5. Repeat steps 6 to 12, for  $i$  going from 2 to  $T$ :
6.  $S_1^{(i)}\coloneqq S_1^{(i)} - t_r'(i - 1)$
7. For  $j \in \{1, 2\}$
$$
\begin{array}{l} \hat {P} _ {j} (i) := O (1) \underset {x \sim S _ {j} ^ {(i)}} {\mathbf {P r}} [ | x | \in A \sigma (i - 5, i + 5) ] \\ + O (1) \sum_ {j = 1} ^ {i - 1} \left(1 / (i - j) ^ {2}\right) \underset {x \sim S _ {j} ^ {(i)}} {\mathbf {P r}} [ | x | \in A j \sigma + A \sigma [ - 4, 5). ] \\ \end{array}
$$
8. Let  $\hat{P}(i) = \hat{P}_1(i) + \hat{P}_2(i)$ .
9. Identify an integer  $k \in [(1 / \alpha \eta), (C / \alpha + 1 / (\alpha \eta)^{\eta})^{1 / \eta}]$  such that
$$
\hat{P} (k)\leq \eta \sum_{j\in \{1,2\}}\mathbf{Pr}_{x\sim S^{(i)}_{j}}[|x|\in A\sigma k]\pm O(\eta /i).
$$
10.  $t^\prime (i)\coloneqq t^\prime (i - 1) + \mathbf{E}_{z\sim S_1^{(i)}}[z\mid |z|\leq A\sigma k] - \mathbf{E}_{z\sim S_2^{(i)}}[z\mid |z|\leq A\sigma k].$
11.  $A\coloneqq \eta A$
12. Return  $t^{\prime}(T)$
$O(\eta \mathbf{Pr}[\xi \in I])$  and  $\mathbf{Pr}[(\xi + z' \in I) \Delta (\xi \in I)] < O(\eta \mathbf{Pr}[\xi \in I])$ , i.e., neither  $z$  nor  $z'$  moves more than a negligible mass of  $\xi$  either into, or out of  $I$ .
To detect such an interval (if it exists), we control the total mass contained in, and moved across the intervals  $\sigma(-i - 1, -i)$  and  $\sigma(i, i + 1)$  when  $z$  or  $z'$  are added to  $\xi$ . We denote this mass by  $P''(i)$ . Claim C.4 shows that there is a function  $\tilde{P}(\cdot)$  which we can compute using our samples from the distributions of  $\xi + z$  and  $\xi + z'$ , which serves as a good upper bound on  $P''(i)$ .
Claim 3.3. There exists a function  $\tilde{P}:\mathbb{N}\to \mathbb{R}^{+}$  which satisfies the following:
1. For all  $i \in \mathbb{N}$ ,  $\tilde{P}(i) \geq P(i, z) + P(i, z')$  which can be computed using samples from  $\xi + z$  and  $\xi + z'$ .
2. There is a  $k \in [(1 / \alpha \eta), (C / \alpha + 1 / (\alpha \eta)^{\eta})^{1 / \eta}]$  such that  $k\tilde{P}(k) < \eta \sum_{j=0}^{k}\tilde{P}(k)$ .
3.  $\sum_{i = 0}^{k}\tilde{P} (k) = O(\mathbf{Pr}[|\xi +z|\leq A\sigma k] + \mathbf{Pr}[|\xi +z^{\prime}|\leq A\sigma k]).$
4. With probability  $1 - \delta$ , for all  $i < (O(1) / \alpha)^{1 / \eta} / \eta$ ,  $\tilde{P}(i)$  can be estimated to an accuracy of  $O(\eta / i)$  by using  $(O(1) / \alpha)^{2 / \eta}\log(1 / \eta\alpha\delta) / \eta^5$  samples from the distributions of  $\xi + z$  and  $\xi + z'$ .
Claim 3.4 shows that if such a  $\tilde{P}(\cdot)$  exists, then the difference of the conditional expectations suffices to get a good estimate. This is because conditioning on  $|\xi + z| < 10k\sigma$  satisfying conditions (2) and (3) above is almost the same as conditioning on  $|\xi| < 10k\sigma$ .
Claim 3.4. Suppose  $i > (\eta \alpha)^{-1}$  and  $\tilde{P}(\cdot)$  satisfies the conclusions of Claim C.4. If  $z, z'$  have  $A\sigma$  bounded means and  $4\sigma^2$  bounded variances for some  $A$ , then  $\mathbf{E}[\xi + z| |\xi + z| \leq A\sigma i] - \mathbf{E}[\xi + z'| |\xi + z'| \leq A\sigma i] = \mathbf{E}[z] - \mathbf{E}[z'] \pm O(A\eta\sigma)$ .
The conclusion of the one-dimensional location estimation theorem then follows by putting these together, and iterating the above steps after re-centering the means. The sample complexity is governed by the following considerations:
(1) We require the  $\mathbf{E}[\xi + z \mid |\xi + z| \leq \sigma \alpha^{-1/2} k] - \mathbf{E}[\xi + z' \mid |\xi + z'| \leq \sigma \alpha^{-1/2} k]$  to concentrate around its mean for some  $i \leq \sigma (C / \alpha + 1 / (\alpha \eta)^{\eta})^{1/\eta}$ . (2) We require  $\mathbf{Pr}[\xi + z \in \sigma(i, i + 1)]$  to concentrate for all integer  $i$  of magnitude at most  $\sigma (C / \alpha + 1 / (\alpha \eta)^{\eta})^{1/\eta}$ .
An application of Hoeffding's inequality (Lemma A.1) and a union bound over all the events prove that it suffices to ensure sample complexity to be  $m \geq \mathrm{poly}(1 / \eta, (O(1) / \alpha)^{1 / \eta}, \log(1 / \delta \eta \alpha))$ .
The following Lemma 3.5 uses the algorithm for one dimension to derive a higher-dimensional guarantee. The idea is to perform a random rotation, which, with high probability, ensures that the variance of the distribution is  $O(\sigma \sqrt{\log(d)} / \sqrt{d})$  along each basis and then apply the one-dimensional algorithm coordinate-wise according to the random bases. We defer the proof to Appendix C. The algorithm referenced in this lemma is named "ShiftHighD", which we use later.
Lemma 3.5 (Location Estimation). Let  $D_{y_i}$  for  $i \in \{1,2\}$  be the distributions of  $\xi + z_i$  for  $i \in \{1,2\}$  where  $\xi$  is drawn from  $D_{\xi}$  and  $\mathbf{Pr}_{\xi \sim D_{\xi}}[\xi = 0] \geq \alpha$  and  $z_i \sim D_i$  are distributions over  $\mathbb{R}^d$  satisfying  $\mathbf{E}_{D_i}[x] = 0$  and  $\mathbf{E}_{D_i}[\| x\|^2] \leq \sigma^2$ . Let  $v \in \mathbb{R}^d$  be an unknown shift, and  $D_{y_2,v}$  denote the distribution of  $y_2$  shifted by  $v$ . There is an algorithm (Algorithm 5), which draws  $\mathrm{poly}(1/\eta, (O(1)/\alpha)^{1/\eta}, \log(1/\delta\eta\alpha))$  samples each from  $D_{y_1}$  and  $D_{y_2,v}$  runs in time  $\mathrm{poly}(d,1/\eta, (O(1)/\alpha)^{1/\eta}, \log(1/\delta\eta\alpha))$  and returns  $v'$  satisfying  $\| v' - v \| \leq O(\eta\sigma)$  with probability  $1 - \delta$ .
4 Algorithmic Results for Stochastic Optimization
Our first main result demonstrates that if we have access to an algorithm that recovers a list of size  $s$  such that one of the means is  $O(\eta \sigma)$ -close to the true mean in  $\ell_2$  norm, then there is an algorithm which is able to recover  $\hat{x}$  such that  $\| \nabla f(\hat{x})\| \leq O(\eta \sigma) + \epsilon$  given access to gradients that are close to the true gradient up to an  $\ell_2$  error of  $O(\eta \sigma)$ .
4.1 Proof of Theorem 1.4: List-decodable Stochastic Optimization  $\rightarrow$  LDME
Algorithm 2 Noisy Gradient Optimization: NoisyGradDesc(α,τ,δ,O,AG,AME)
Input:  $\alpha, \eta, \delta$ ,  $(\alpha, O(\eta \sigma), s)$ -LDME algorithm  $\mathcal{A}_{ME}$  for  $\mathcal{D}_{\sigma}, \mathcal{O}_{\alpha, \sigma, f}(\cdot), (O(\eta \sigma), \epsilon)$ -learner  $\mathcal{A}_G$ .
1. Let  $m' = m_{A_{ME}}$
2. Query  $\mathcal{O}_{\alpha, \sigma, f}(\bar{0})$  to get samples  $S^* \gets \{\nabla f(\gamma_i, 0) + \xi_i \mid i \in [m']\}$ .
3. Let  $\mathcal{L}_0\coloneqq \{g_1,\ldots ,g_s\} \gets \mathcal{A}_{ME}(S)$
4. Initialize starting points  $x_{i}^{0} \coloneqq g_{i}$  for  $i \in [s]$ .
5. For each  $i\in [s]$ , run  $\mathcal{A}_G$  with  $x_{i}^{0}$  as the initial point and InexactOracle  $(x;\mathcal{O}_{\alpha ,\sigma ,f}(\cdot),O(\eta \sigma),\mathcal{L}_0)i$  (the  $i$ -th element of the list) as gradient access to output  $x_{i}^{final}$ .
6. Return  $\{x_i^{final} \mid i \in [s]\}$ .
Proof. The key to our proof is to recognize that at every step, we effectively have an oracle for an inexact gradient of  $f$  in the sense of  $\mathcal{O}_{\mathcal{O}(\eta \sigma),f}^{\mathrm{grad}}$  as defined in Definition 2.2.
Algorithm 3 Inexact Gradient Oracle: InexactOracle  $(x;\mathcal{O}_{\alpha ,\sigma ,f},\tau ,\mathcal{L}_0)$
input:  $x$ , oracle  $\mathcal{O}_{\alpha, \sigma, f}$ , error  $\tau$ , a list  $\mathcal{L}_0$  of candidates such that  $\min_{g \in \mathcal{L}} \| g - \nabla f(0) \| \leq O(\eta \sigma)$ .
1. Let  $\eta \coloneqq \sigma /\tau$  and  $m^{\prime} = \bar{O} ((1 / \eta \alpha)^{3 / \eta})$
2. Query  $\mathcal{O}_{\alpha, \sigma, f}(0)$  to get samples  $S^* \gets \{\nabla f(\gamma_i, 0) + \xi_i \mid i \in [m']\}$ .
3. Query  $\mathcal{O}_{\alpha, \sigma, f}(x)$  to get samples  $S \gets \{\nabla f(\gamma_i, x) + \xi_i \mid i \in [m']\}$ .
4.  $v\coloneqq \mathrm{ShiftHighD}(S,S^{*},\eta ,\sigma)$
5. Return  $\mathcal{L}_0 + v$
Claim 4.1. Given an initial list  $\mathcal{L}_0\coloneqq \{g_1,\ldots ,g_s\}$  such that there is some fixed  $i\in [s]$  for which  $\| g_{i} - \nabla f(0)\| \leq O(\eta \sigma)$ , a point  $x$ , access to  $\mathcal{O}_{\alpha ,\sigma ,f}(\cdot)$ , Algorithm 3 returns a list  $\mathcal{L}\coloneqq \{g_1',\dots ,g_s'\}$  such that  $\| g_i^{\prime} - \nabla f(x)\| \leq O(\eta \sigma)$  for the same  $i$ .
Proof.  $\mathcal{O}_{\alpha, \sigma, f}(y)$  returns samples of the kind  $\{\xi_i + e(\gamma, x)_i + \nabla f(x)\}_{i=1}^m$  and  $\{\tilde{\xi}_i + e(\gamma, 0)_i + \nabla f(0)\}_{j=1}^m$  for  $y = x$  and  $y = 0$ , with  $e(\gamma, y)_i$  being drawn from a distribution with 0 and variance bounded by  $\sigma^2$  and  $\xi_i$  and  $\tilde{\xi}_i$  being drawn from  $D_\xi$  where  $\mathbf{Pr}_{\xi \sim D_\xi}[\xi = 0] \geq \alpha$ . Hence, one can interpret the samples drawn as being in the setting of Lemma 3.5 with the shift  $v = \nabla f(0) - \nabla f(x)$ .
Let  $S_{1}$  and  $S_{2}$  be drawn from  $\mathcal{O}_{\alpha, \sigma, f}(y)$  with  $y = 0$  and  $y = x$ . Running Algorithm 5 on  $S_{1}, S_{2}$ , we recover  $v$  satisfying  $\| v - (\nabla f(x) - \nabla f(0)) \| \leq O(\eta \sigma)$ . A triangle inequality now tells us that if we set  $g_{i}' := g_{i} + v$ , we get  $\| g_{i}' - \nabla f(x) \| = \| g_{i}' - g_{i} + g_{i} - \nabla f(0) + \nabla f(0) - \nabla f(x) \| \leq O(\eta \sigma) + O(\eta \sigma) = O(\eta \sigma)$ .
$g_{i}^{\prime}$  can be interpreted as the output of  $O_{O(\eta \sigma),f}^{\mathrm{grad}}(x)$ , since  $\| g_i^\prime -\nabla f(x)\| \leq O(\eta \sigma)$ . The result then follows from the guarantees of  $\mathcal{A}_G$ , since for at least one of the sequences of gradients, Algorithm 2 replicates every step of  $\mathcal{A}_G$ .
Substituting the guarantees of Fact 2.1 for the list-decoding algorithm in the above theorem then gives us the following corollary, the proof of which we defer to Appendix E.
Corollary 4.2. Given access to oblivious noise oracle  $\mathcal{O}_{\alpha,\sigma,f}$ , a  $(O(\eta\sigma),\epsilon)$ -inexact-learner  $\mathcal{A}_G$  running in time  $T_G$ , there exists an algorithm which takes  $\mathrm{poly}((O(1)/\alpha)^{1/\eta^2}, \log (T_G / \delta \eta \alpha))$  samples, runs in time  $T_G \cdot \mathrm{poly}(d, (O(1)/\alpha)^{1/\eta^2}, \log (1/\eta \alpha \delta))$ , and with probability  $1 - \delta$  returns a list  $\mathcal{L}$  of size  $\tilde{O}((1/\alpha)^{1/\eta^2})$  such that  $\min_{x \in \mathcal{L}} \| \nabla f(x) \| \leq O(\eta \sigma) + \epsilon$ . Additionally, the exponential dependence on  $1/\eta$  in the size of the list is necessary.
4.2 Proof of Theorem 1.6: LDME  $\rightarrow$  List-Decodable Stochastic Optimization
In this subsection, we show the converse of the results from the previous subsection, i.e. that list-decodable stochastic optimization can be used to perform list-decodable mean estimation.
Proof. Assume there exists an algorithm  $\mathcal{A}$  that can recover a  $s$ -sized list containing  $\widehat{x}$  such that  $\| \nabla f(\widehat{x}) \| \leq \epsilon$  when given access to  $\mathcal{O}_{\alpha, \sigma, f}^{o}(\cdot)$ . From the list-decodable mean estimation setting, denote  $D$  to be the distribution of  $\xi + z$  where  $\mathbf{E}[z] = \mu$  and  $\mathbf{E}[\|z - \mu\|^2] < \sigma^2$ , and  $\mathbf{Pr}[\xi = 0] \geq \alpha$ .
The goal of LDME is to recover  $\mu$  from samples from  $D$ . We will show that we can recover an  $s$ -sized list that contains a  $\widehat{\mu}$  satisfying  $\| \widehat{\mu} - \mu \| \leq \epsilon$ .
We do this by simulating the oracle  $\mathcal{O}_{\alpha, \sigma, (1/2) \| x + \mu \|^2}(\cdot)$  for  $\mathcal{A}$ . To do this, whenever  $\mathcal{A}$  asks for a query at  $x$ , we return  $x + p$  where we sample  $p \sim D$  of the list-decodable setting. This effectively simulates the oracle for the objective function  $f(x) = (1/2) \| x + \mu \|^2$ , where the oblivious noise  $\xi$  is the same, and the observation noise is  $(z - \mu)$ . Hence  $\mathcal{A}$  will return a list  $\mathcal{L}$  of size  $s$  containing  $\widehat{x}$  satisfying  $\| \widehat{x} + \mu \| \leq \epsilon$ . Finally, return  $-\mathcal{L}$ . This is the solution to list-decodable mean estimation because  $-\mathcal{L}$  contains  $\widehat{\mu} := -\widehat{x}$ , which satisfies  $\| \widehat{\mu} - \mu \| \leq \epsilon$ .
In this paper, we have initiated the study of stochastic optimization in the presence of oblivious noise, which extends the traditional heavy-tailed noise framework. In our setting, the stochastic gradient is additionally affected by independent oblivious noise that lacks bounded moments and may not be centered. We have also designed an algorithm for finite-sample noisy location estimation based on taking conditional expectations, which we believe is of independent interest. We note that while the exponential dependence on  $1 / \eta$  for the size of the list is unavoidable, but it is an open problem to show that this is the case for the problem of noisy location estimation.