SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| \hat {\mu} _ {1} - \mu^ {*} \right\| \leqslant O \left(\varepsilon \sqrt {\log (1 / \varepsilon)} + \sqrt {\varepsilon \left(\left\| \Sigma_ {1} ^ {\mathcal {L}} - \gamma \cdot \operatorname {I d} _ {d} \right\| + \varepsilon \log (1 / \varepsilon)\right)}\right),
$$
where  $\Sigma_1^{\mathcal{L}}$  is defined analogously to  $\Sigma_2^{\mathcal{L}}$ . Second, we show that for the filtering approach to work, it is enough that the transformed distribution, i.e.,  $\nabla \mathcal{L}(\pmb{y} - \mu^{*})$  for  $\pmb{y} \sim D_{2}$ , satisfies the stability condition. This indeed follows since for our choice of  $\mathcal{L}$  (the entry-wise Huber-loss),  $\nabla \mathcal{L}(\pmb{y} - \mu^{*})$  is sub-Gaussian.
However, since we do not work with the quadratic loss anymore, there are several technical obstacles. For the quadratic loss, the gradient is the identity function, and this fact is extensively used in the analysis of the Gaussian setting. For the Huber-loss gradient, different arguments are needed. To exemplify this, consider the following example - note that we do not expect the reader to see at which point in the analysis this step is necessary, it should merely illustrate the types of problems that arise. Let  $S_{g}$  denote the set of uncorrupted samples,  $T \subseteq [n]$  be a set of size at least  $(1 - \varepsilon)n$ , and  $\hat{\mu}(T)$  be the minimizer of  $\mathcal{L}$  over samples in  $T$ . In the analysis, terms of the following nature arise
$$
\frac {1}{| T \cap S _ {g} |} \sum_ {i \in T \cap S _ {g}} \nabla \mathcal {L} (\hat {\mu} (T) - \boldsymbol {y} _ {i} ^ {*}) \nabla \mathcal {L} (\hat {\mu} (T) - \boldsymbol {y} _ {i} ^ {*}) ^ {\top}
$$
and we need to show that this term is bounded from below, in Loewner order, by  $(\gamma - O(\varepsilon \log (1 / \varepsilon)))\mathrm{Id}_d$  (uniformly for all  $T$  of size at least  $(1 - \varepsilon)n$ ). In case of the quadratic loss, this follows easily from the stability conditions by adding and subtracting  $\hat{\mu}(T \cap S_g)$ .
When  $\mathcal{L}$  is the entry-wise Huber-loss, a more sophisticated argument is required. To describe our argument, we assume for simplicity that the entries of  $\pmb{\eta}$  are mutually independent. We first show that  $\hat{\Delta} \coloneqq \hat{\mu}(T) - \mu^{*}$  has entries of magnitude  $O(\varepsilon)$ . Then we show that for arbitrary but fixed (that is, non-random)  $\Delta$  with entries of small magnitude, the distribution of  $\nabla \mathcal{L}(\Delta - \pmb{\eta}^{*})$  is sub-Gaussian. We use this fact to show that with overwhelming probability
$$
\frac {1}{| T \cap S _ {g} |} \sum_ {i \in T \cap S _ {g}} \nabla \mathcal {L} (\Delta - \boldsymbol {\eta} _ {i} ^ {*}) \nabla \mathcal {L} (\Delta - \boldsymbol {\eta} _ {i} ^ {*}) ^ {\top} \succeq (1 - \tilde {O} (\varepsilon)) \operatorname {C o v} (\nabla \mathcal {L} (\Delta - \boldsymbol {\eta} ^ {*})).
$$
Then we use the fact that  $\Delta$  has small entries to show that  $\mathrm{Cov}\left(\nabla \mathcal{L}(\Delta -\eta^{*})\right)\succeq (\gamma -O(\varepsilon))\mathrm{Id}_d$ . Finally, we use an  $\varepsilon$ -net over all possible  $\Delta$  to get the desired lower bound uniformly for all  $\Delta$ , including  $\hat{\Delta}$ . This last  $\varepsilon$ -net argument is where we incur the (possibly) sub-optimal  $O(d\log d)$  term (instead of the optimal  $O(d)$ ) in our sample complexity. Finally, if the entries of  $\eta$  are not independent, we need to use this argument conditioned on the absolute values of the entries of  $\eta$ .
Summarizing, we have shown how to adjust the well-known filtering technique and incorporate the Huber-loss, to design robust algorithms for symmetric distributions, often matching what is known for the Gaussian distribution in (quasi-)polynomial time.
[Bec09] Ikhlef Bechar, A bernstein-type inequality for stochastic processes of quadratic forms of gaussian variables, arXiv preprint arXiv:0909.3595 (2009).
[Ber06] Thorsten Bernholt, Robust estimators are hard to compute, Tech. report, Technical report, 2006.
[CGR18] Mengjie Chen, Chao Gao, and Zhao Ren, Robust covariance and scatter matrix estimation under huber's contamination model, The Annals of Statistics 46 (2018), no. 5, 1932–1960.
Note that the deviation of  $f(\Delta - \eta^{*}) f(\Delta - \eta^{*})^{\top}$  from its mean scales with  $\mathbb{E}f(\Delta - \eta^{*})$ , which can be large. But we can still show the lower bound of the form  $\left(1 - \tilde{O}(\varepsilon)\right) \operatorname{Cov}(f(\Delta - \eta^{*}))$ , where  $\tilde{O}(\varepsilon)$  does not contain the terms that scale with  $\mathbb{E}f(\Delta - \eta^{*})$ . This is enough for our argument.
[Fan18] Kai Wang Fang, Symmetric multivariate and related distributions, Chapman and Hall/CRC, 2018.
[Fra04] Gabriel Frahm, Generalized elliptical distributions: theory and applications, Ph.D. thesis, Universität zu Köln, 2004.
[GLP23] Shivam Gupta, Jasper CH Lee, and Eric Price, Finite-sample symmetric mean estimation with fisher information rate, The Thirty Sixth Annual Conference on Learning Theory, PMLR, 2023, pp. 4777-4830.
[HL18] Samuel B Hopkins and Jerry Li, Mixture models, robustness, and sum of squares proofs, Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, 2018, pp. 1021-1034.
[HL19] , How hard is robust mean estimation?, Conference on Learning Theory, PMLR, 2019, pp. 1649-1682.
[HLZ20] Sam Hopkins, Jerry Li, and Fred Zhang, Robust and heavy-tailed mean estimation made simple, via regret minimization, Advances in Neural Information Processing Systems 33 (2020), 11902-11912.
[Hub11] Peter J Huber, Robust statistics, International encyclopedia of statistical science, Springer, 2011, pp. 1248-1251.
[Kel70] Douglas Kelker, Distribution theory of spherical distributions and a location-scale parameter generalization, Sankhya: The Indian Journal of Statistics, Series A (1970), 419-430.
[KMZ22] Pravesh K Kothari, Peter Manohar, and Brian Hu Zhang, *Polynomial-time sum-of-squares can robustly estimate mean and covariance of gaussians optimally*, International Conference on Algorithmic Learning Theory, PMLR, 2022, pp. 638-667.
[KS17a] Pravesh K. Kothari and David Steurer, Outlier-robust moment-estimation via sum-of-squares, CoRR abs/1711.11581 (2017).
[KS17b] (2017), Outlier-robust moment-estimation via sum-of-squares, CoRR abs/1711.11581
[KSS18] Pravesh K Kothari, Jacob Steinhardt, and David Steurer, Robust moment estimation and improved clustering via sum of squares, Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, 2018, pp. 1035-1046.
[Li19] Jerry Li, Robustness in machine learning, lecture notes, https://jerryzli.github.io/robust-ml-fall19.html, 2019, Lecture 5.
[Lin75] John Lintner, The valuation of risk assets and the selection of risky investments in stock portfolios and capital budgets, Stochastic optimization models in finance, Elsevier, 1975, pp. 131-155.
[LRV16] Kevin A Lai, Anup B Rao, and Santosh Vempala, Agnostic estimation of mean and covariance, 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS), IEEE, 2016, pp. 665-674.
[MSS16] Tengyu Ma, Jonathan Shi, and David Steurer, Polynomial-time tensor decompositions with sum-of-squares, FOCS, IEEE Computer Society, 2016, pp. 438-446.
[MZ23] Arshak Minasyan and Nikita Zhivotovsky, Statistically optimal robust mean and covariance estimation for anisotropic gaussians, arXiv preprint arXiv:2301.09024 (2023).
[OR83] Joel Owen and Ramon Rabinovitch, On the class of elliptical distributions and their applications to the theory of portfolio choice, The Journal of Finance 38 (1983), no. 3, 745-752.
[PRB20] Adarsh Prasad, Sivaraman Balakrishnan, and Pradeep Ravikumar, A robust univariate mean estimator is all you need, International Conference on Artificial Intelligence and Statistics, PMLR, 2020, pp. 4034-4044.
[PJL20] Ankit Pensia, Varun Jog, and Po-Ling Loh, Robust regression with covariate filtering: Heavy tails and adversarial contamination, arXiv preprint arXiv:2009.12976 (2020).
[RHRS11] Peter J Rousseeuw, Frank R Hampel, Elvezio M Ronchetti, and Werner A Stahel, Robust statistics: the approach based on influence functions, John Wiley & Sons, 2011.
[ST21] David Steurer and Stefan Tiegel, *Sos degree reduction with applications to clustering and robust moment estimation*, Proceedings of the 2021 ACM-SIAM Symposium on Discrete Algorithms (SODA), SIAM, 2021, pp. 374-393.
[Sto75] Charles J Stone, Adaptive maximum likelihood estimators of a location parameter, The annals of Statistics (1975), 267-284.
[SZF20] Qiang Sun, Wen-Xin Zhou, and Jianqing Fan, Adaptive huber regression, Journal of the American Statistical Association 115 (2020), no. 529, 254-265.
[TJSO14] Efthymios Tsakonas, Joakim Jaldén, Nicholas D Sidiropoulos, and Björn Ottersten, Convergence of the huber regression m-estimate in the presence of dense outliers, IEEE Signal Processing Letters 21 (2014), no. 10, 1211–1214.
[Tro15] Joel A. Tropp, An introduction to matrix concentration inequalities, Foundations and Trends in Machine Learning 8 (2015), no. 1-2, 1-230.
[Wai19] Martin J. Wainwright, High-dimensional statistics: A non-asymptotic viewpoint, Cambridge Series in Statistical and Probabilistic Mathematics, Cambridge University Press, 2019.
A Overview of Appendices
We will provide a general framework underlying the proofs of Theorem 1.5 and 1.6 in Appendices C to E. We will instantiate it to prove (a slight generalization of) Theorem 1.5 in Appendix F and to prove Theorem 1.6 in Appendix G. We will prove Theorem 1.4 in Appendices H and I. Appendix H contains the proof of identifiability and Appendix I contains the adaptation of the filtering algorithm.
As alluded to in the introduction, we will consider the following (slight) generalization of  $(\rho)$ -semiproduct distributions (Definition 1.2) which allows for even less probability mass around the location. Setting  $\alpha = \frac{1}{100}$  recovers Definition 1.2.
Definition A.1 (Semi-Product Distributions (generalized form)). Let  $\rho, \alpha > 0$ . We say a distribution  $\mathcal{D}$  over  $\mathbb{R}^d$  is an  $(\alpha, \rho)$ -semi-product distribution, if for  $\eta \sim \mathcal{D}$  it holds that
1. For all  $j \in [d]$ , the distribution of  $\pmb{\eta}_j$  is symmetric about 0,
2. For all  $j\in [d]$ $\mathbb{P}(|\pmb {\eta}_j|\leqslant \rho)\geqslant \alpha$
3. The random vectors  $(\mathrm{sign}(\pmb{\eta}_j))_{j=1}^d$  and  $(|\pmb{\eta}_j|)_{j=1}^d$  are independent, and the random variables  $\mathrm{sign}(\pmb{\eta}_1), \dots, \mathrm{sign}(\pmb{\eta}_d)$  are mutually independent.
We will prove the following two (slight) generalizations of Theorems 1.4 and 1.5. Setting  $\alpha = \frac{1}{100}$  recovers Theorems 1.4 and 1.5, respectively.
Theorem A.2. Let  $\mu^{*} \in \mathbb{R}^{d}, \varepsilon, \rho, \alpha > 0$  and  $\mathcal{D}$  be a  $(\alpha, \rho)$ -semi-product distribution with location  $\mu^{*}$ . Let  $C > 0$  be a large enough absolute constant and assume that  $\sqrt[3]{\varepsilon} \leqslant \alpha / C$  and  $n \geqslant C \cdot \frac{d\log(d) + \log(1 / \delta)}{\alpha^4\varepsilon^2\log(1 / \varepsilon)}$ . Then, there exists an algorithm that, given an  $\varepsilon$ -corrupted sample from  $\mathcal{D}$  and  $\alpha$ , runs in time  $n^{O(1)}$  and outputs  $\hat{\mu} \in \mathbb{R}^{d}$  such that with probability at least  $1 - \delta$  it holds that
$$
\left\| \hat {\mu} - \mu^ {*} \right\| \leqslant O \left(\rho \cdot \left[ \sqrt {\frac {d \log (d) + \log (1 / \delta)}{\alpha^ {4} n}} + \frac {\varepsilon}{\alpha^ {3}} \sqrt {\log (1 / \varepsilon)} \right]\right).
$$
Theorem A.3. Let  $\mu^{*} \in \mathbb{R}^{d}, \varepsilon, \alpha, \rho > 0$  and  $\mathcal{D}$  be a  $(\alpha, \rho)$ -semi-product distribution with location  $\mu^{*}$ . Let  $C > 0$  be a large enough absolute constant and assume that  $\varepsilon \leqslant \alpha / C$  and  $n \geqslant \frac{C}{\alpha^4} \cdot d^{C \log(1 / \varepsilon)}$ . Then, there exists an algorithm that, given an  $\varepsilon$ -corrupted sample from  $\mathcal{D}$  and  $\alpha$ , runs in time  $n^{O(1)}$  and outputs  $\hat{\mu} \in \mathbb{R}^{d}$  such that with probability at least  $1 - \delta$  it holds that
$$
\| \hat {\mu} - \mu^ {*} \| \leqslant O \left(\rho \cdot \left[ \sqrt {\frac {d + \log (1 / \delta)}{\alpha^ {4} n}} + \frac {\varepsilon}{\alpha^ {2}} \right]\right).
$$
We remark that if, the algorithm receives  $\rho$  as input, the error guarantee improves to
$$
\left\| \hat {\mu} - \mu^ {*} \right\| \leqslant O \left(\rho \cdot \left[ \sqrt {\frac {d + \log (1 / \delta)}{\alpha^ {2} n}} + \frac {\varepsilon}{\alpha} \right]\right).
$$
Note that for the setting of  $\alpha = 1 / 100$  discussed in the introduction, both guarantees are the same (up to constant factors). Further, if  $\rho$  is given as input, it is not necessary to receive  $\alpha$  as input. We emphasize, that  $\rho$  need not be known, and can be estimated from the corrupted samples (given  $\alpha$ ).
We also remark, that our upper bound on the fraction of corruptions is qualitatively tight, since  $\varepsilon$  can be at most roughly  $\alpha$ : Consider a mixture distribution that with probability  $1 - \alpha$  outputs a sample from  $N(\mu^{*}, \sigma^{2} \mathrm{Id}_{d})$  for  $\sigma^{2}$  arbitrarily large and with probability  $\alpha$  outputs a sample from  $N(\mu^{*}, \mathrm{Id}_{d})$ . This is  $(\alpha, \Theta(1))$ -semi-product. Yet, if an adversary changes the  $N(\mu^{*}, \mathrm{Id}_{d})$  component to  $N(\mu^{*}, \sigma^{2} \mathrm{Id}_{d})$ , we cannot hope to achieve error better than  $\sigma^{2} \sqrt{\frac{d + \log(1 / \delta)}{n}}$ , which can be arbitrarily large. We did not attempt to optimize the range of  $\varepsilon$  allowed for our algorithms.
Notation We use bold-font for random variables. We use regular (non-bold-font) for random variables that potentially have been adversarially corrupted. Further, we use  $\gtrsim$  and  $\lesssim$  to suppress absolute constants which do not depend on any other problem parameters. For a (pseudo)-distribution  $\mu$ , we denote by  $\mathrm{supp}(\mu)$  its support.
Sum-of-Squares Proofs and Pseudo-Expectations In this section, we will introduce sum-of-squares proofs and their convex duals, so-called pseudo-distributions.
Let  $p \colon \mathbb{R}^n \to \mathbb{R}$  be a polynomial in formal variables  $X = (X_1, \ldots, X_n)$ . We say the inequality  $p(X) \geqslant 0$  has a sum-of-squares proof (short SoS proof) if we can write  $p(X)$  as a sum of squares in  $X_1, \ldots, X_n$ . Further, if every polynomial in this decomposition has degree at most  $t$ , we say that the sum-of-squares proof has degree- $t$  and write  $\left| \frac{X}{t} \right| p \geqslant p'$  if  $A \left| \frac{X}{t} \right| p - p' \geqslant 0$  and  $A \left| \frac{X}{t} \right| p = p'$  if  $A \left| \frac{X}{t} \right| p \leqslant p'$  and  $A \left| \frac{X}{t} \right| p \geqslant p'$ . It is not hard to verify that the following composition rule holds: If  $\left| \frac{X}{t} \right| p \geqslant p'$  and  $\left| \frac{X}{t'} \right| p' \geqslant p''$ , then it also holds that  $\left| \frac{X}{t''} \right| p \geqslant p''$ , where  $t'' = \max\{t, t'\}$ .
Next, we introduce so-called pseudo-distributions, the convex duals of sum-of-squares proofs: For  $d \in \mathbb{N}_{\geqslant 1}$ , a degree-  $d$  pseudo-distribution is a finitely-supported function  $\mu \colon \mathbb{R}^n \to \mathbb{R}$  such that  $\sum_{x \in \operatorname{supp}(\mu)} \mu(x) = 1$  and  $\sum_{x \in \operatorname{supp}(\mu)} \mu(x)f^2(x) \geqslant 0$  for all polynomials  $f$  of degree at most  $d/2$ . The pseudo-expectation corresponding to a pseudo-distribution  $\mu$  is defined as the linear operator mapping a degree-  $d$  polynomial  $f$  to  $\tilde{\mathbb{E}}_{\mu}f \coloneqq \sum_{x \in \operatorname{supp}(\mu)} \mu(x)f(x)$ . We say that a pseudo-distribution  $\tilde{\mathbb{E}}_{\mu}$  satisfies a set of inequalities  $\{q_1 \geqslant 0, \dots, q_m \geqslant 0\}$  at degree  $r$ , if for all  $S \subseteq [m]$  and  $h$  a sum-of-squares polynomial such that  $\deg(h \cdot \Pi_{j \in S}) \leqslant r$  it holds that
$$
\tilde {\mathbb {E}} _ {\mu} h \cdot \Pi_ {j \in S} \geqslant 0.
$$
We relate pseudo-distributions and sum-of-squares proofs using the following facts.
Fact B.1. Let  $\mu$  be a degree- $d$  pseudo-distribution that satisfies  $\mathcal{A}$  and suppose that  $\mathcal{A} \Big|_{t} p \geqslant 0$ . Let  $h$  be an arbitrary sum-of-squares polynomial, if  $\deg h + t \leqslant d$  we have  $\tilde{\mathbb{E}}_{\mu} h \cdot p \geqslant 0$ . In particular, we have  $\tilde{\mathbb{E}}_{\mu} p \geqslant 0$  as long as  $t \leqslant d$ . If  $\mu$  only approximately satisfies  $\mathcal{A}$  it holds that  $\tilde{\mathbb{E}}_{\mu} p \geqslant -\eta \| h \|_2$  for  $\eta = 2^{-n^{\Omega(d)}}$ . Further, if there is no degree- $d$  sum-of-squares proof that  $\mathcal{A} \Big|_{t} p \geqslant 0$ , then for there exists a degree- $d$  pseudo-distribution  $\tilde{\mathbb{E}}_{\mu}$  satisfying  $\mathcal{A}$  and  $\varepsilon > 0$  such that  $\tilde{\mathbb{E}}_{\mu} p < 0$ .
An essential fact is that pseudo-distributions approximately satisfying a system of polynomial inequalities can be found efficiently as long as the constraints have bit-complexity at most  $(m + n)^{\mathcal{O}(1)}$ . Fact B.2. Given any feasible system of polynomial constraints  $\mathcal{A} = \{q_1 \geqslant 0, \dots, q_m \geqslant 0\}$  over  $\mathbb{R}^n$  whose bit-complexity satisfies the constraints mentioned above and  $\eta = 2^{-n^{\Theta(d)}}$  we can in time  $(n + m)^{\mathcal{O}(d)}$  find a degree- $d$  pseudo-distribution that satisfies  $\mathcal{A}$  up to error  $\eta$ .<sup>8</sup>
C Proof of Identifiability based on Transformed Moments
In this section, we give a formal proof of the statement of identifiability based on transformed moments that we sketched in the Techniques section. We will give the proof assuming an abstract set of assumptions and verify that they indeed hold for semi-product and elliptical distributions in Appendices F and G.
We use the same notation as in Appendix H, with the only difference being that we now use an abstract loss function: Let  $F \colon \mathbb{R}^d \to \mathbb{R}$  and denote by  $f = \nabla F$  its gradient. Further assume that  $F$  and  $f$  are such that  $f$  is 1-Lipschitz. For  $w \in \mathbb{R}^n$  with non-negative entries, we define  $\mathcal{L}^w \colon \mathbb{R}^d \to \mathbb{R}$  as
$$
\mathcal {L} ^ {w} (\mu) := \sum_ {i = 1} ^ {n} w _ {i} F \left(\mu - y _ {i}\right).
$$
It follows that  $\nabla \mathcal{L}^w (\mu) = \sum_{i = 1}^n w_if\left(\mu -y_i\right)$ . Let  $2k\in \mathbb{N}_{\geqslant 2}$  and define  $\hat{\mu} (w)\coloneqq \min_{\mu \in \mathbb{R}^d}\mathcal{L}^w\left(\mu\right)$  as well as
$$
\bar {\mathfrak {m}} _ {2 k} := \max  _ {v \in \mathbb {S} ^ {d - 1}} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\langle f \left(\boldsymbol {\eta} _ {i} ^ {*}\right), v \right\rangle^ {2 k} \right] ^ {\frac {1}{2 k}} \quad \text {a n d} \quad \hat {\mathfrak {m}} _ {2 k} ^ {w} := \max  _ {v \in \mathbb {S} ^ {d - 1}} \left[ \sum_ {i = 1} ^ {n} w _ {i} \left\langle f \left(\hat {\mu} - y _ {i}\right), v \right\rangle^ {2 k} \right] ^ {\frac {1}{2 k}}.
$$
For technical reason we also assume that  $\mathcal{A}$  is explicitly bounded meaning it contains a constraint of the form  $\| x\| _2^2\leqslant B$  for some large number  $B$ . Usually we can even take this to be at least polynomial in our variables. We remark that for all the problems we are considering this can be added without changing our proofs.
The positivity and normalization constraint can be satisfied exactly however.
We will prove the following theorem.
Theorem C.1. Let  $\alpha \in (0,1)$  and let  $k$  be a positive integer and  $w \in \mathcal{W}_{2\varepsilon}$ . Suppose that  $\varepsilon^{1 - \frac{1}{2k}} \leqslant 0.01 \cdot \alpha$  and
$$
\alpha \| \hat {\mu} ^ {w} - \mu^ {*} \| ^ {2} \leqslant 1 0 \langle \nabla \mathcal {L} ^ {w} (\mu^ {*}), \mu^ {*} - \hat {\mu} ^ {w} \rangle .
$$
$$
\left\| \hat {\mu} ^ {w} - \mu^ {*} \right\| \leqslant \frac {1 0 0}{\alpha} \cdot \left(\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} f \left(\boldsymbol {\eta} _ {i} ^ {*}\right) \right\| + \varepsilon^ {1 - \frac {1}{2 k}} \cdot \left(\hat {\mathfrak {m}} _ {2 k} ^ {w} + \bar {\mathfrak {m}} _ {2 k}\right)\right).
$$
This theorem easily follows from the following lemma.
Lemma C.2. Let  $k$  be a positive integer and  $w \in \mathcal{W}_{2\varepsilon}$ . Then
$$
\left\langle \nabla \mathcal {L} ^ {w} \left(\mu^ {*}\right), u \right\rangle \leqslant 5 \cdot \| u \| \cdot \left(\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} f \left(\boldsymbol {\eta} _ {i} ^ {*}\right) \right\| + \varepsilon^ {1 - \frac {1}{2 k}} \cdot \left(\hat {\mathfrak {m}} _ {2 k} ^ {w} + \bar {\mathfrak {m}} _ {2 k} + \| u \|\right)\right),
$$
where  $u = \hat{\mu}(w) - \mu^{*}$ .
We first give the proof of Theorem C.1.
Proof of Theorem C.1. Let  $u = \hat{\mu}(w) - \mu^{*}$ . Using Lemma C.2 and  $\varepsilon^{1 - \frac{1}{2k}} \leqslant 0.01 \cdot \alpha$ , cancelling the common factor of  $\| u \|$  and rearranging now yields
$$
\| u \| \leqslant \frac {1 0 0}{\alpha} \cdot \left(\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} f (\boldsymbol {\eta} _ {i}) \right\| + \varepsilon^ {1 - \frac {1}{2 k}} \cdot \left(\hat {\mathfrak {m}} _ {2 k} ^ {w} + \bar {\mathfrak {m}} _ {2 k}\right)\right).
$$
Next, we prove Lemma C.2.
Proof of Lemma C.2. Again, let  $u = \hat{\mu}(w) - \mu^*$ . We can decompose  $\langle \nabla \mathcal{L}^w(\mu^*), u \rangle$  as
$$
\begin{array}{l} \left\langle \nabla \mathcal {L} ^ {w} \left(\mu^ {*}\right), u \right\rangle = \sum_ {i = 1} ^ {n} w _ {i} \left\langle f \left(\mu^ {*} - y _ {i}\right), u \right\rangle \\ = \sum_ {i = 1} ^ {n} w _ {i} \left\langle f \left(\mu^ {*} - \boldsymbol {y} _ {i} ^ {*}\right), u \right\rangle + \sum_ {i = 1} ^ {n} \mathbf {1} \left\{i \in S _ {b} \right\} \cdot w _ {i} \left\langle f \left(\mu^ {*} - y _ {i}\right) - f \left(\mu^ {*} - \boldsymbol {y} _ {i} ^ {*}\right), u \right\rangle . \\ \end{array}
$$
For the first sum,
$$
\sum_ {i = 1} ^ {n} w _ {i} \left\langle f \left(\mu^ {*} - \boldsymbol {y} _ {i} ^ {*}\right), u \right\rangle = \sum_ {i = 1} ^ {n} w _ {i} \left\langle f \left(\boldsymbol {\eta} _ {i} ^ {*}\right), u \right\rangle \leqslant \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} f \left(\boldsymbol {\eta} _ {i} ^ {*}\right) \right\| \cdot \| u \| + \sum_ {i = 1} ^ {n} \left(w _ {i} - \frac {1}{n}\right) \left\langle f \left(\boldsymbol {\eta} _ {i} ^ {*}\right), u \right\rangle .
$$
By Hölder's Inequality,
$$
\begin{array}{l} \sum_ {i = 1} ^ {n} \left(w _ {i} - \frac {1}{n}\right) \langle f (\boldsymbol {\eta} _ {i} ^ {*}), u \rangle \leqslant \left(\sum_ {i = 1} ^ {n} \left| w _ {i} - \frac {1}{n} \right|\right) ^ {1 - \frac {1}{2 k}} \left(\sum_ {i = 1} ^ {n} \left| w _ {i} - \frac {1}{n} \right| \cdot \langle f (\boldsymbol {\eta} _ {i} ^ {*}), u \rangle^ {2 k}\right) ^ {\frac {1}{2 k}} \\ \leqslant 4 \cdot \varepsilon^ {1 - \frac {1}{2 k}} \cdot \bar {\mathbf {m}} _ {2 k} \cdot \| u \|. \\ \end{array}
$$
For the second sum, we bound using Hölder's Inequality
$$
\begin{array}{l} \sum_ {i = 1} ^ {n} \mathbf {1} \left\{i \in S _ {b} \right\} \cdot w _ {i} \cdot \langle f (\mu^ {*} - y _ {i}) - f (\mu^ {*} - \boldsymbol {y} _ {i} ^ {*}), u \rangle \leqslant \varepsilon^ {1 - \frac {1}{2 k}} \left(\sum_ {i = 1} ^ {n} w _ {i} \cdot \langle f (\mu^ {*} - y _ {i}) - f (\mu^ {*} - \boldsymbol {y} _ {i} ^ {*}), u \rangle^ {2 k}\right) ^ {\frac {1}{2 k}} \\ = \varepsilon^ {1 - \frac {1}{2 k}} \left(\sum_ {i = 1} ^ {n} w _ {i} \cdot \langle f (\mu^ {*} - y _ {i}) - f (\mu^ {*} - \boldsymbol {y} _ {i} ^ {*}) - f (\hat {\mu} - y _ {i}) + f (\hat {\mu} - y _ {i}), u \rangle^ {2 k}\right) ^ {\frac {1}{2 k}} \\ \end{array}
$$