Robust High-Dimensional Mean Estimation With Low Data Size, an Empirical Study
Authors: Cullen Anderson
Jeff M. Phillips
Venue: Accepted by TMLR ()

Abstract: Robust statistics aims to compute quantities to represent data where a fraction of it may be arbitrarily corrupted.   The most essential statistic is the mean, and in recent years, there has been a flurry of theoretical advancement for efficiently estimating the mean in high dimensions on corrupted data.  While several algorithms have been proposed that achieve near-optimal error, they all rely on large data size requirements as a function of dimension. In this paper, we perform an extensive experimentation over various mean estimation techniques where data size might not meet this requirement due to the high-dimensional setting.  

For data with inliers generated from a Gaussian with known covariance, we find experimentally that several robust mean estimation techniques can practically improve upon the sample mean, with the quantum entropy scaling approach from Dong \etal (NeurIPS 2019) performing consistently the best.  However, this consistent improvement is conditioned on a couple of simple modifications to how the steps to prune outliers work in the high-dimension low-data setting, and when the inliers deviate significantly from Gaussianity. In fact, with these modifications, they are typically able to achieve roughly the same error as taking the sample mean of the uncorrupted inlier data, even with very low data size. In addition to controlled experiments on synthetic data, we also explore these methods on large language models, deep pretrained image models, and non-contextual word embedding models that do not necessarily have an inherent Gaussian distribution.  Yet, in these settings, a mean point of a set of embedded objects is a desirable quantity to learn, and the data exhibits the high-dimension low-data setting studied in this paper.  We show both the challenges of achieving this goal, and that our updated robust mean estimation methods can provide significant improvement over using just the sample mean. We additionally publish a library of Python implementations of robust mean estimation algorithms, allowing practitioners and researchers to apply these techniques and to perform further experimentation.

Published in Transactions on Machine Learning Research (02/2025)
Robust High-Dimensiont Mean Estimation With Low Data Size, an Empirical Study
Cullen Anderson
Syendersonnummass.co edu
University of Massachusetts Amherst
Jeff M. Phillips
jeffp utah. edu
University of Utah
Reviewed on OpenReview: htips: // openreview. net/ met/ forum? id- 1QcI99nA9k
Abstract
Robust statistics aims to compute quantities to represent data where fraction of it may be arbitrarily corrupted. The most essential statistic is the mean, and in recent years, there has been a flurry of theoretical advancement for efficiently estimating the mean in high dimensions on corrupted data. While several algorithms have been proposed that achieve near-optimal error, they all rely on large data size requirements as a function of dimension. In this paper, we perform an extensive experimentation over various mean estimation techniques where data size might not meet this requirement due to the high- dimensional setting.
For data with inliers generated from a Gaussian with known covariance, we find experi- mentally that several robust mean ostimation techniques can practically improve upon the sample mean, with the quantum enlropy scaling approach from Dong el.al. (NeurIPS 2019) performing consistently the best. However, this consistent improvement is conditioned on couple of simple modifications to how the steps to prune outliers work in the high-dimension low-data setting, and when the inliers deviate significantly from Gaussianity. In fact, with these modifications, they are typically able to achieve roughly the same error taking the sample mean of the uncorrupted inlier data, even with very low data size. In addition to controlled experiments on synthetic data, we also explore these mothods on large language models, deep raited image models, and non-contextual word embedding models that do not necessarily have an inherent Gaussian distribution. Yet, in these settings, a mean point of of embedded objects desirable quantity to learn, and the data exhibits the high-dimension low-data setting studied in this paper. We show both the challenges of achieving this goal, and that our updated robust mean estimation methods can provide significant improvement over using just the sample mean. We additionally publish a library of Python implementations of robust mean estimation algorithms, allowing practitioners and researchers to apply these techniques and to perform further experimentation.
1 Introduction
Given samples from unknown distribution, mean estimation is perhaps the most-fundamental and oldest problems in data analysis. And it is even more relevant in modern analysis for learning and AI tasks where data is very high dimensional, and there is little else one can reliably compute at least not without first grappling with the mean.
In the past several years, there has been flurry of thcoretical advancement on this topic, including improved asymptotic bounds (Lee & Valiant 2022 Gupta et al., 20023] Catoni 2011 Gupta et al. 2024 Lugosi & Mendelson 2017). and the development of more robust methods for dealing with adversarially corrupted

---

Published in Transactions on Machine Learning Research (02/2025)
data distributions Lai et al. 2016 Diakonikolas et al. 2017a 2019a Cheng et al. 2019a Dong et al. 2019
Deshmukh et al. 2022). This paper supports this development in two key ways:
1. We provide a large experimental study of many new methods, which had not been thoroughly compared. In the non-corrupted case, with moderate data size we do not see substantial improvement over the classic sample mean approach. However, in the corrupted setting, we find that some methods can significantly improve upon the sample mean. In some cases consistent improvement on the sample mean requires adjustments that we develop. As a summary, the quantum entropy scaling approach of Dong et al. (2019) (using an adjust ment we describe) consistently performs the best as long as inliers are reasonable similar to Gaussian, and often basically matches the mean of the (unknown) inlier data.
2.
We bring to the fore the d > n setting, where there are more dimensions d than data points n, or at least we do not have n as substantially larger than d. This setting is becoming more common as dimensionality grows, but has not typically been considered because the theoretical advancements did not provide exciting new bounds here. In this setting, we revisit some algorithmic derivations and empirically explore what is possible. In particular, we revisc common and critical outlier pruning step, and the key adjustment is ultimately simple: a vd/n term, which vanishes when n >>>> d, needs to be included in a key threshold. This is detailed in Section 3.1
Our experimental study considers mean estimation in a varicty of settings, focusing on when < d or n is not much larger than d. While other experimental studies have been done, many like in Diakonikolar et al. (2017a) provided a comparison in the n >> d case. And while Deshmukh et al. (2022) has some experiments with n not much larger than d, these are not nearly as comprchensive as our study. First, we consider standard Gaussian data with known covariance, and no corruption. Then we extend this to the setting with various types of adversarial corruption. We also consider some limited cases with unknown covariance. However, because straight- forrard adaptations of mnta-estimatiinn approaches towards estimating covariance (mapping to a (2) -dimensional problem) further stresses the need for data size as a function of d, we defer a thorough exploration of this challenge to future work. Finally, we consider real world data scenarios where data is generated via embeddings resulting from large language models, deep pretrained image models, and word embedding models; here we do not have direct enforcement of Gaussianity of the data, but desire a high-dimensional mean nonetheless. In all cases, we consider a wide variety of efficient mean estimation approaches, including both classical ones and modern ones with stronger guarantees in the large setting. We provide an anonymous link to our code for easy reproducability here: hhttps://gitee com/ cullena20/ RobustMeanfatinatiion
2 Background
We consider as input set X C Rd of n samples from an unknown distribution, and the goal is to estimate the mean of that distribution. Consider first the case where the distribution is the Gaussian 2) where I is the identity matrix representing an isotropic covariance. For x ~ Na I) we have [[]]=]]]] = d. For the sample mean ă E Rd from n points drawn iid from Na I) we have 6 [[] = d/n and more importantly it strongly concentrates as Pr[ d/nl > < 2 exp(-Ctt) for a constant C' (Vershynin 2011).. This implies for n = ((d/²)) we have ||ж|| < E with high probability; but for d > n we do not get useful concentration results. The Gaussian is the most studied and used distribution for many reasons including that it has Normal marginals for any dimension, is casy to sample from, models an & loss, and is the limiting distribution of the central limit theorem. As such, it is our main object of study. However, we note that other distributions have distinct behavior for the large d setting. For instance, for n samples from a distribution with mean μ and covariance Σ, the expected squared deviation from the mcan in d dimensions can be bounded by Tr(E)/n / (c.f., (Lee & Valiant 2022)). This implies for instance if X drawn uniformly from unit ball (so Tr(E)) 1) or other distributions with bounded Tr(E), then the behavior for d >> n can
key notation
n
# data samples
# dimensions
error bound
n
true corruption
expected corruption
2

---

Published in Transactions on Machine Learning Research (02/2025)
still be well-concentrated. On the other hand, other unbounded and heavy-tailed distributions where, like Gaussians, Tr(E)) = (d)) present similar challenges in the d > n setting.
Corrupted data models. Another selting considers some fraction 1) E (0. 2) of the data to be adversarially corrupted from X (Huber 1964 Diakonikolas & Kane 2023). Under the Huber model, we draw data X ~ (1-))))++ where P is the set of inliers with mean j (we consider P -a(p,, 1) as identity covariance Gaussian data), and Q is any adversarial outlier distribution. The stronger total variation corruption model first draws X" ~ P (with mean (), and then creates X by adversarially changing any 7-fraction of X' to a new location. That is, it can also adversarially subtract data from the inlier data in addition to adding outliers. How accurately can we recover the mean j under these settings? We mostly focus on the Huber model, and observe that subtractive corruption (a component of the stronger total variation model) can induce a consistent and hard to avoid error, and does not scem to expose significant differences between approaches.
As the mean minimizes the sum of squared deviations, the sample mean is very susceptible to outliers. A single point of corruption can arbitrarily affect the sample mcan. On the other hand, such corruption can be easily detected by filtering out the furthest points from the sample mean, and recomputing the sample mean on the remainder of the data. A more challenging setting relocates points to roughly vd from the mean, where the inliers are, but all in a tight cluster; then no individual points can be so easily filtered, but the sample mean can be given a non-trivial bias of as much as ((n/d). We will empirically consider variety of challenging n-corruption situations.
For many years, when dealing with high dimensions, practitioners were faced with either potentially large error (e.g., on order of nydd) in using the sample mean or other generalizations of the median (Small 1990). or one could spend time exponential in d and return an estimator that is guaranteed to be close to the true mcan (Tukey) 1975) (or c.f., (Chen et al. 2015 Zhu et al. 2020a)). Around 2016, two papers broke this barrier (Lai et al. 2016) and (Diakonikolas el al. 2019a). They considered X ~ Na(uu, I), and allowed an y fraction of the data to be corrupted and return an estimate of the mean á so that - S log I/n) or I On//logd). These works however assume (dd/1")) otherwise one runs into the roadblock that even the sample mean of the inliers (the uncorrupted points) has more than 1J error. Since then, much follow-up work has furthered our understanding. Some work (Dong et al. 2019 Cheng et al. 2019a Depersin & Lecué, 2019) improved the time complexity of robust mean estimation algorithms, and our understanding of the problem's hardness [Diakonikolas et al., 2017c; Hopkins & Li 2019). Others provide formulations where gradient descent can be used despite non-convexity (Cheng el al. 2020 Zhu et al. 2020b). There has also been effort to improve other robust statistics tasks such as covariance estimation (Chen et al. 2015; 2017 Cheng et al. 2019b), sparse estimation (Balakrishnan et al.) 2017 Diakonikolas et al. 2019c Cheng et al. Diakonikolas et al,
2022 Diakonikolas et al.) 2022
2024). list decodable learning (Charikar et al., 2017
2017b), robustly learning mixtures of Gaussians (Bakshi et al. 2022), robust optimization (Diakonikolas et al.] 2019b Prasad et al.
2018 robust regression (Diakonikolas et al. 2018 Klivans et al.) 2020), , or in the conlext of adversarial machine learning (Tran el al. 2018). Impprtantly, robust. statisties are more amenable to differential privacy, in particular to privacy through noise addition, and privacy mechanisms are naturally robust (Dwork Loi 2009. Liu et al., 2021a Hopkins et al.] 2023 Asi ct al.] 2023). Recent
work has also expanded methods for dillerent corruptions models (Liu et al. 2021b Zhu et al.] 2020c For more thorough review see the recent textbook by Diakonikolas & Kane 2023
There has also been significant complementary work in mean estimation under heavy-tailed distributions (Lu-
gosi& & Mendelson 2021: Lugosi 2022 Gupla el al., 2024 Caloni 2011 Lugosi & Mendelson 2017 Devroye
et al. 2015 Lee & Valiant 2022]; see the recent survey by Lugosi & Mendelson (2019) Recent work has
also developed connections between optimality under heavy-tailed distributions, and optimality in the Huber
corruption setting (Prasad al. 2019)
We use standard asymptotic notation so for some constants C1, C2, Cs and functions f.g then g(z) O(f(z)) implics Va C3 then pfxr) 5 +)f()) gfz) = (x)())) implies Va C' then g(x)) 2 Cif(x)+ C2. with possibly different constants; and g(x) = @(f(x)) implies g(z) O(f) and g(x)= (((()).
3

---

Published in Transactions on Machine Learning Research (02/2025)
3 Mean Estimation Algorithms
Here we will document the mean estimation algorithms considered in this paper. Some are classic, and we also include several ones from the recent literature designed to be potentially practical and algorithmically efficient. Some include asymptotic theoretical bounds which use astronomical constants; we make a best effort to replace them with reasonable values so they remain practical. Some use an expected corruption parameter meant to be an upper bound true corruption, 1/- The ones we consider are as follows:
sample_mean The sample mean simply returns j = 지ें Σmex x.
The coordinate- wordinate-wise median computes the median of each coordinate individually so Ps = median((z1,) 24 EX)).
goord_trimmed__ee mean: First compule a trimmed meon eslimator for each coordinate individually, param- eterized by value (0, 1). That is, in one dimension, it sorts the data, and removes r|X] points which have the smallest values, and also removes r|X] with largest values. Then it computes the mean of the remaining (1 - [ points. The cordinate-wise trimmed mean applies this estimator separately for each coordinate; which points are removed in coordinate j have no bearing on which points are removed from coordinate j' (Lugosi & Mendelson 2021
median_oof_mann Split the data into & chunks, find the mean of each chunk, take the coordinate wise median of these k means (Lugosi & Mendelson, 2019 Minsker 2023b al. As default, we set k = 10; this
hyperparameter is explored in Appendix A.5
geometric_median The geomelric median is the point which minimizes the sum of distances to all sample points. This is iteratively approximated using the Weiszfeld algorithm (Small 1990 Vardi & Zhang 2001
lee bre_valiant- (Lee & Valiant (0022 The Lee and Valiant algorithm first estimates the mean j on a y percentage of data points X, using a mean estimator. It then centers all points to X' = f='===n'² x E X). Let X, be the points in X so their corresponding x' have the largest norm. Let X,, be the subset consisting of x' E X' with their corresponding points not in X,, than the extremely large constants in the original paper, we set y = 0.5 and = T|X|. As default, we use median _aaiinoff___eee of_menss; cstimator with k = 10 to obtain the initial mcan estimator p'
in X,. Then return je' + 1
[X] Erex; x'. Rather
LRV: [Lai al. (0166) The LRV method recursively reduces the dimension by half, until 2 dimensions remain. Following the original author's code in the (S d)-diemnso base case, it returns @oord_mdiian. The recursive step has three components. First, it calculates weight w; for each point
w; = exp(-Ila;;- al /l3//(C²²) where s² is a robust sample estimate of the trace of the true covariance matrix, is a rough estimator of the mean chosen as _erdinn and C is a hyperparameter. We use C = 1; this hyper parameter is explored in Appendix A.5 Second, it computes pnu = + Σx,ex W;K,, which is the weighted M which is the weighted covariance of the input. Let V by the span of the top [d/2] singular vectors of Ew; let V1 be the span of the bottom [d/2] singular vectors of En. Third, recurse on data projected onto V, and return an estimate μr. We also build an estimator #2 of the data projected onto the [d/2] dimeeniinll remainder space V1 using the weighted p2 = 1x Ex! EX, waa where X is the data projected onto Vi.
[X]
mean of the input, and En 지 Em,ex wi(x; - pw)
sample mean projected onto Vi: that is
Finally return #1 122
ev w_filtering: (Diakonikolas et al. 20119 b)) This method observes that when inliers are from standard Gaussian, then set of corrupted data which substantially affects the mean estimate must result in sufficiently large top cigenvalue after centering (i.c., of the sample covariance matrix), and this can be remedied by pruning points which are far along the top eigenvector. In this method, if after centering by the sample mean Å the top eigenvalue exceeds O(T log 1/r) (Diakonikolas et al. 2017a iimplements this as + 37 log(1/T)), then this data is considered additively corrupted along the direction of the top eigenvector. We call this the corruption detection step. Then they consider all points projected onlo the associated top eigenvector and sorted P = (pi,..., Pa)); and then a set of points furthest from the median med(P) P)) are
T httee//sstttllcocorrrraaasaattet http:///ghhul.ccc/hhooon/rrrbbrs
4

---

Published in Transactions on Machine Learning Research (02/2025)
pruned. We call this the pruning step. The determination of which points to prine is based on those which exceed a Gaussian concentration inequality. Specifically, it finds the smallest index i so T; = P; med P) 2r where erfc is the complementary error function (1- the
satisfies "= > /erff(())/ /22/2 (022)2++/(0/ log(dd////////////////////////// odf of the Normal) and prunes all points or larger. Intuitively, the centered projected data is expected to be a standard Normal distribution, and this bound compares the true percentage of points that exceed a threshold, T; with the probability that points will exceed that threshold, given by erfc with some slack terms added. Then the algorithm is recursively called with all points not-yet pruned until the top eigenvalue threshold is not violated. This algorithm critically assumes identity covariance and n 2(d/r²) > d.
(Dong et al. (2199) Quantum Entropy Scoring, QUE for short, scores outliers based on quantum entropy regularization, and returns a mean using the same structure as ev ev_filtering, but with a modified pruning procedure. Rather than pruning points based on their projection onto the top eigenvalue, points are given outlier scores relevant to all directions. First, calculate the normalized matrix exponential U = (o((a))/r((ex((()))) where a 2 a hyperparamater and E is the sample covariance. Then, calculate a vector of quantum entropy scores, w, with w; = (Iz - ,²)"U((; - n²), where x; is the ith data point and p is the sample mean. This is implemented efficiently using Chebyshev expansion of the matrix exponential and
QUE:
approximations. Points with the largest scores are pruned, and the algorithm continues recursively with the remaining points until the top eigenvalue threshold is not violated. Following the original author's code (Dong et al., 2 we prune 7/2 percentage of points during every iteration. Additionally, while the author's provide a theoretical threshold on the top eigenvalue, the constants are not given. Rather than tuning this threshold, we implement
using the same threshold as ev_filterin;; that is 1 + 3r log T/T. Because of this threshold, the algorithm critically assumes identity covariance and = (dd/r²) >> d. We set 4 as in the author code; simple experiments show little variation with a between 0.5 and 200.
PGD: Cheng et al. (2020)) Projected Gradient Descent, PGD for short, frames robust mean estimation as a non-convex optimization problem, and despite non-convexity, directly solves this using gradicnt descent. PGD finds a vector, w, of outlier scores, which can then be used to return a mean estimate j = |X| Exzex WyZt w is found to minimize the spectral norm of the standard weighted covariance matrix, Ew, subject to the constraint that the weights represent at least a (1-7)-density fractional subset of the dataset. The vector w is found as an approximate stationary point to this objective by first performing gradient descent on the spectral norm of the weighted covariance matrix, and then projecting onto the simplex of feasible weight vectors. First, define a function F(u, w) = u u.'E.u. Then, repeat the following for y iterations, where Y, following the conventions of a code implementatinn by the samn author as the original paper (Cheng & Lin 2021 is a hyperparameter. Calculate the top eigenvector, ut, of Ew. which corresponds to finding the unit vector such that F(w, uz) 2 (1 - m),, u). Then, update w as w = P(w aV F(w, uz)) where P projects onto An,27 = fw E R" : ||l = and 0 I w S (1-27)n }, and Fw u,) = Xu @ Xu - ( where @ indicates element- wwse multiplication, and a is the learning rate, initialized as 1/n and updated dynamically through learning. We set the number of iterations = 15; this hyperparameter is explored in Appendix A.5
b min: Deshumukh et al. (2022) This method frames robust mean estimation as a semi-definite program (SDP). Similar to PGD, a vector, w, of outlier scores is found, and the weighted mean p' = { Ex,ex waz, is returned. The Ep norm for hyperparameter 0 < p I 1 is maximized with respect to w, under the constraint that the top eigenvalue of the weighted covariance matrix is less than a constant. The weight vector w is iteratively updated by solving a SDP until the number of iterations is less than a bound determined by T, in which case á defined above is returned. Update by approximately solving an SDP to maximize w in ||l]| over A Each step of the optimization problem is convex and can be solved as the following packing SDP:
Tefef"
Inxx
max
s.t. w. 2 0 Vi,
Σ
wi
(x Hw) - pT) 3
c,nlaxd]
where, Cr is function of T. This analysis of this algorithm critically assuines identity covariance and n = ((d/72) > d.
5

---

Published in Transactions on Machine Learning Research (02/2025)
3.1 New Algorithms and Variants
We also consider a few new methods, with subtle but important extensions of these existing ones.
The primary insight needed to adapt methods to the d 2 n case is found by revisiting how we identify outliers with respect to a d-dimensional Gaussian distribution. The bounds used in the n > d case have enough data in each direction d to concentrate, whereas the d 2 n case we need to account. for this additional variance. The key result leverages a theorem of Vershynin (2011) to understand the concentration of the top eigenvalue of the sample covariance matrix.
Theorem 1. Let X be x d matria whose entries are independently drawn from Nu,, 1). Let E 1 1(X p)) p) be the sample covoriance matric of X, where j = E, and X, is the ith row of X. Then for every > 0, with probability of at least 1 - 3exp( 12),,, one has
(1+ V+Vd/++t/vm Vd+ 2dt 12 2
|E S 1+
n
The proof is deferred to Appendix A.1 and can be absorbed into the probability of failure.
A more convenient form shows that the fourth term is lower-order
Corollary 1.1. Under the same setting as Theorem 1 if one assumes d/n S 16, 2 16, t 2 5, then with
probability of at least 1 - 1-3exp( -expp((( (t)),, one has
E0 - (1+vd/n+t/vi)
ev ww_filterinnglow The ev w_filtering algorithm assumes the sample size is n= ((d/72). This assumption is used in several parts of the analysis, and it allows the filtering bound to be simplified to 1 + r log((/T)) however, when n = o(d/>"), this simplification does not hold, and the filtering bound needs to depend on d.
We instead filter points the top eigenvalue Amax > (1 + vd/n + +t/ v/2 using Corollary 1.1 We set t to achieve almost 100% (s= 0.999) success. All other steps of the algorithu remain the same.
algorithm extends the same filtering bound ev filtering. Although their paper mentions Olvd/n factor in the error, the code seems to assume n = ((d/e²) and is implemented very similar to ev ev_iitterng.. As this approach does not work under low data size, in our newly proposed variant, we instead filter points if the top eigenvalue Amax > (1+ /d/n + /n) using Corollary 1.1
QUE_low_n
The
b f__min_oo__ The mp_ algorithm uses the condition that the top eigenvalue of the weighted covariance matrix is bounded by c,n where G, is a hyperparameter suggested to be set at log(/T). As previously observed, this threshold does not hold when o(d/72) and to account for this, in our newly proposed variant we set C, (1 vd/n + +t/vn)², using Corollary 1.1
lee s____aliatt_simpll We use simplified version of the Lee and Valiant algorithm (Lee & Valiant
2022
which aligns with an informal description in their abst Tt completely removes the 7 percentage of points classified as outliers rather than simply downweighting them. That is, it returns P = І Ewex; x; the average of all points X! which were not in the original estimate, nor from the pruned set furthest from μt.
4 Experiments
We generally evaluate the performance of these mean estimation algorithms as data size n, dimension d, and corruption are varied. Error is measured as the Euclidean distance I| [ull All between the true mcan μ and the estimate f returned by a mean estimation algorithu. We set the default values as n = 500, d = 500, and 7 = 0.1. We examine the performance as we fix one of these variables and vary the others under various distributions for both the uncorrupted and corrupted data. We first examine uncorrupted standard normal Gaussian data, demonstrating that nothing really improves upon sample_ mean, and observing the robustness of mean estimation techniques when applied to uncorrupted data. We then examine corrupted Gaussian data
6

---

Published in Transactions on Machine Learning Research (02/2025)
over various covariances and noise distributions. The example distributions are chosen among challenging examples in the literature meant. to distinguish various models. Experiments were run on 2022 Macbook Air with Apple M2 Chip, 16GB memory, running MacOS 12.
At one point in our experiment, the values n, d, and are used to generate data according to a supplied data generation function and noise scheme (both of which will vary depending on the experiment). A mean estimate is made on this data using each of the mean estimators being tested. For each mean estimator, error is then stored as the Euclidean distance between the true mean of the data and the returned mean estimate. These errors are accumulated over 5 runs and averaged. We additionally plot the error incurred by the sample mcan of the original uncorrupted data, which we call the serves as a valuable baseline for comparison. In practice, we can not expect to achieve error better than the sample mean of the inliers. Therefore, a reasonable goal for a robust estimator is to closely match the performance of godd_sampee_ment thereby removing the effects of corrupted data points. Could a robust mean estimator somehow improve upon this? We do not observe this; but we will observe methods that basically match _as___smpl_______ even without O(d/E2).
mean error. This
Fraction of corrupted data. Some algorithuss are designed for data where a ryfraction of the data has been corrupted. And some cases, this fraction is taken as a parameter т used within the algorithms booodd_trimmed_meee lee dee_valiant_simple lee_ valiant, ev ww_fitternng, ev __fittrring_ow__ QUE, QUE_low__ PGD, min, 6,,,,,,-2-0 m__min_low_n)
In theory, these algorithms work best using their parameter T set to the true fraction of corrupted data and may oven result in arbitrary error if the paramcter 7 is not set to at least an upper bound for the truc fraction. However, increasing the value of r in the algorithms also theoretically increases the error incurred by algorithms. Recent work by Jain et al. (2022) showed a meta algorithm that allows robust estimation algorithms to perform asymptotically optimal without knowing true corruption 7). We investigate robustness to expected corruption, т, empirically, in Appendix A.6 as fix and vary true corruption r7. We observe that the best algorithms do not show a strong dependence on this relationship, so long as is an upper bound on n. Hence, for all other experiments, we simply set the parameter according to the truc corrupted fraction or to r = 0.1 if the data is not corrupted.
Selecting algorithmic variants. There are many algorithms to be considered, and plots can become cluttered. To reduce this, we perform some comparison among variants. We summarize key findings here, with further details deferred to Section 7
n 500, d = 500
n 200, d d= 500
Algorithm
Error
Time (s)
Error
Time (s)
sample_me mean
2.47 + 0.04
0.00019 + 0.000002
2.74 + 0.05
0.00019 + 0.000001
LRV
1.14 + 0.04
0.81 0.12
1.76 = 0.10
0.64 0.03
PGD
1.08 + 0.02
82.4 + 8.8
1.68 + 0.05
72.5 + 3.2
filteringg__oooe low
1.07 + 0.02
0.20 + 0.02
1.69 + 0.04
0.08 + 0.03
filtering
13.49 + 3.56
0.48 0.15
17.06 = 5.92
0.05 + 0.02
QUE_low.
1.04 + 0.03
0.71 = 0.05
1.70 + 0.049
0.35 + 0.03
QUE
20.81 + 0.40
2.70 + 0.08
20.88 + 0.38
1.99 + 0.04
min mimiminnwnnn
1.17 + 0.04
1182.6 + 35.3
1.67 + 0.03
265.9 + 15.2
min
1.62 + 0.04
1076.8 + 43.9
5.62 + 0.40
250.07 + 17.2
Table 1: Error and Runtime Across Simple Corrupted Identity Covariance Gaussian
First, we do not consider mi_ and mo__mi_low___ in our plots due their exceptionally large runtimes. We report runtimes and errors (defined as the Euclidean distance from the estimated mean to the true mean) of selected algorithms under n = 500 and d = 500 and under n = 200 and 500 over a simple corrupted Gaussian distribution in Table D We report results as the mcan + the standard deviation, averaged over runs. With the notable exceptions of fp_min__o__ low_low_nn n, fp_min, and PGD, most estimators are efficient and took under 3 seconds to with 500 and d = 500 for simple corrupted Gaussian distribution. min
7

---

Published in Transactions on Machine Learning Research (02/2025)
and fi_min___ loww___w_______________ rely on an SDP solver, which we implement with the evxpy DDimmond & Boyd 2016 Agrawal et al. 2018) package and the mosek solver. Although this is theoretically efficient, it is slow in practice for the data scale and dimesionality we consider in this paper. For n = 500 and d = 500, both algorithms took about 1100 seconds, or about 18 minutes, to return a mcan estimate over a simple corrupted data scheme. For that. reason, and since we run many trials of each input size and error level, we do not consider these algorithms our plots. However, we note that employing Corollary 1.1 for fp_min_low__ achieves a noticeable performance increase over minm showing gains from 1.62 error to 1.17 error in the 500, d = 500 case and from 5.62 to 1.67 error in the 200, d = 500 case. PGD is also much slower than other robust estimators, taking approximately 80 seconds to run with n = 500 and d = 500. While this significant slow down is relevant when considering practical algorithm, it is not as prohibitive as f_min. As a result, we include it in all of our plots.
Second, we observe that when the data does not satisfy that n >> d, then both ev w_filtering and QUE can have catastrophic failure. Our variants ev w_filtering__ow d RJ n settings, while basically matching the effectiveness of their original versions when they do not have catastrophic failure. This result is highlighted in Table I where we observe that both QUE and ev o_filteriing achieve significantly worse error than QUE_low_n and ev filtteringglow_n respectively. As a result, we use ev w_ffltering_low_n and QUE_low_n in all comparisons.
and QUE_low_n avoid this issue in the d > n and
Thirdly, we find that ree__aliant_simpe performs slightly better than the original lee__valeeee valiant; however the difference is fairly small. We also do not notice any meaningful advantages from using vaaiaattliimaaaaaaa or vel_aat with different choices of initial mean estimators. As such, we only use lee vaaliantlsimpaaa in all comparisons.
4.1 Uncorrupted Gaussian Data with Identity Covariance
We first evaluate the performance of mean estimation algorithms over uncorrupted Gaussian data with identity covariance. In particular, we draw uncorrupted data X Na(u, T), where μ is an arbitrary mean and I is identity covariance. For these experiments, set μ to be the all-fives vector, but did not find performance to depend on μ.. For algorithuns that utilize т, expected corruption, as input, we use the default value of r == 0.1.
We provide our first experimental plots in Figure D most further experiments will follow this same sct-up, consisting of set of charts, each measuring the Error || - full on the y-axis. The top two charts vary the data size n along the z-axis, but on different scales. The top left shows a large scale from n 20 to 5020, focusing on the d 500 paradigm. The top right shows < d paradigın. The bottom left plot show the effect of varying the dimension from d = 20 to d = 1020 while fixing n = 500. The bottom right shows varying the algorithm's parameter, for the expected noise from 0 to 0.45 with fixed n = 500, d = 500. Each algorithm is shown as a curve, with the average error of independent data generations at regular intervals on the a-axis. A shaded area is shown at a radius of 1 standard deviation from that average error value.
20 to n 520, focusing
the
The plots are bit cluttered because most algorithms perform about the same, including sample_ _ean. No algorithm can be seen to noticeably outperform sample_mean, which, as the MLE for this data, and by the Gauss-Markov theorem, is not surprising. Methods LRV, ___fitteringllo__. QUE_low_n, PGD, coord_trimmed__nee mean, geometric_mediant and se__allaantsimple have about the same error in most cases. However, median_of_meande and coord coord_median perform slightly worse, with the gap becoming more apparent in high dimensions. Moreover, lee__ pooliant_simplle and coord_timmmed_met do significantly worse with a higher expected corruption parameter т. This result of expected corruption, т, being a hyperparamcter that directly controls the percentage of points to prune. Finally, as predicted by basic theory, with n fixed the dimension d increases, the measured error increases at rate roughly Vd.
4.2 Corrupted Gaussian Data with Identity Covariance
We evaluate corrupting noise added to Gaussian data with identity covariance. In particular, we draw X ~ P + 11Q where P= Na(u,, ( I) and Q describes the corrupted data distribution. This is equivalent
8

---

Published in Transactions on Machine Learning Research (02/2025)
Erfor Dota Sice Dimensions, 500, Comuption Parcentage. 0.10
Errer vs Dota Size
Dimensions, 500, Corruption Perzentage. 0.10
sampk mean enoor
med Errr
geometnic medion Encr
See satani sample Linrr
irdiin of means emor
by Frier
ev. bening low.n Enor
low Erter
103C
2000
3000
403C
3000
300
200
300
400
300
Dute Sizz
Dull Stat
Error us Dimensions
Enarr vs Expected Corruction Data 500, Dir ensions.n 500
Data 500, Comuption Persentage.
18
1.30
3.25
14-
E11
1.0
3.15
1.10
0.6
1.00
022
0.35
0.0
Dimensions
Expected Comuption
Figure 1: Uncorrupted Gaussian Identity Covariance
to the more general case where any covariance E is known, as we could simply scale the data to have identity covariance, apply these methods, and scale the mean estimate back. We provide a wrapper in our implementation to perform this operation.
Gaussian noise shifted to variance shell. We first consider corrupted data distribution Q= --NNall 1
so || -/|l = vd. Since Eg<pllæ - d, corrupted data from Q is not casily identified. The location of this cluster is determined by a random rotation at every generation to ensure that no coordinate-axis specific bias introduced. This is shown Figure 2 in the same experiments as with uncorrupted data, except now the bottom right figure varies 7, the fraction of corrupted data from Q, along the x-axis. We set the expected corruption hyperparameter equal to true corruption, that is n.. In Appendix A.6 we explore the relation between expected corruption versus actual corruption y; for the most part as long as >> 1).
There is now more clear separation between the algorithms designed for adversarial corruption, and those not. Here ev ww_filterrnng_ow.o the best, even appearing better than srod__sapple_ee @emple_mean mean for large dimensions, although within standard deviation error margin. Due to the high dimensionality, d, good_sample _ean, the sample mcan of points from the uncorrupted part of the distribution P, does not have error approaching until n is very large. ev_fitterigg_lw QUE_low_n, and PGD work so well that they are nearly overlapping this best possible standard. Also, perhaps surprisingly, median_of of means also does nearly as well, especially under larger though it degrades much worse with larger 1).
QUE_low and PGD do the best among all settings, with LRV, perhaps doing
In contrast, coord_median, sample_mean coord_trimmed__eal geometric_median, and lee_valiant_simple all do considerably worsc, even with large data size. With large corruption levels, these all even seem do worse than just sample_mean, indicating that the algorithms prune the wrong data points or face some other similar issue.
9

---

Published in Transactions on Machine Learning Research (02/2025)
Erfor Dota Sice Dimensions, 500, Comuption Parcentage. 0.10
Errer vs Dota Size
Dimensions, 500, Corruption Perzentage. 0.10
sampp naan enoor
med Erre
gesmetnic Encr
See satani, sample
iredio means enor
by Frier
ev beninn lown Enor
low
PITOE
103C
2000
3000
403C
3000
300
200
300
400
360
Dute Sizz
Duts Stat
Error us Dimensions
Error vs Corruption
Data 500, Comuption Persentage.
Data 500, Dinrensions, 500
37.5
35.
35.0
30
22.5
7.5
1.5
5.0
2.5
0.0
0.0
Dimensions
Co Compption
Figure 2: Corrupted Gaussian Identity Covariance: Additive Variance Shell Noise
Large + Subtle outliers: DKK Noise. We now recreate the noise distribution from Diakonikolas el al. (017a), which utilizes a more sophisticated corruption scheme that includes both easier and harder to detect outliers. Half of the noise is drawn from the product distribution over the hypercube where every coordinate is -1 or 0 away from the true mean at that coordinate with equal probability. The other half drawn from the product distribution where the first coordinate is either 11 or -1 away from the true mean at that coordinate with equal probability, the second coordinate is -3 or -1 away from the corresponding true mean coordinate with equal probability, and all remaining coordinates are 1 away from the true mean. We call this corruption scheme DKK Noise. This is shown in Figure 3 with similar results. ev_filtering_lownn, QUE_low_n PGD, and LRV achieve performance nearly matching good_sample_meant with almost as well at least while the dimension and rate of corruption t are on the smaller side. Other than
_mamss also doing
_f_meass,,,, all classic methods perform noticeably worse than good_sample__entt _emmppe__mennn and achieve similar error to sample_mess mean. The only difference of note here is that lee_ tee__aliant_simeee error bars, suggesting that its performance may vary significantly depending random initializations made within the algorithm. Also, LRV may even outperform
exhibits far larger
mean for very large dimensions.
Subtractive noise. We additionally consider subtractive noise in Figure 4 Here, an adversary is able to remove 17 percentage of points from the data distribution. We implement this by removing the 7-percentage of points which most extreme in some direction. Unlike in the additive corruption case, there is strict upper bound on the error under subtractive corruption from standard Gaussian distribution; the error induced is bounded as O(1) even using sample_mean, and clustering the subtracted points as most extreme in some direction ensures their effect 22(7) under sample_mest mean. In general, we wish to consider noise distributions that may add outliers and also remove inliers through such subtractive noise. However, we do not find any surprising capabilities among methods in this scenario. As a result, for the remainder of this paper, focus additive corruption.
10

---

Published in Transactions on Machine Learning Research (02/2025)
Emery vs Data Sizo
Ertor Dota S:ze
Di mensions, 502, Corruption Fercantage. 0.10
Dimensions, 500. Comuption Rarcentage. Percttaen D:10
san parppknnnnnnnnn eroo
goometnic mosian
lae, vallan, simple
. medar medear_o_meen Lmmmars BrT
nteinn Enor by Funr
pg
2-
1000
2000
3300
4000
3000
0
100
200
200
400
200
Dutaa Siae
Duta Sie
Ernor vs Dimonsions
Error VS Corruption
300, Comuption Percentage.
Sizo. DI mennioooss
30
Eror
1.5
9.
Olmeroiins
Compattion
Figure 3: Corrupted Gaussian Identity Covariance: DKK Noise
Under subtractive corruption, nothing outperforms @ample_mean; and now nothing can match good_sample_mean in error in most settings. However, ev_fiitering_low__, QUE_low_n, PGD, and LRV all nearly match the performance of sample_mean. Unlike in the previous additive corruption schemes, median_of__ee performs significantly worse under subtractive corruption, always achieving error notably Among other estimators, geometric_median nearly matches sample_mean error
worse than sample_mee across all settings, lee__ lis___vllat___mmee and woord_trimmed _man perform similarly but degrade much more under larger corruption, while
performs significantly worse
We find similar results across several other noise distributions. In addition to the hard-to-detect distributions, we also show that ev __fffteeringlowe. These details are deferred to Appendix A.2
@UU__lw_nn PGD, and LRV are generally robust to arbitrary outliers.
4.3 Corrupted Gaussian Data with Unknown Covariance
We now evaluate corrupted Gaussian data for general unknown covariance. Since ev wv_fitterin_llw__ and QUE_low_n rely the identity covariance assumption,
employ simple heuristic to adapt these algorithms to the unknown covariance case. We estimate the trace of the covariance as Tr(E) Ele || - where á is the sample mean and E is the sample covariance. We rescale the data to
X x - - 2z/V TY(E) x; E X). We then estimate the mean of X', rescale this estimate by V Tr(E)) and report the results. This heuristic is used for ev_filteringloow_n and QUE_low_ across all unknown covariance experiments, and not used for any other algorithms. The other standard algorithms are either invariant to this linear rescaling, or themselves account for it; this was supported by our own observations.
Another possible method is to utilize robust covariance estimate instead of a sample trace estimate, discussed in Diakonikolas & Kane 2023). We do not evaluate such methods, as this would involve thorough
11

---

Published in Transactions on Machine Learning Research (02/2025)
Enror US Data Size
Erfor vs Data Size
Dimensions, 500, Computtion Porcentage. 0.30
Dimensions, 500, Compption Parcentage. 0.10
simple veaa enor
Dror
çeametnic mrelion
vituan jemple
4
median meeme Enor
lo Frror
et ninningnngg. JoN Error
trisr
Ernor
3-
2-
1200
2000
3000
4000
3000
0
300
200
500
400
300
Data Size
Duta Size
Error vs Dimensions
Error vs Comruption
Sizo. 500, Corraption Parcentage: 0.10
500, Dimans
2.30
175
1.0
2.00
1.5
Eno
Emt
100
2.75
0.75
0.50
125
0.25
Dimerslons
Commatiin
Figure 4: Corrupted Gaussian Identity Covariance: Subtractive Noise
study into robust covariance estimation methods over low data size, which goes beyond the scope of this work. Naively treating robust covariance estimation as robust mean estimation in d² dimensions further exasperates issues related to low data size. We also choose to use a simple sample trace estimate rather than the robust approach proposed by Lai et al. (2016). We find that the approach proposed often results in significant underestimates across difficult noise distributions, causing ev filteringg_oww_rr and QUE_low_ fail catastrophically. We note that these underestimates are potentially more harmful than overestimates through them, even the inlier data may not pass the threshold, causing continuous pruning. While sample trace estimate approach is more prone to overestinates, this can be remedied by naively pruning large outliers. Diakonikolas et al. (2017a) also provides an algorithm for unknown covariance mean estimation similar to ev filtering, but the corruption detection threshold is not easily adapted to the low data size case.
4.3.1 Unknown Spherical Covariance
We evaluate corrupting noise added to Gaussian data with spherical covariance. We draw X ~ (1- (1-1) where P = Na(p, o² and Q describes the additive corrupted data distribution. We consider to be the all-fives vector and o 5.
Gaussian noise shifted to scaled variance shell We adapt the identity covariance noise distribution models by appropriately scaling coordinates by a. We first consider the corrupted data. distribution Q N(n', 11) so || μ' = ovd. With P now having covariance I² E.<plle- =||l = odd, and corrupted data from Q is not easily identified. Results are show in Figure
While the overall error here higher, matching the theory that even for uncorrupted data, the sample mean is expected have error of ((avd//n) the relative performance of algorithms nearly identical to the identity covariance case. ev wilterieinggggn.........r low_n, n, QUE_low_n, and PGD nearly match good_samplll mean error
12

---

Published in Transactions on Machine Learning Research (02/2025)
Error Dota Size
Enerr Dato Size
Dimensions, 500, Comuption Parcentage. 0.10
Dimensions. 500, Comupton Percentage. 0.10
sangle_mein enor
median
goometric
Me valaet, smmple
70-
mediin, means bror
TV Trior
CV. www.enennggg. Iow Enior
12714
low Lirer
Ernor
10-
3000
2000
3000
4000
2000
300
200
300
400
200
Dute Sizz
Dala Siee
Error us Dimensions
Error vs Corruption
Comuption Persentage:
Dimansions.
16
80
12
60
Emor
Comuption
Figure 5: Corrupted Gaussian Large Spherical Covariance: Additive Variance Shell Noise
throughout, with LRV performing only slightly worse. median_of__nteee _e__s lags behind both estimators but still performs noticeably better than sample_ _eree__een.___ However, lee lee__eveee___e____e____eee valiant valiant_simple performs much better in this sccnario, nearly exactly matching good_sample_ment except with large cnough corruption - where with 7 = 0.45, it and median sfroaoof_______ _f__means probably confuse which points are inliers and have much worse error. Other methods perform similarly to sample_mest _een or worse.
As in the identity covariance case, find similar results across noise distributions. The only notable exception is for w__filterrngg_ow which sometimes performs slightly worse and doesn't always converge to good_sample_ment error as increases, probably due to instabilities in the trace scaling heuristic. We additionally show that relative performance of algorithms is mostly independent of the choice of a. The exception to this LRV, which notably outperforms all other methods, including good_sample_mean with large enough a across noise distributions. These details are deferred to Appendix A.3
Unknown Non-Spherical Covariance In Appendix A.4 we also explore the unknown, non-spherical covariance case. This is even more sensitive to the covariance estimate, and so is further outside the pri- mary scope of this study, Nonetheless, we continue to observe that the best robust estimators, including QUE_low_n, continue to perform well.
5 Large Language Model Experiment
To evaluate whether robust mean estimation methods are overly sensitive to distributional assumptions, we evaluate performance over real world data. We first study the problem of estimating the mean of vectors from language models. Such "word embeddings" have had an enormous impact on natural language processing, starting from simple sparse fee-freqyency vectors (Robertson al. (2009). Second generation
word embeddings (e.g., GloVE (Pennington et al.] 2014) and word2vec (Mikolov et al. 2013)) made the
13

---

Published in Transactions on Machine Learning Research (02/2025)
advancement of creating "low" dimensional vector (about 300 dimensions) for each word, where Euclidean (and cosine) distances could be used as a proxy for the similarity between words based on how they are used. As a serendipitous side-effect, structure emerged where dot-products, means, and linear classifiers made sense in this embodding space (Bolukbasi et al., 2016; Dev & Phillips] 2019). Third generation embeddings created representations for each word in the context of the nearby words; that is, each use of a word had a different embedding. These were a first main use of transformer architectures, and implicitly capture more meaning and context with progressive layers of a neural network. As is most common, we use the last layer of the embedding network the representation of word. Our first study, shown next, uses these third generation word embeddings. Further experiments over deep pretrained image model embeddings and second generation word embeddings are deferred to Appendix A.7 and Appendix A.S. respectively.
We first examine performance over third generation embeddings of a homonym word where all instances correspond to the same meaning. We then examine performance where embeddings corresponding one mcaning of a word are corrupted by embeddings corresponding to another meaning of the same word. Effectively calculating this mean may be important to many downstream tasks (e.g., topic modeling GGrifliths & Steyvers 2004 Blei & Lafferty 2009), bias estimation and attenuation (Bolukbasi et al. 2016 Dev & Phillips 2019 This models a rcalistic form of corruption that may arisc within LLM embedded data.
We build a dataset of 400 sentences that use the word "field" corresponding to the following definition: 'an area of open land, especially one planted with crops or pasture, typically bounded by hedges or fences". We build another dataset of 400 sentences that use the word 'field" corresponding to the following, alternate definition: "a particular branch of study or sphere of activity or interest'. From now on, we refer these as 'fields of land" and "fields of study'. We generated these sentences using ChatGPT-4o. For more details how we generate this dataset, and the exact sentences used, see Appendix AA.1 We embed these sentences and extract the in-context embeddings for the word 'field" using 4 LLMs of varying embedding dimensions: MiniLM (Wang et al. 2020), T5 (Raffel et al.) 2023). BERT (Devlin et al. 2019), and ALBERT (Lan et al., 2020). MiniLM has an embedding dimension of 384, T5 has one of 512, BERT and ALBERT have embedding dimensions of 768. We choose these 4 LLMs to sample a variely of models across different dimensionalities.
5.1 Common Definition Embeddings
We first consider performance over embeddings corresponding to the same definition. This is analogous to the uncorrupted data case, As an error metric, we use Leave One Out Cross Validaton (LOOCV). We only average over the bottom 90% of errors to account for potential bias introduced by words that less clearly belong to specific category. LOOCV error is defined here as w En ||estimato((X-))- -xx-,))),lll -zyl where n' = 0.9n is 90% of the number of data points in the dataset X (those with smallest errors), x; is the ith data point in X, and X-; is X excluding z;. LOOCV under the sample mcan represents a valuable bascline for comparison as it demonstrates the minimum error to be expected across this data set under the 10% expected corruption (T- = 0.1) modeled by the algorithms. We take the dataset of 400 sentences corresponding to the 'field of land' definition. We vary data size from = 10 to n 400, and, in prior experiments, average results over 5 runs and report shaded regions to denote 1 standard deviation of error. For algorithms that utilize T, expected corruption, as input, we use the default value of r = 0.1. We employ the sample trace scaling heuristic for ev_ _fllteiing_owww
and QUE_low_r We additionally halt QUE_low_n whenever more than 27 percentage of the data has been pruned, regardless of whether or not the threshold is passed. We note that the early halting heuristic necessary for QUE_low__ to perform well under this setting and that it does not mcaningfully improve ev ev_ u=_flttering_oww low__ this further explored in Scction 5.3
We show results in Figure
Our results do not match our synthetic experiments, suggesting that some robust mean estimation algorithms are sensitive to (Gaussian) distributional assumptions, at least under small data sizc. Across LLMs, no algo- rithm siguificantly beats the error of sample_meess _eaa. Moreover, ev l_lltterinn_oow_are performs significantly worse than sample_mean _eaeee_meeeea_ over all embeddings. This is unsurprising due to the sensitivity of knowledge of the true covariance. Despite having a similar dependency to knowledge of the true covari- ance, QUE_low_n achieves performance nearly matching sample_mes _maee____an_____ error throughout. We also find that LRV performs meaningfully worse than sample_ mean across all LLMs except MiniLM, though not quite
to
14

---

Published in Transactions on Machine Learning Research (02/2025)
LOOCV Error vs Data Size
LOOCV Error vs Data Size
Dimensions: 384, Expected Corruption: 0.10
Dimensions: 512, Expected Corruption: 0.10
samplo mean
median
2.6
coord trimmed mean
median
2.5
lce simple
17
LOOCV Entor
LOOCV Emro
filtering_low
que low_n
2.3
1.6
2.1
100
150
200
100
300
350
400
(a) MiniLM
(b) T5
LOOCV
Dimensions: Expected Corruption:
Dimensions: 768, Expected Corruption:
12.5
LOOCV trror
LOOCV Ener
12.0
8.5
11.5
8.0
10.5
7.5
7.0
50
100
150
200
250
300
350
400
0
50
100
150
200
250
300
350
400
Data Size
Data Size
(c) BERT
(d) ALBERT
Figure 6: LOOCV Error on "Field of Land" Embeddings
as catostrophically as ev __lfffeerinllw_nnn Aside from coord_median, which performs noticeably worse than sample_meat mean over all LLMs besides MiniLM, all other estimators perform similarly to sample_mean.
5.2 Corrupted Embeddings
We examine performance over corrupted embeddings of the word "field". We draw corrupted data X - ( 112, where the inlier distribution, P, consists of embeddings of the word "field" corresponding the word "field"
to the 'field of land' definition, and the outlier distribution, Q, consists of embeddings corresponding to the "field of study" definition. with previous experiments, we measure the Error || Al on the g-axis, taking μ as the mean of all 400 "field of land' embeddings, and á the estimate returned by mean estimation algorithm. We measure Error vs 7), vary 7) from to 0.45, and always have n 400. We average results over 5 runs and report shaded regions to represent 1 standard deviation of error. is plotted to represent the mean of the data before corruption. These results are shown
in Figure
We find that mean estimation algorithms can indeed significantly improve performance on this real- world task, but do not observe the same trends as our synthetic data experiments. QUE_low_n and lee s___vllant__mmmpe are the best estimators, with both significantly outperforming sample_ment In fact, QUE_low performs nearly identical to good_ssapll mean across all I.LMs,, and lee. paliant_simpll per-
15

---

Published in Transactions on Machine Learning Research (02/2025)
Error vs Corruption
Error vs Corruttion
Data Size: 400, Dimensions: 384
Data Size: 400, Dimensions: 512
sample_mm mean
coord trimmed mean
2.0
geometric_
lee see_vallant_smpe
IV
1.5
w_fltering__ow l
que que_low_n Jow_n
Pgd
1.0
good sampls mean
1.0
0.5
0.5
0.0
0.0
0.0
0.1
0.3
0.4
Corruption
Corruption
(a) MiniLM
(b) T5
Corruption
Corruption
Dimensions:
14-
12-
M
10
M
5
0-
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2 Corruption
0.3
0.4
Corruption
(c) BERT
(d) ALBERT
Figure 7: Error on "Field of Land" Embeddings Corrupted with 'Field of Study" Emneddings
forms similarly, except on MiniT.M, where degrades worse with larger 7) but stiill outperforms sample_mean. The performance of lee lee_valiant_simpl supports the observation that it is a more effective naive pruning method, which happens to work among the best in these experiments. Moreover, very elfectively here, always significantly outperforming sample_mean, However, neither ev ww_filtering_low nor LRV perform effectively. LRV outperforms sample_mme with large enough 77, but these results are not consistent across LLMs suggesting sensitivity to distributional assumptions. Additionally, it almost never outperforms median _eatian_of_mass,,,,___ means, lee_ valiiaat_iimppaoo or QUE_low__ and achieves much worse error with lower 7. This may suggest that LRV finds irregularities the uncorrupted data, causing to return a mean signifi- cantly different from good_sample _ean. While this could be beneficial, suggesting that good pood__smppeee_ mean isn't the best error metric, this is not supported by the 'uncorrupted" LOOCV error results, where LRV also generally results in degraded LOOCV error. ev_filtterngglow__ fails catastrophically across LLMs. This matches the 'uncorrupted" LOOCV error results, supporting the observation that ev ww_filtering_low_1 may fail calostrophically without sufficient knowledge of the true covariance mairix. As in the 'uncorrupted" LOOCV error results, there is the somewhat surprising observation that despite seemingly having a similar dependency on knowledge of the true covariance matrix to ev __lltteringlowwnnn QUE_low_r performs nearly optimally here. This suggests the superiority the quantum entropy based scoring method over naively ranking outliers based the top eigenvalue of the sample covariance.
performs
16

---

Published in Transactions on Machine Learning Research (02/2025)
LOOCV Error vs Data Size
LOOCV Error vs Data Size
Dimensions: 384, Expected Corruption: 0.10
Dimensions: 768, Expected Corruption: 0.10
sample_ mean
3.0
w__iitering_ha halt
ov. hitcring no halt
halt
que no halt
good_sat salp_ssmlle_mann
LOOCV Error
12-
2.8
10
8-
2.2
150
200
250
300
(a) LOOCV MiniLM
(b) LOOCV BERT
Corruption
Corruption
Dimensions:
Enrrr
Enor
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
Corruption
Corruption
(c) Corruption MiniLM
(d) Corruption BERT
Figure LLM Comparison With and wothout early halling
5.3 Effect Of Early Halting and Ablation
Eorly halting is the following strategy with respect given threshold т: pruned by algorithm, then this halts the pruning process (independent of other criteria) and returns the sample mean of remaining data.
If more than 27 points have been
Here we examine the effect of early halting on QUE_low and ev ___fiterrin_low_nn in the context of these real world data where inliers are not generated directly from a prescribed Gaussian distribution. In other settings explored in this paper, this strategy is almost never invoked, so has no visible effect. We compare the performance of both of these algorithm with and without enforcing carly halting. We cxamine LOOCV and Corruption Error over MiniLM and BERT embeddings. Results shown in Figure g Across all 4 experiments the performance of QUE low_n shows significant degradation without early halting, going from nearly matching sample_mean in LOOCV error and nearly matching sond__ample_maee mean in corrupted error, to yielding error siguificantly worse than sample_mean and sad____mle__ee__e without early halting, Additionally, QUE_low without halting yields far larger variance results. Meanwhile, ev ww_filtering_ooe low_n performs only slightly better with early halting, and still fails catastrophically across experiments.
We perform further ablations on these LLM experiments in Appendix A.9 We explore the effect of different pruning methods filtering Jowwlww___ and different weighting methods on LRV. In both cases, find that
17

---

Published in Transactions on Machine Learning Research (02/2025)
LOOCV error can be improved using non- non-Gaussian pruning and weighting methods, whereas corrupted error is not meaningfilly improved. The failure of ev filtering_oow across pruning methods suggests the fun- damental sensitivity of the outlier scoring method used in ev sw_fiitering_low_r to distributional assumptions, which is not seen in QUE_low_r (with early halting). We additionally examine performance over these same experiments with the roles of 'field of study' and "field of land" embeddings flipped, finding nearly identical results despite differences in distribution.
Error Corruption
Data Size: 100, Dimensions: 300
Data Size: 1000, Dimensions: 2040
sample_ mren
122
ccard modien trir
coord Dimmed nean Enorr
35-
geomwing medien smor
lec, val.albntttoo single erier
2.5-
mex Einor.
quc, Eo
Pgs fine
good cample moan Emorr
1.0
022
0.0
0:20 Compption
0.20
Corruption
(a) ResNet-50 2048 Dimensional Image Embeddings:
(b) GloVe 300 Dimensional Word Embeddings:
Cat Images Corrupled With Dog Images
Pleasant Words Corrupted With Unpleasant Words
Figure 9: Additional Real World Corrupted Experiments
5.4 Additional Real World Experiments
We additionally examine the performance of robust mean estimation algorithms on corrupted embeddings from deep pretrained image models and non-contextual word embedding models. For the image embedding experiment, utilize set of images of cats and dogs from the CIFARIO dataset (Krizhevsky, 2009) with 2048 dimensional embeddings generated from a pretrained ResNet-50 model (He et al.) 2015 For the word embedding experiment, we utilize a datasel of pleasant and umpleasant words from (Aboagye el al. 2023) with 300 dimensional embeddings generated from a pretrained GloVe model (Pennington et al. 2014). Experiments are run analogously to the LLM experiments with identical settings for the mean estimators. For the image embedding experiment, inlier data is defined as embeddings of cat images, outlier data is defined as embeddings of dog images, and data size fixed at n = 1000. For the word embedding experiment, inlier data is defined as embeddings of "pleasant' words, outlier data is defined as embeddings of "unpleasant" words, and data size is fixed at 100. Results shown in Figure g
We find similar results to the LLM experiments, with QUE_low_n using early halting noticeably outper- forming sample_mean across both settings, and nearly matching good lood__smple_rggee dings. Other robust estimators tend to perform similarly or worse to sample_mean over image embed- mean, with ev wv_filteringg_ee again demonstrating significant degradation without knowledge of distributional assumptions. Additionally, lee geeliat__siplle which performs strongly in the LLM experiments, does not perform as well, demonstrat- ing its sensitivity to distributional assumptions. Similarly, median_od _______ass,_, does not perform as well across word embeddings as it does across LLM and image embeddings, no longer noticeably outperforming other estimators.
We find similar results across varying dimensionalities of image embedding and GloVe models. We examine additional image embeddings models of varying dimensionalities under LOOCV and corrupted error in Appendix A.7 We additionally recrcate the corrupted data experiment, but vary data size instead of corruption, finding that even with n > d, only QUE_low_r and median_od saciin_of___antt of_means significantly outperform sample_me mean mean error, with both estimators nearly converging to good_sample___ mean error with corruption
18

---

Published in Transactions on Machine Learning Research (02/2025)
7 0.1. We also examine GloVe models of varying dimenstonalities under LOOCV and corrupted error in Appendix A.8
6
Synthetic Data Experiments
We examine the performance of robust mean estimators across a few non-Gaussian syuthetic data exper- iments. As before, we draw X ~ (1 7) P + 77,, where is an inlier data distribution and Q is the corrupted data distribution. Similar to real world experiments, we employ the trace scaling heuristic on ev_ w_ffittering_low and QUE_low_ n and enforce early halting on QUE_low_n if more than 2T percentage of the data has been pruned.
Eroor ys Data Size
Error vs Data Size Dimensions: 500, Corupttion Percentage:
Dimensions: 500, Corruption Parcentage: 0.10
samoli_mman
coard, med
coont mmedl mean
gesmetr m_med
medion
Wi.feeeing Iowun quc_Jownn
agd good pample_menn
1-
230
500
Error Dimensions
Error Cornuption
Data Size: 500, Corruption Percentage: 0.30
Data Size: 500, Dimensions: 500
12.5
10.0
2.0
1.0
400
1000
0.9
Corrption 0.3
Dimrra ens
Figure 10: Corrupted Multivariate t-distribution
Multivariate t-distribution Define P as the multivariate t-distribution parametrized by as the all- fives vector, Σ as the identity matrix, and degrees of freedom v = 3. Observe that the covariance is LnzE, not Σ. This is a heavy-tailed distribution with polynomial tail decay. As other experiments, consider corrupted data distribution Q Na(u'', 1) where || - μ'll vd. Results are shown in Figure PGD, LRV, and median_od of means all
10 As with the other experiments, QUE_low_r ev_ l_lffieerinn_loww notably outperform sample_mean _ean here. However here, QUE_low_n, ww_filterinnglow. _eann. This trend is particularly notable under lower data size and
PGD, and LRV all
consistently outperform high dinensions. This is explainable given that the sample mean is known to be sub-optimal estimator for heavy tailed distributions. These results suggest that robust estimators designed for the Huber contamination model have practical application the heavy-tailed distribution setting. We leave further investigation into this connection to future work, building on Prasad al. (2019). We also note that there a more notable separation between the best performing methods in this settings than in the inlier Gaussian data scenario. Tn
19

---

Published in Transactions on Machine Learning Research (02/2025)
particular, ev_ w__fittering_lw_n and LRV consistently outperform QUE_low_n, and also QUE_low_r exhibits areas of high variance in error.
Laplace Distribution Define P as the product distribution of independent Laplace distributions, cach with mean 5 and scale 1. Then, the true mean μ defined as the all-fives vector. Again, consider cor- rupted data distribution Q Na(u', T) where I|æ- -11l = vd. Results are shown in Figure 11 Results among the best estimators are nearly identical to the Gaussian inlier data scenarios, with QUE_low_n" ev_ w__fiitering_low
low n, PGD, mean error and with LRV performing marginally worse.
all nearly matching We note that although the Laplacian distribution has heavier tail than the Gaussian, we do not observe the same trends as in the multivariate t-distribution. This can be explained by the fact that Laplacian tails still decay exponentially, whereas the tails in the multivariate t-distribution decay polynomially.
Error vs Data Size
Error vs Data Size
Dimensions: 500, Corruption Percentage: 0.10
Dimensions: 500, Comupption Percentage: 0.10
5a sample, meen
coard median
coond,
que
pat
good www.:sssmoa_ooo mocn
2000
3020
4003
2002
100
330
302
vs Corruption
Data Size: Cornuption Percentage: 0.10
Sizo: 500
15.0
12.5
Ends
10.0
5.0
25
200
400
650
1000
6.0
0.1
0.2 Comuptinn
Dimensoon
Figure 11: Corrupted Laplace Distribution
Poisson Distribution Define as the product distribution of independent Poisson distributions, each with mean 5. Then, the true mean y is defined the all-fives vector. Again consider corrupted data distribution Q = Na(u', 10 where ||w- μ'll = Vd. Results are shown in Figure 12 Results among the best estimators are again nearly identical to the Gaussian inlier data scenarios.
Mixture of Gaussians Define P as a mixture of Gaussians with three components, each equally weighted. The components have means μI = T, μ = 0, and μa = -T, where I and 0 are the d-dimensional vectors of all ones and all zeros, respectively. Each component has identity covariance. Define Q as Ar, 1//1 where I is the all-twos vector. Results are shown in Figure 13 While robust methods tend to outperform sample_mean, relative performance here differs from the Gaussian inlier data setting. Firstly, note that QUE_low_n does not perform the best and shows irregular areas of high error under lower dimensionality and moderate corruption levels. The observation that QUE_low_n, low_n, ev_ ww_filtering_low lowrrngg__wwwn,,. and PGD do not perform as well is likely because they rely significantly the Gaussianity assumption for the inliers. Meth
20

---

Published in Transactions on Machine Learning Research (02/2025)
Error vs Dota Size
Error vs Data Size
Dimensions: 500, Corruption Perceniage: 0.10
Dimensions: 500, Corruption Percentage: 0.10
14
sample_mcon
coard conntziimeed_mm cumedioo
median menns
fittiing Jowun
EnU
que Jos
Dgd
good sumple mezn
1000
2000
3000
1000
5000
200
500
Dimenslons Size: Comuuption Percentage:
vs Corruption
10.0
2-
Eimenslons
Commptoon
Figure 12: Corrupted Poisson Distribution
such ____mmmns,,_ the other hand which do not rely directly this model are not as effected. Furthermore, ev ww_filterinng_owwo LRV, and PGD all show significant degradation as corruption levels increase, which is not observed in the Gaussian inlier settings. This may be occurring if they completely filter one of the three "inlier" modes as outliers. Interestingly, LRV notably outperforms all other estimators and even good_sample_mtee excopt with corruption т > 0.2 and especially for n < d. The strong performance of LRV in this setting interesting, and may be consequence of the three inlier distributions means lying on 1-dimensional subspace.
7 Comparing Algorithm Variants and Ablation
In this section, we justify and explore our adaptations to ev_filtering QUE, and lee_valiant We use these adaptations for the remainder our experiments.
Eigenvaluc-based Threshold Here we compare ev ev_filtering and ev_ si_fflteringgloo n, along with QUE and QUE_low We observe that when do not have n very large compared to d, then ev w__fitering and QUE dramatically shift from low error abysmal error rates.
We recrcate the experiment over corrupted Gaussian data with identity covariance and DKK Noise from Section 4.2 This is shown in Figure 14a We find that ev_filtering and QUE fail catastrophically with insufficient data, performing far worse than sample_mean, However, ev w__filtering_lo__ and QUE_low_r never perform worse than sample_met _ean and achieve near optimal performance regardless of data size. With sufficient data, ev w__filtering and QUE abruptly begin to work, and achieve near identical performance to their adjusted threshold counterparts.
21

---

Published in Transactions on Machine Learning Research (02/2025)
Emor vs Data Size
Error vs Data, Size
Dimensions:: 500, Corruption Percentage: 0.10
Dimensions: 500, Corruption Percentage: 0.10
sanokkmmcin
coard median
coont_timmedd_mm
10
geomettic
median_otmmens
aV Riffiinnngg.... on.n
Enur
6--
que, pwrn
ogd
good sample mean
1000
2030
1000
5000
130
300
500
Corppption
35
25
3-
15-
Dimens
Connuction
Figure 13: Corrupted Mixture Of Gaussians
True Eigenvalue Data Size
Dimensions: 500, Expected Corruption:
Crror Deta Size
True Eigenvalues
Dimansions: 500, Comruption Necentage: 0.10
Original Thresheld
mple,
60
New Threshoid Values
Eigenvalue Threshold
M
que
que lon Ener
15-
D1er
10
5--
2000
1000
6000
3000
10000
Data Size
Data Size
(a) Corrupted ldentity Covariance DKK Noise
Uncorrupted Identity Covariance
Figure 14: Eigenvalue Based Filtering Comparison
The failure of ev ev_filtering and QUE occurs in the corruption detection step. This corruption detection threshold on the top eigenvalue is initialized as + 37 log(1/r) in ev_filtering and QUE. This constant threshold uses the fact that with large enough data size, the top eigenvalue of an identity covariance matrix approaches and the 37 log(1/7) term can account for tolerable noise. However, this does not account for corruption due to low data size, in which the top eigenvalue of the uncorrupted data will necessarily
22

---

Published in Transactions on Machine Learning Research (02/2025)
have larger expectation as data size decreases. Therefore, ev filtering and QUE can never be expected to work since even without any corruption, the top eigenvalue will exceed the threshold. As a result, we find that ev fy_iitering and QUE keep on pruning until there are only very few data points left, resulting in the catastrophic crror exhibited. ev ev_ incorporating our new result. (Corollary 1.1p as a threshold on the lop eigenvalue of the covariance madrix in the threshold. This empirically shown Figure 14b The threshold in ev w_filtering and QUE does not become a true upper bound on the top eigenvalue of the uncorrupted data until the vertical red line, which roughly corresponds to the point that ev l__fittering_lo__n begins to perform better. QUE begins to perform better with much less data size than ev_ ev_fittering, but this is unsurprising given the rapid convergence of the inlier top eigenvalue to 1. Meanwhile, our new threshold used in ev w__fiterring_oww and QUE_low_D is always an upper bound on the top eigenvalue, and is near-optimal in practice.
filtering_low_r and QUE_low_n remedy this problem by simply
Error us Data Size Dimensions: 500, Corruption Parcentaga: 0.10
Error vs Data Sizp
Dimensioos: 500, Corruption Perconzage:
lec, volicoiinn.... mmplc mom Eroor
lee yoduni cnggnll mam tnor
lee Single In enror
velient ample firor
engioal
et nitering lou Eror
3
good wampp. mean Entrr
Dror
2000
5000
200
560
Dalt
Error Dimensions
Corruption
Data Size: Corruption Porcentage: 0,10
Data 500,
500
4.1
10-
12.5
30.0
7.5
5.0
2.5
0.5
0.0
200
400
400
500
1000
0.0
0.1
02
03
34
Dimonsions
Co Compopon
Figure 15: Lee Valiant Variants: Identity Covariance Additive Variance Shell Noise
Lee and Valiant variants. Here we compare different variants of the Lee and Valiant algorithm. We observe that lee_ praliant_simple performs a bit better than lee_ valiant, and median__f__meddee _f___eess__a__eee_eeee___ is an illustrative choice for initial estimator. We recreate the experiment over Gaussian data with identity covariance and additive variance shell noise from Scction 4.2 across different variants of the Lee and Valiant algorithm. We test lee ses__valant__mmma simple and lee valiant using median_ peedian_of_mean,, means, LRV, and ev ww_filtering_lon low_n as initial mean estimators. We additionally plot LRV and ev ev_filtering__ww. as baselines. This is shown in Figure 15
ee__aliant_simpl differs from lee_valiant in two ways: (1) it removes outlicrs completely instead of down- weighting them and (2) it does not use initial mean estimate in the final result. We see that lee dde_valiattssmmpa performs better than lee vel_vat in practice, especially with larger n or d. Both te__valanttsimmll and lee valiant see benefit the use of improved initial estimators, with this improvement being more signif- icant in lee valiantt The difference in the relative improvement in performance between the algorithms is explained by the fact that lee valiant additively incorporates the initial estimate directly into its final esti-
23

---

Published in Transactions on Machine Learning Research (02/2025)
mate. However, there is no benefit gained from combining lee valvaii or lee ree__valantt_siie with improved initial estimator compared to using the initial estimator alone. As result of these findings, we only evaluate lee_ vaaiiattliiipppppa in our experiments.
8 Conclusion
We perform the first wide-scale experimental study of robust mean estimation techniques in high dimensions and relatively-low data size. We showed that under Gaussian data with bounded covariances, robust mean estimation techniques can significanlly outperform sample mean, nearly matching the optimal error oblain- able, regardless of data size, dimensionality, or corruption level. We provide an updated eigenvalue filtering bound that is useful this high-dimensional sctting, and use to devise a small but novel and meaningful modification to (wo existing robust mean estimatiin algorithms; eigenvalue pruning from Diakonikolas et al.) (2019a) and quantum entropy scoring from Dong et al. (211)) This enables these methods to almost exactly match optimal error regardless of data size that is almost matching the error of the so-called good sample mean, which is the mean of the inliers. It seems QUE_low_n works so well because it can identify all ways input distributions deviate from Gaussianity, whereas other mods may require more iterations, which may ultimately prune too many points in the n S d setting before able to filter outliers in each direction that has them.
However, all methods perform significantly worse than the mean of all inliers under subtractive corruption where f-frcction of data points can be removed adversarially. This suggests that in this Gaussian modeled data regime, practical improvements may be possible in considering the effect of subtractive corruption.
We also provide novel evaluation on realistic settings based on the embeddings generated from large language models, deep pretrained image models, and word embodding modols. These are representative of real world settings where the data size n may be smaller or not much larger than the dimension d. In these settings, quantum entropy scoring with early halting tends perform near optimally, suggesting its potential application to real world data distributions regardless of data size. However, other robust mcan estimation algorithms do not work well when the inlier data is not Gaussian, as perhaps foreshadowed by theoretical results leveraging this assumption.. This suggests that further valuable results may be derived by moving away from the assumption that inliers are precisely Gaussian. Our initial explorations for corrupted data with non-Gaussian inliers shows the same techniques mostly perform well, but which method performs the best can vary based on the inlier distribution. Notably, some methods can even have less error than good_sample_mean when the sample mean not the MLEE.
Overall, our work demonstrates that there is value in applying robust mean estimation techniques to data, even with insufficient data size for the classic theoretical bounds. We hope that our work inspires researchers to further consider, both experimentally and thcoretically, the crucial case of high-dimensional robust statis- tics under low data size.
Acknowledgments
The authors thank Meysam Alishahi for early discussions, especially towards the analysis in Theorem 2. We also thank the NSF REU project 224492, under which CA was participant and initiated this work. JP also thanks funding from NSF 2115677 and 2421782, and Simons Foundation MPS-AL-00010515.
References
Prince Osei Aboagye, Yan Zheng, Jack Shunn, Chin-Chia Michael Yeh, Junpeng Wang, Zhongfang Zhuang, Huiyuan Chen, Liang Wang, Wei Zhang, and Jeff Phillips. Interpretable debiasing of vectorized language representations ions with ilerative orthogonalization. In The Eleventh International Conference on Learning Representations, 2023. URL
Akshay Agrawal, Robin Verschueren, Steven Diamond, and Stephen Boyd. A rewriting system for convex optimization problems. Journal of Control and Decision, 5(1):42 60, 2018.
24

---

Published in Transactions on Machine Learning Research (02/2025)
Hilal Asi, Jonathan Ullman, and Lydia Zakynthinou. From robustness to privacy and back. In International Conference on Machine Learning, pp. 1121-1146. PMT.R, 2023.
Ainesh Bakshi, Ilias Diakonikolas, He Jia, Daniel M. Kane, Pravesh K. Kothari, and Santosh S. Vempala. Robustly learning mixtures of k arbitrary gaussians. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2022, pp. 1234-1247, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450392648. doi: 10.1145, 11145//511993331199555 URL https://doi...... org/ 10. 001111//551935.3519999
Sivaraman Balakrishnan, Simon S. Du, Jerry Li, and Aarti Singh. Computationally efficient. robust sparse ostimation in high dimensions. In Satyen Kale and Ohad Shamir (eds.), Proceedings of the 2017 Conference on Learning Theory, volume 65 of Proceedings of Machine Learning Research, pp. 169 212. PMLR, 07 10 Jul 2017. URL https://procoeedings mlr. ppress/v6//aalarriiiiie hatmrihhshnanaaamm
David M Blei and John D Lafferty. Topic models. In Terl mining, pp. 101 124. Chapman and Hall/CRC, 2009.
Tolga Bolukbasi, Kai- Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man to computer programmer as woman to homemaker? debiasing word embeddings. Advances neural information processing systems, 29, 2016.
Olivier Catoni. Challenging the empirical mean and empirical variance: a deviation study, 2011. URL
2048
Moses Charikar, Jacob Steinhardt, and Gregory Valiant. Learning from untrusted data, 2017. URL https: 02315
Mengjie Chen, Chao Gao, and Zhao Ren. Robust covariance matrix estimation via matrix depth. arXiv preprinl 100: 00091, 2015.
Mengjie Chen, Chao Gao, and Zhao Ren. Robust covariance and scatter matrix estimation under huber's 00691
contamination model, 2017. URL
Yu Cheng and Honghao Tin. Robust learning of fixed-structure bayesian networks in nearly-linear time, 2021. URL
05555
Yu Cheng, Ilias Diakonikolas, and Rong Ge. High-dimensional robust mean estimation in nearly-lineat time, In Proccedings of the thirtieth annual ACM-SIAM symposium discrete algorithms, pp. 2755-2771. SIAM, 2019a.
Yu Cheng, Ilias Diakonikolas, Rong Ge, and David Woodruff. Faster algorithmss for high-dimennsionll robust covariance estimation, 2019b. URL
04661
Yu Cheng, Ilias Diakonikolas, Rong Ge, and Mahdi Soltanolkottbi. dingh-immesionall robust mean estimation via gradient descent, 2020. URL.
01378
Yu Cheng, Ilias Diakonikolas, Rong Ge, Shivam Gupta, Daniel M. Kane, and Mahdi Soltanolkotabi. Outlier- robust sparse estimation via non-convex optimization, 2022. URL htttps:///axxi..rr//bbbbb
11515.
Jules Depersin and Guillaume Locué. Robust subgaussian estimation of a mean vector in nearly linear time, 2019. URL atttpa//aaxxv.orr//aa////// 0005/
Aditya Deshmukh, Jing Liu, and Venugopal V. Vecravalli. Robust mean estimation in high dimensions: An outlier fraction agnostic and efficient algorithm, 2022. URL
http://xxvvv.ogaaaaaaaaaa 08573
Sunipa Dev and Jeff Phillips. Attenuating bias in word vectors. In The 22nd international conference artificial intelligence and statistics, pp. 887. PMLR, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL
04800
25

---

Published in Transactions on Machine Learning Research (02/2025)
Luc Devroye, Matthieu Lerasle, Gabor Lugosi, and Roberto I. Oliveira. Sub-gaussian mean estimators, 2015. 058455
URL
Maa Diakonikolas and Daniel M Kane. Algorithmic high-dimensional robust statistics. Cambridge university press, 2023.
Tlis Diakonikolas, Gautam Kamath, Daniel M Kane, Jerry Lí, Ankur Moitra, and Alistair Stewart. Being robust (in high dimensions) can be practical. In International Conference on Machine Learning, pp. 999-1008. PMLR, 2017a.
Ilis Diakonikolas, Daniel M. Kane, and Alistair Stewart. List-cccodabeee robust mean estimation and learning mixtures of spherical gaussians, 2017b. URL htttas//aaraiivorrg/bb///1111
Ilias Diakonikolas, Daniel M. Kane, and Alistair Stewart. Statistical query lower bounds for robust estimation of high-dimensional gaussians and gaussian mixtures, 2017c. URL http://awwivv.oogabb/////a 034733,
Tlias Diakonikolas, Weihao Kong, and Alistair Stewart. Efficient algorithms and lower bounds for robust linear regression, 2018 URL https://aaxix: http///wwww...r///////..00000 httg:/bb/18006.00
Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Ankur Moitra, and Alistair Stewart. Robust ostimators in high-dimensions without the computational intractability. SIAM Journal on Computing, 48 (2):742 864, 2019a.
Tlis Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Jacob Steinhardt, and Alistair Stewart. Sever: A robust meta-algorithm for stochastic optimization. In International Conference on Machine Learning, pp. 1606. PMLR, 2019b.
Ilias Diakonikolas, Sushrut Karmalkar, Daniel Kane, Eric Price, and Alistair Stewart. Outlier-robust high- dimensional sparse estimation via iterative filtering, 2019c. URL httpp::///// htttt:::/aaxxvv.oor//////////// 1ts///b//1911....00 08085
Tlas Diakonikolas, Daniel M. Kane, Sushrut Karmalkar, Ankit Pensia, and Thanasis Pittas. Robust sparse mean estimation via sum of squares. In Po-Ling Loh and Maxim Raginsky (eds.), Proceedings of Thirty Fifth Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pp. 4703- 4763. PMLR, 02 02-05 Jul 2022. URL httptp////gccoeeii.....rppeeseep
mlr ptwppee///1///dcaaoookooooooooa
html
Tlas Diakonikolas, Daniel M. Kane, Sushrut Karmalkar, Ankit Pensia, and Thanasis Pittas. Robust sparse estimation for gaussians with optimal error under huber contamination, 2024. URL https://arxiv.org
abs/2403 10416
Steven Diamond and Stephen Boyd. CVXPY: A Python-embeddeed modeling language for convex optimiza- tion. Journal of Muchine Learning Research, 17(83):1 5, 2016.
Yihe Dong, Samuel B. Hopkins, and Jerry Li. Quantum entropy scoring for fast robust mean estimation and improved outlier detection. In NewrIPS, 2019. URL
11366
Cynthia Dwork and Jing Lei. Differential privacy and robust statistics. In Proceedings of the forty-forst annual ACM symposium Theory of computing, pp. 371-380, 2009.
Thomas Griffiths and Mark Steyvers. Finding scientific topics. Proceedings of the National academy of Sciences, (Ssuppl_1):5225 1):5228 5235, 2004.
Shivam Gupta, Jasper CH Lee, and Eric Price. Finite-sample symmetric mean estimation with fisher in- formation rate. In The Thirty Sixth Annual Conference on Learning Theory, pp. 4777-4830. PMLR, 2023.
Shivam Gupta, Samuel B. Hopkins, and Eric Price. Beyond catoni: Sharper rates for heavy-tailed and robust mean estimation, 2024. URL /ttps://arxiv.orgg/ abs///11.b/31/..../// 13010
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL
26

---

Published in Transactions on Machine Learning Research (02/2025)
Samuel B. Hopkins and Jerry Li. How hard is robust mean estimation?, 2019. URL https://arxiv..rgg
Samuel B. Hopkins, Gautam Kamath, Mahbod Majid, and Shyam Narayanan. Robustness implies privacy in statistical estimation. In Proceedings of the 55th Anmual ACM Symposium on Theory of Computing, STOC 2023, pp. 497-506, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450399135. doi:
URL https:///dooi http:////o.or///..../// 1145//354246.35851155 3585115
Andrew Howard, Mark Sandler, Grace Chu, Liang- Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoe V. Le, and Hartwig Adam. Searching for mobilenetv3,
2019. htthttts::////// https://arxiv. http::////wxvv..oraaaaaas// wttp.ob////../02//.2033
Wenpeng Hu, Jiajun Zhang, and Nan Zheng. Different contexts lead to different word embeddings. In Yuji Matsumoto and Rashmi Prasad (eds.), Proceedings of COLING 2016, the 26th International Conference on Compulational Linguistics: Technical Papers, pp. 762 771, Osaka, Japan, December 2016. The COLING 2016 Organizing Committee. URL htttps:://aaaaattholoooooo www.oooyyor.////////100
PJ Huber. Robust estimation of a location parameter. Ann. Math. Statist,, 35:73- 101, 1964.
Ayush Jain, Alon Orlitsky, and Vaishakh Ravindrakumar, Robust estimation algorithms don't need to know the corruption level, 2022. URL
Adam Klivans, Pravesh K. Kothari, and Raghu Meka. Efficient algorithms for outlier-robust regression, 2020. URL httpt:://aaxiii http://wwwrr/abb/////e 13030033113 03241
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. URL. https:
semanticscholar. wwt. 18268744
Kevin A Lai, Anup B Rao, and Santosh Vempala. Agnostic estimation of mean and covariance. In 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS), pp. 665-674. IEEE, 2016.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations, 2020. URL https://arxi>:
org/ abs//90919 11942
Beatrice Laurent and Pascal Massart. Adaptive estimation of a quadratic functional by model selection. Annals of statistics, pp. 1302 1338, 2000.
Jasper CH Lee and Paul Valiant. Optimal sub-gaussian mean estimation in very high dimensions. In 13th Innovations in Theoretical Computer Science Conference (ITCS 2022), 2022.
Xiyang Liu, Weihao Kong, Sham Kakade, and Sewoong Oh. Robust and differentially private mean estima- tion, 2021a. URL /ttrp::/aaxxi..rr/abbb.sss
09159
Zifan Liu, Jong Ho Park, Thcodoros Rekatsinas, and Christos Tzamos. On robust mcan estimation under coordinate-level corruption. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Reseorch, pp. PMLR, 18-24 Jul 2021b. URL https://proceeddnggg
mlr www.spresssviie watsss/vvv////11 0a099//lu211... wat.22rrhhm .tml
Gábor Lugosi. Mean estimation in high dimension. In Proc. Int. Cong. Math, volume 7, pp. 5500-5514, 2022.
Gabor Lugosi and Shahar Mendelson. Mean estimation and regression under heavy-tailed distributions- a survey, 2019. URL https://arxiv.. wwwtaxiiv..g//eee wtggabb//1906.4255
Gábor Lugosi and Shahar Mendelson. Robust multivariate mean estimation: The optimality of trimmed mean. The Annals, 49(1):393-411 410, 2021.
Gábor Lugosi and Shahar Mendelson. Sub- Sub-gaussian estimators of the mean of random vector, 2017. URL https://arxiv http///wwww..rrraa///////////./0
27

---

Published in Transactions on Machine Learning Research (02/2025)
Tomas Mikolov, Ily Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
Stanislay Minsker. Efficient median means estimator, 2023a. URL
18681
Stanislav Minsker. U-statistics of growing order and sub-gaussian mean estimators with sharp constants, 2023b. URL https://arxiv..rrr www.aasss/.// 11842
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word repre- sentation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532 1543, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/D144112.. URL httpa://cclaathhoooy.oor///////i
Adarsh Prasad, Arun Sai Suggala, Sivaraman Balakrishnan, and Pradeep Ravikumar. Robust estimation via robust gradient estimation, 2018. URL http:://aaarii..o/aab///////
Adarsh Prasad, Sivaraman Balakrishnan, and Pradeep Ravikumar. A unified approach to robust mean estimation, 2019. URL
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limils of transfer learning with a unified texi-to-text transformer, 2023. URL http:: /tapxiv///xxv..oo/ wwt.:/s.//////..oeraooo 1910. 10683
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends@ in Information Retrieval, 3(4):333-389, 2009.
Christopher G Small. A survey of multidimensional medians. International Statistical Review/Revue Inter- nationale de Slalislique, pp. 263-277, 277, 1990.
Mingxing Tan and Quoc V. Le. Efficiettnet: Rethinking model scaling for convolutional neural networks, 2020. URL
11946
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks, 2018 URL 00636
https://arrs fttrps:/aarxiv.ogee
John W Tukey. Mathematics and the picturing of data. In Proceedings of the International Congress of Mathemalicians, Vancouver, 1975, volume 2, pp. 523-531, 501, 1975.
Yehuda Vardi and Cun-Hui Zhang. A modified weiszfeld algorithm for the fermal-weber location problem. Mathematical Programming, 90:559 566, 2001.
Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices, 2011. URL https:
//arxiv. org/ httpg//bs/1011.... 1////011130000
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention
distillation for lask-agnostic compression of pre-trained transformers, 2020. URL https: httpss://axxit http://axxxvvorro/
10957
Banghua Zhu, Jiantao Jiao, and Jacob Steinhardt. When does the tukey median work? In 2020 IEEE
International Symposium Information Theory (ISIT), pp. 1201 1206. IEEE, 2020a.
Banghua Zhu, Jiantao Jiao, and Jacob Steinhardt. Robust estimation via generalizod quasi-gradionts, 2020b.
URL httll//tsss//aav.ooo//////////// 14073
Banghua Zhu, Jiantao Jiao, and Jacob Steinhardt. Generalized resilience and robust. statistics, 2020c. URL
https://arxi: http:///axvvvvvv arrg/bss/990/ 08755
28

---

Published in Transactions on Machine Learning Research (02/2025)
A
Appendix
A.1 Updated Eigenvalue Threshold
Here we leverage & theorem in Vershynin 2011 to bound the complexity of aggregated high-dimensional Gaussian random variables. We will use it in a couple ways.
Theorem 2 Vershyyin (2011) Thm. 5.35). Let A be a anxe d matrix whose entries are independent standard normal random variables. Let ||A||2 denote the spectral norm of A. Then for every 2 0, with probability of least 2 eexp(-12/2) one has
Vn- Vn-vd- Va-va-iS (A) S 11 Vn+ Vd+L.
Where Smin (A) is the smallest singular value of A. The lower bound assumes n > d, if not the roles are reversed.
We prove the following implication.
Theorem 3 (restalement of Theorem D. Let X be x d matrix whose entries are independently drown from ((,1), I). Let D = =(X - " (i) be the sample covariance matric of X, where = -, E, X, and X, is the ith row of X. Then for every > 0, with probability of at least - 3exp( (2/2 /2), one has
1|112 1+ +tv///+t//ne 1/+++ Vd 22
22dt
n
Proof. Let = x - X be the contered matrix, equivalent to each entry being drawn from N(0,, I). Let Z X - p be the matrix centered by the sample mean. Then | = || + Iu xlll2 S || + || - Ølla by triangle inequality.
First, by Theorem we have that | S vn + vd+t. +t, t, with probability at least 1 - 2 cxxp
Second, to bound || /ll we first decompose by coordinate |/ - 113 ΣE p. Now consider d random variables Bj = μ, py for j = ....d, d, and further write B, = A - F; where F. ~(, 1). As a result By ~ (/0 1/n) = Jo (0, since the average of n normals still normal with variance reduced by factor As result Bj is squared normal distribution, and B n|lu ح = E B3 is chi-squared distribution (d). Hence we have Laurent & Massart 2000)
Pr[B 2 d 2 dt + 20 S exp(-s²).
Inside the probability expression, using v BB/n = 1| - All, and letting & = /2s, we can rewrite this as
Pr |p Alll < / (d+ /2 +t/n 2 21 -1-eep(-t?/2).
So now if both of these events hold, which by union bound occurs with probability at least 1 - exxp(- (tap--)/22) we have that
d 2dt + t²
|| la < ||x||2 + || pll & (VT+
VV va+t)+
Notice that
|E|| = 1l |
= | Thus we have
|>ح|| = 1211
< (++V//ntt//
2 dd 2dt + +2
n
11
29

---

Published in Transactions on Machine Learning Research (02/2025)
Note that the fourth term in this bound, coming from the error in || pill is lower order effect. This is captured in the following corollary.
Corollary 3.1 (restatement of Corollary drown from Nu, 1). Let E (X n)" (X (t) be the sample covariance matriz of X, where μ = Σ, X; and X; is the ith row of X. If one assumes d/n I 16,n 2 16,t 2 5, then with probability of at least 3 3xxp(-t2/8) one has
Let X be x d matric whose entries independently
Ell2 - (1 t//n+t/VV))
Proof. Starting with the bound Theorem we have
2
1 vd/n+t/Vrr Vd+ 22dt + +
| =
n
d/t2 + /2d/t+1 +1
1 vdfn +IV
Vn
Vn.
1+ /d/n t
1 V (d/n)/P /1/
vn
t
1+ vd/n +
1+ (16)/12 V2V16///2 1/m
Vn
1
1+ dd/n
1 v V32 +1/n
Vn
t
1+ /d/n +
1+ 16/22 + (5 4) 1/16
Vn
< (1+ Vd/n+ 2t/ vt
Adjusting 2t in the probability of failure, so is 3exp(-f//8) instead of 3 exp(-rf/2), completes the proof.
A.2 Corrupted Gaussian Data Identity Covariance: Additional Noise Schemes
We examine the performance of robust mean estimators across additional corruption schemes. We still draw X ~ (1 -1--1 P + n/Q where P = (N) and Q describes the corrupted data distribution, where μ is the all-fives vector. We utilize the following additional corruption schemes:
Two Gaussian clusters shifted to variance shell. 0.7Na(p 1o 0.3Na(p 1) where ||и - 1 Vd, || 1|p-ы n'll Vd, and f 75 where 0 is the an- gle between of and ntt. The location of μt is determined by random rotation matrix to prevent any coordinate-axis specific biases. Results over this noise distribution are shown in Figure 16
Consider corrupted data distribution Q
In Distribution Noise. Consider corrupted data. distribution, Q, where for each corrupted data point ga Q, each coordinate j of is drawn from Uniform(yg, + 2). Here μj represents the jth coordinate of the true mean μ. Results over this noise distribution are shown in Figure 17
Large Outlier Noise. Consider corrupted data distribution Q 0.7Ma a 1// where || -u-~0' 1| 10yd, 1 = 20,d, and 0 75 where 0 is the angle between n and n² The location of n determined by a random rotation matrix prevent any coordinate-axis specific biases. Results over this noise distribution are shown in Figure 18
30

---

Published in Transactions on Machine Learning Research (02/2025)
Large Outlier Noise Mixes. Consider corrupted data distribution Q 0.5L U 0.5Q' where L is the large outlier corruption scheme previously described and Q' is subtle corruption scheme. We examine two settings for Q': additive variance shell corruption with one cluster, shown in Figure 19 and DKK corruption,
shown in Figure 20
Error ys Data Siee Dimendonns Comuption Percentage 0.10
Evey us Data pe Compption Percentage:
coont Of SITCE
Oree
Thiiing bor Frre
Ege
2033
3000
9030
3900
239
00E2 9020
3003
920
Dimensions
Frmr Compution
Data Stze: 500, com--nttnn Peccentaee:
Data Shee 500, Eimendone 900
1%5
10-
10
210
30
306
0.5
Uliransions
Compoon
Figure 16: Corrupted Gaussian Identity Covariance: Two Variance Shell Clusters
Across all of these distributions, including those with large outliers, ev ___fiteriig_low_n. QUE_low_n, PGD, and LRV perform the best, suggesting that their performance is not overly sensitive to the noise distribution. However, we note that across schemes with large liers, PGD sees areas of higher variance and slightly worse performance, and LRV degrades worse as 7 increases with large outliers. This downgrade in performance can be remedied by first preprocessing data by removing large outliers through a naive pruning method, but this step doesn't appear necessary for other methods. We remark that LRV requires outlier weights to be clipped to avoid numerical instability issues under large outlier schemes. Otherwise, it will degrade poorly over large outliers and large 7) predicted outliers will be assigned near-zero weights. We also see again that median_of_ means significantly outperforms other simple estimators, especially under large data size, al- though its performance degrades poorly under certain conditions, such as with larger v. With large outliers, lee_valiant_simpl nearly matches good_sample_ment error across conditions, achieving much better perfor- mance than does across subtle noise distributions. As it additionally outperforms poord__trime__ng mean, this suggests that lee lee_valiant diaiiantsimmlle may operate as a more effective naive pruning method, as seen in the LLM experiments.
Dependence true mean We additionally verify that performance does not depend on the choice of true mean, μ. We recreate experiments over Additive Variance Shell Noise and DKK Noise over different choices of μ.. We replicate the same experimental setup as before, but draw every coordinate of μ from N0 50) at every iteration in the experiment rather than fixing as the all-fives vector. As a reminder, this occurs for every choice of the independent variable over every run. If performance depended on μ, we would expoct achieve high variance results. Instead, we find nearly identical results to the original experiments across both distributions. These results are shown in Figure 21 and Figure 22
31

---

Published in Transactions on Machine Learning Research (02/2025)
Eru Dala Sige
Erus Duto 5 So
Omersions: 50D Compppon Pensensage:
Dimonsiers Comuption Porcortage:
crere IETEINS prase vmne
poaneti,, Jec Dose
Erter
Stering lOw, Yor 2Pt, Joy Eror
1200
2000
3000
5000
220
402
Daca Siz
200
500
Emary
Dota Size: 580, Cortuction Percentage 0.30
Dato Size: 500, Dimendors: 500
13.5
25-
2003
Aeaaddnn
composn
Figure 17: Corrupted Gaussian Identity Covariance: In Distribution Noise
Fricy us Data ze
Errr vs Dato Size
Percentagn:
Percentage
75
aar ripp moon Ersr
firemed Evor
grenoniic andiin wan
loz, Lalleramate bier mod
fhwetng for
lin. S/1 roRl Time
12-
po0c www.prrriiiiannnnmommmmmo Erer
10C0
2000
3003
1300 922e
1353 size
Emor
Data Siz 53C, Corpettin Perrentagee 0.10
Data Sue: 500, Dimersioon 500
323
25
75-
32
23-
Cimsnnsions
Gemagtto
Figure 18: Corrupted Gaussian Identity Covariance: Large Outliers
32

---

Published in Transactions on Machine Learning Research (02/2025)
Euct Duta Sizo Dimonslons: Comuator Porcentage:
Erg vo Dula Siee
Dimenslens; 500, Comppion Porcontage:
12
sumple Errr
Porer For
Toc
median
post Cre
pood Jamp
2000
3000
4000
5000
202
218
300
452
330
EX0 5e2
Data 500
Emorys aption
Data Stee:50 Comuution Percentage: 0.10
Deti Slat: $20, Divendions 500
33.0
22.0
32-
510
20-
30
30
50-
Uptrastions
compptio
Figure 19: Corrupted Gaussian Identity Covariance: Large Outliers w/ Additive Variance Shell Noise
Ercor us Data Size
Eror vs Data Size Compption Percentage
Percentagn:
12-
aemple Erar
20
coord trroned Lov
gevemntr endian wirr
loc und seraac toe
medion
Wharing
1991 tre
2002
3000
3000
5000
392
230
302
322
D300 size
DSta 5000
aption
Disas Sae 503, Coempption Permentage 0.30
Data Sizer 520, Dirreniton 100
210
210-
15.
1.
25.
50
Oemeessions
Compptin
Figure 20: Corrupted Gaussian Identity Covariance: Large Outliers w/ DKK Noise
33

---

Published in Transactions on Machine Learning Research (02/2025)
Erun vo Dula Siee
Erus to Outa See
compptin Tercontage:
520, Coruption Percentage:
coort
coert riverns PAN FO17
goancct wonoee DH1 JATOA Ene
CV Stter fiwen
(MC In, tre
1100
2000
4003
500
300
452
330
DVG 5.00
Eroon Cormuttion
Data Stac: 500, Com, pilion Parcentage: 3.10
Dato Size: 500, Dimensere: 500
40
13.5
15
10
15
8 3.0
75
14
10
05
15-
60
LEiwecc ond
Camppoer
Figure 21: Dependence On True Mean: Identity Covariance, Additive Variance Shell Noise
Frror ys Data Size
Eroor vs Dota se
Percentage:
coond DEENTA WIT
Theiing bn i Frre
que
Ege
4-
2033
3000
8030
3000
202
330 920
Dimensions
Frmr
Data cor-ppttnn Preceetace: 10
Data She 500, Eimendonee
10-
3.5
10
225
0.5
Citrnnsoons
Compoon
Figure 22: Dependence On True Mean: Identity Covariance, DKK Noise
34

---

Published in Transactions on Machine Learning Research (02/2025)
A.3 Corrupted Gaussian Data Unknown Spherical Covariance: Additional Corruption Schemes
We examine the unknown spherical covariance case across additional corruption schemes. We utilize the same uncorrupted distribution as before, P= =Na(u, where is the all-fives vector and o = 5. We find similar performance across the distributions we test.
Adapting noise distributions to spherical covariance As in the Gaussian noise shifted to variance shell case, we utilize the well know Gaussian concentration inequality that for data X ~ Na(p,al) Ex> [||-- nll3) = ddd. This observation is used to adapt noise distributions in the identity covariance case to this case. For two additive variance shell clusters, each cluster has mean j¹ for E [0, 1] where I|u - μان - ovd, with other conditions remaining the same; results are shown in Figure 23 For DKK noise, half the noise is drawn over the hypercube where every coordinate is -a or O away from the true mean at that coordinate with equal probability. The other half is drawn from the product distribution where the first coordinate is cither 11 or -a away from the true mean al that coordinate with equal probability; the second coordinate -3a or -a away from the corresponding true mean coordinate with equal probability, and all remaining coordinates are away from the true mcan. Results are shown in Figure 24 For in distribution corruption, we draw cach coordinate of a corrupted data point from Uniform(p Mj 20); results are shown in Figure 25 We also perform subtractive corruption, using the same scheme as in the identity covariance case; results are shown in Figure 26
Fror es Data Size Percentaçe: 0:10
Eroor vs Data
Simenstons:
Dimensions:
Percentagei
Srnsk mosr BIA
coe wreshat Erp
Unnam
TV Erer
ey. Elrikng
12.5
33-
0
3000
2008
3902
4002
3030
300
230
303
402
332
Error vs Dimoniions Compgilsn
Eriee ys Corr gplion
34-
12-
33-
30
200
400
600
800
2003
0.
6.1
03
24
Llmonatora
Compttin
Figure 23: Corrupted Gaussian Large Spherical Covariance: Two Variance Shell Clusters
We find similar results to the identity covariance case across the best estimators, again observing the near op- timal performance of QUE_low n and PGD, along with the slightly worse but still near optimal performance of LRV. As result of using the scaling data heuristic, ev wv_filtering_low.co degrades slightly. This is especially noticcable with DKK noise, where it does not converge to good_sample_mest error as data size increases. QUE_low_n appears to be less sensitive to the trace scaling heuristic, retaining its performance in these experiments. There is some variance among other estimators. Notably, lee_valiant_singg optimally across two variance shell corruption and in distribution noise with spherical covariance (its perfor- mance in these plots hidden amongst the best estimators which approximately match srod__smple_meet error); whereas it performs comparably worse across analogous noise distributions for identity covariance
performs near
35

---

Published in Transactions on Machine Learning Research (02/2025)
Gae
Sze Dimens 500, Comupiton Farcentage:
Dimensione: computtor Percentage: 0.10
cample roon 111
connd rinda Enter
coold mein
Eng
35.0
12.5
Murtenng Drerr
13.0
Dree
11.0
17.1
13
2.5
1958
2033
5000
200
533
Erice us Dimondions
Endor us Comupttion
Parcentago:
10-
255
ebc
1506
69
comptton
Figure 24: Corrupted Gaussian Large Spherical Covariance: DKK Noise
va 2m
es
Limensoos:: 500, Comuption "arcnttoe:
Clemerstons: 500, Coruption Nercentage: 0.10
set pic, renn Tra
coovt medis
copsl (erveed mean
20-
reddar oreess ETTP
15-
23-
32
0140 runqle paat Frrr
11-
2000
3033
Enoor va Dimmestoos stons
Erior us Comuution
Forcantage: 3.10
13
30-
12
20-
700
450
600
872
3002
5.0
Figure 25: Corrupted Gaussian Large Spherical Covariance: In Distribution Noise
data. Still, dee_valiant_simpe does not generally perform better in the spherical covariance case compared to the (known) identity covariance case; performing consistently worse than sample_mean compared to out- performing sample_ sample_mean mean except with large y) under identity covariance.
36

---

Published in Transactions on Machine Learning Research (02/2025)
vs Deta Cimenstora: 500,
Fime Cimerator:: Comupiiin 0.10
01-
cample room 1113
coald rinda
coard mean
Eng
Murttigg Droer
Drcer
35-
33-
2000
5335
2011
Erorr Dimonsions
Eroe Comupttinn
Com piton Parcontage.
Sto 500,
1907
0.1
comption
Figure 26: Corrupted Gaussian Large Spherical Covariance: Subtractive Noise
Varying a We rerun several experiments we vary from = 0.1 a 200 the coordinate wise standard deviation of the true covariance matrix and fix other variables as their default values. In particular, examine Additive Variance Shell Noise, DKK Noise, In Distribution Uniform Noise, and Two Variance Shell Clusters Noise. Results are shown in Figure 27 As expected, eπror tends to increase linearly with a. Generally, relative performance of the algorithms remains the same, with ev_filtering_low__ QUE_low_n error throughout. Surprisingly, LRV error does not grow linearly with a, consistently outperforring even geod_sample_ment mean with large enough choices of a. similar trend is also seen for coord_median but only across DKK Noise, in which it noticeably the best estimator with larger values of d.
and PGD nearly identically matching
37

---

Published in Transactions on Machine Learning Research (02/2025)
Error Coordinate Wise STD
Error Coordirate Wise S1D
Data Size: 500. Dimersions: Coruption Perce Perge: 0.10
Deta Size: 500. Dimensions: 500. Corruption Percentage: 0.10
sample Frir
metian
800
coord brimoed geomnericc
Icc Col.oolant stmpl Emor medien of moens the
Iv Error
litrinng los Trize
200
300
good pow__sampte_meet
125
200
25
300
750
315
Coordinate Wine
Coandimate STD
(a) Additive Variance Shell Noise
(b) DKK Noise
Wise
Data Size: 530, Dimersions: 500, Comuption Parcentage: 0.10
Data Size: 500, Dimeissions: 500, Comuption Percentage: 0.10
520
400
200
100
150
(c) In Distribution Noise
(d) Two Variance Shell Clusters
Figure 27: Corrupted Gaussian Large Spherical Covariance: Varying Coordinate Wise Standard Deviation
38

---

Published in Transactions on Machine Learning Research (02/2025)
A.4 Corrupted Gaussian Data Unknown Non Spherical Covariance
A.4.1 Unknown Diagonal Covariance
Here consider the performance of mean estimators on corrupted Gaussian data with unknown diagonal non-spherical covariance. We draw uncorrupted data from Na(u,, Σ) where is the all fives-vector and E has large diminishing covariance. In particular, the diagonal elements uniformly decrease from 25 to 0.1.
Noise Distributions We adapt the variance shell additive noise distribution to cluster outliers to be a standard deviation away from the true mean along every coordinate axis. That is, consider corrupted data distribution Q =Nalp', 1o with lu!, رyل Σ,, where Σ₁ is the jth diagonal clement in E, 14 is the jth coordinate of p', and M4 is the jth coordinate of the true mean ii results are shown in Figure 28 We adapt in distribution uniform noise to draw each coordinate j of a corrupted data point from Uniform (Pj, P4 + +2); results are shown in Figure 29 For large outlier noise we weight the distance of clusters That is, consider corrupted data distribution Q 0.7Na(p", ()) 1)0.3Naa 1) where Tr(E) Vd, and = 75 where 0 is the angle between μ² and p²:: results are shown in Figure 30 We also utilize subtractive noise which already works in this case; results are
from μ by Ir(2)
|| | 10
Tx Vd, || | = 20
d
shown in Figure 31
Error Size
Entr vs Dimensionn:: Comruttion Percentoge: C.10
Dimansioro: Corrustior Percentage: D.:0
UnTrenE moam
dimple LITTY
22-
median of moin EITCE Evir
18-
Frier
gontt curple
2000
202
218
Error va Dimaracons Comuutiion Parcentage:
Error Comruption
500
37.5
50
33.0
35-
25
00 201
400 500 1000
03
0.
Figure 28: Corrupted Gaussian Large Diminishing Diagonal Covariance: Additive Variance Shell Noise
Again, we find that QUE_low_r n and PGD nearly match good_sample, mean error across distributions, with LRV doing slightly worse but still significantly outperforming sample_mean mean. ev wu_filtering_low lo____ww_nn________________ still does among the best here, but, as in the large spherical covariance case, sees slight degradation a result of the scaling data heuristic. In particular, error does not clearly converge to srrod__ample_meeeee error as n increases over Additive Variance Shell Noise and In Distribution Noise. QUE_low does not encounter this issue despite employing the same heuristic to generalize to non-identity covariance data, suggesting that it is more robust to distributional assumptions. Additionally, median_of_meed performs best among simpler estimators, outperforming the sample_mes _mae with n z d and performing similarly to @erd__smpe__ee___ with sufficiently large n.
39

---

Published in Transactions on Machine Learning Research (02/2025)
vs Deta Sae
Fime vs
Eimensoon:: 500, "arcentage: 0.10
Cimerator:: Comupitin Percentage: 0.30
sample maon tny
cos1) ridda Ener
cand men
20-
mod Jun
22-
Iy Entr
J.attngg Jou' Dror
15-
33-
1000
2000
5335
2011
901
Error Dimonsions
Err Comuuttonn
500, Com pition Parcontage. 3.10
Stoo. 500, Dimersiont,
31-
12-
92-
21-
33-
2003
200
1937
5.
comptio
Figure 29: Corrupted Gaussian Large Diminishing Diagonal Covariance: In Distribution Noise
Size
va
Dimension:: 300, Coruptiinn Nercentage: 0.10
Dimnraaioos:: 530, Comsuption Percentape: 0.10
333
sergle 11841 Dror
370
coc mredan Shor
cocrd Lremmnd moin
lart
mediin moore Foer
Viaiigg EA Broo
glvd oirple yean Free
23
2003
5000
100
Friee Simerstens
Emror vs Comuptionn
Commttion Percentage:
3.0
360
32-
230
D11
150
20
200
430
630
800
1200
0.
0.7
02
Figure 30: Corrupted Gaussian Large Diminishing Diagonal Covariance: Large Outliers
Varying top Eigenvalue We rerun Additive Variance Shell Noise and In Distribution Noisc as we vary the squareroot of the top eigenvalue of the true covariance matrix, labeled as a, from 0.1 to 200. In partieular, the diagonal of the covariance will uniformly decrease from o² to 0.1. For every choice of a,
40

---

Published in Transactions on Machine Learning Research (02/2025)
vs Tetx Cimenstor:: 500,
Fime
Clmerators: Comupiiin Percentage: 0.10
33-
cample room 1113
coald rinda
coard mean
25
Eng
Drerr
Drce
33-
2000
5335
2011
Enor Dimonsions
Eroe Comupttinn
Com piton Parcontage.
1937
5.
compaion
Figure 31: Corrupted Gaussian Large Diminishing Diagonal Covariance: Subtractive Noise
the noise scaled described previously. These results shown in Figure 32 find that the relative performance of algorithms remains nearly identical throughout choices of
Like in the spherical case,
Eraar us Squareroot Top Eigenvalue 50C Curruption
Enror vs. Squa equererrot Top Exgerveeuee
Sizw:
0.10
200
stmpll meon
coard tiimes
400
gezmeniic median
Icc, valla semple Errr median, of mears tmtrr Error
300
800
or. Bitring Jox, Ener que, Jnc, Eiror
poel Entoe
200
200
goed cample meon Emor
100
300
25
50
100
125
150
375
200
30
75
500
125
150
175
200
Squarsroct Of Thp Eigervalue
Sigirtorctt Top Eiperratiio
(a) Additive Variance Shell Noise
(b) Distribution Noise
Figure 32: Corrupted Gaussian large diminishing diagonal covariance: Varying the square root of the top eigenvalue:
41

---

Published in Transactions on Machine Learning Research (02/2025)
A.4.2 Unconstrained Covariance
So far, we have only examined inlier data with diagonal covariance matrices. However, in line with the intuition that there is nothing inherently special about the standard orthonormal basis, we hope for a robust estimator to work well regardless of the choice of coordinate axis. Since the covariance matrix is always symmetric, it is also diagonalizable by taking the cigenvectors as the orthonormal basis. Then, any possible data distribution over unconstrained covariance can be framed as a dala distribution over a diagonal malrix by using these eigenvectors as the orthonormal basis. As a result, any robust estimator that does not leverage the standard orthonormal basis should perform equally well on unconstrained covariance. However, this does not necessarily hold for the estimators that we examine. We employ a trace estimate to adapt ev w__filtering_lw__r_ and QUE_low_n to the unknown covariance case. coord_ _erdinnnnn,, coord cord__trimed____ _mean, and median_of_ means all directly utilize coordinate wise calculations. LRV utilizes trace estimate when downweighting points. In this section, we evaluate the performance of robust mcan estimators over data with non-diagonal covariance matrices.
Rotated Data Noise Because the covariance matrix is always symmetric, it is diagonalizable, and exper- iments over unconstrained covariance can be framed as an ablation on noise distributions over inliers with diagonal covariances. We reuse data and noise distributions, but randomly rotate everything before esti- mation, resulting in unconstrained true covariances and appropriately difficult noise distributions. Random rotation is implemented by generating standard normal matrix and utilizing its QR decomposition. We examine the performance on Rotated Identity Covariance with DKK Noise in Figure 33 Rotated Identity Covariance with Subtractive Noise in Figure 34 Rotated Large Spherical Covariance with Additive Variance Shell Noise (with coordinale-wise standard deviation a = 5) in Figure 35 and Rotated Large Diminishing Covariance with Additive Variance Shell Noise (with squareroot of the top eigenvalue o = 5) in Figure 36 As in the original experiments, we sot the true mean, 14; to be the all-fives vector.
(Tata Soy Eimenstona: 500, Comppition Parcentages 0.15
Frint Data
Dimmensions: 500, Compuptton Farzentage:
EOGTORII erecken Fror
que lou, Crrie
Egi esror
good wwwptt
1-
2032
1000
5000
309
Deta St2
3ita S28
Dats Stze: 5CC, Corr-pi Corr-plten Svecentinge 3.10
Data Sue 505, Dimendann 500
35
10-
10
440
35
10-
15
33c
3.5
Figure 33: Corrupted Rotated Identity Covariance
DKK Noise
We find nearly identical results among the best estimators to the corresponding non-rotated data experiment. While many of these algorithms induce a bias to the coordinate axis, they are not enough significantly skew results in the distributions that we examine. There is some variation between coord_median and mean with the corresponding non-rotated data experiments, but major changes in their
42

---

Published in Transactions on Machine Learning Research (02/2025)
Gae
Dimensione: compuptrr Percentage: 0.10
Dimens 500, Comupiton Farcentage: 0.13
cample room 1T1
connd rinda EnTEr
coold mein
Eng
Muuttng Droer
Drcer
1958
2033
5000
Erice us Dimondions
Entor us Comuuphon
Parcentago
2.2-
150
6.0
125
255
ebc
1608
00
0.3
comption
Figure 34: Corrupted Rotated Identity Covariance Subtractive Noisc
Dimensiors: 500, Cortuption Percentage: 0.10
Dimersiors: 500. comuptoo Necentage: 0.10
menn
25-
georddir reskin Enr
lec La mak Gver
madar Franss
35-
giec pample yoant Eiter
Deta
Data Size
Ulmmn stomes
Data Slat: 500, Comupiion Parcentage: 0.10
Data Slol: 500, Dimersions: 500
23-
13-
12
01
32
01-
23
Figure 35: Corrupted Rotated Large Spherical Covariance Additive Variance Shell Noise
trends. There is no such variation for QUE_low_n, ww_filtering_lo_rn or median of means among the settings that we test.
43

---

Published in Transactions on Machine Learning Research (02/2025)
Fimz ys Data Sie
Fror en Pata Siae
coont mennn Eng
coerd nterres mesn oner
geaveticc medion Eoor
Drcer
er. Teinng Iain Ergrr
que Der axme
33-
54
X009
4002
219
Data
Data Sime
Data Sizo. 500, paon Porcontage. 0.10
Data Sico. 500, Direndions, 500
12-
11-
12
D10
compatinn
Figure 36: Corrupted Rotated Large Diminishing Covariance Additive Variance Shell Noise
44

---

Published in Transactions on Machine Learning Research (02/2025)
A.5 Hyperparameter Tuning
In this section we tune the hyperparameters of some of the most interesting algorithms, median_of means, LRV, ev ww_filterinng_owec
and PGD. We utilize the best hyperparameters in this section across all other exper- iments. In general, we find that nonc of these algorithms are overly sensitive to choices of hyperparamaters, as long as they lay within a reasonable range.
We evaluate performance over a subset of corrupted data distributions previously discussed: Identity Covariance with DKK Noise; Identity Covariance with In Distribution Noise; Large Spherical Covariance with Variance Shell Additive Noise; Large Diminishing Covariance with Variance Shell Additive Noise. We manually pick the hyperparameters that achieve the best performance across these distributions, or when similar use default ones from the corresponding paper.
Median of how many means? Here we explore the parameter in median_ _ettian_o__meas means algorithm. This parameter controls the number of chunks that we split the data into; then we take the median of k means determined by these chunks. We vary ls in the set [3. 5, 10, 15, 20, 30). For the case where n < k, we simply set k n. These results are shown in Figures 37 38 39
We find that although there is not always an obvious choice for k, that k = 10 tends to perform well throughout most scttings. However, we find that this and larger choices of k are more prone to error as v increases than smaller choices of k. Approximately when > 0.15, k = 3 becomes the best choice of k. However, with smaller corruption, such as 1j 0.10 which we generally test, h = 3 performs notably worse, making = 10 a better choice. Since we utilize = 0.1 as a default value, we set = 10 throughout our experiments.
40
Ener Data Size
Emoo va Data Sine Percentage:
Ieecentage:
Erar
medion et nAMAnS Enar
recher al more 13 Ertr
Le Ltwerp-1 Ener
11:00
A001
.0.0
43:23
1400
300
Sta
Error Camuption
Dats Ca cummitton ion Percartage 0.10
Dats Siw: 100, Dimensions:
2.3
2.-
35
10-
2-
15
200
RDC
0.1
mensione
Compptiin
Figure 37: Median Of Means Number of Chunks k: Identity Covariance, DKK Noise
45

---

Published in Transactions on Machine Learning Research (02/2025)
Exer us Dato Size
Erioe Orco, Size
Fercentage:
Dinenns
Percentage:
Engr
wedion of ne krter
redia L Linoer 12 brit
ne meoe tntr
1000
3000
4033
100
233
480
Dute Skze
3ite
Data 20: 500. Comuuccinn
wwr.coonnnnnntttaaoooooooo Parcentage 0.30
Data 5:ku 200. Dimonsions. 500
1.5
2.-
10
03
700
mencions
Geraati
Figure 38: Median Of Means Number of Chunks k: Identity Covariance, In Distribution Noise
Emeor va Duka su Dimensions: 500, Comuptio Narcentage:
Erur Data Sieu
Cimerstors: comuptin Nercentage:
23
reees 10
reder perre tre
20
32-
1000
2000
3000
4000
5022
120
200
400
Duta 5:00
Deta Suo
Data Stac: 500, Com, pitin Farcentage: 3.10
Dato Sioe: 500, Dimeersoons: 500
25
33*
comprion
Figure 39: Median Of Means Number of Chunks A: Large Spherical Covariance, Additive Variance Shell Noise
46

---

Published in Transactions on Machine Learning Research (02/2025)
Emeor Data S00, Comuptio "ercentager
Siep Cimersions: Comuption Nercentage:
reres tee
meter Je tre
20-
20
0
2000
3000
5022
200
400
Deta 5.00
Data Sui
Data Sloc: 500, Com Com-piion Parcantage: 3.10
Data Siot: 500, Dimersions:: 500
23-
comption
Figure 40: Median Of Means NNmmeer of Chunks k: Large Diminishing Covariance, Additive Variance Shell Noise
47

---

Published in Transactions on Machine Learning Research (02/2025)
LRV Weighting Procedure Here explore the weighting procedure in LRV. First, examine the parameter C' in the weighting procedure for LRV. This parameter is used when we calculate weights for each [ - all2/(C'= + ))). We vary C' in the set [0.1, 1, 5, 10, 20, 50). Results are shown 44 We notice that performance may degrade with choices of C' that are too high or 0.5 and 50. We also notice that smaller choices of C tend to degrade worse with throughout our experiments, which consistently performs among the best throughout the hyperparaameter trials that we test, and is the default value used the original author's implementation of LRV. Although there are cases where larger choices of C' noticeably outperform C' = 1, this is not robust as such choices may perform meaningfully worse over different noise distributions. For example, C' = 20 noticeably outperforms C' = 1 over Large Diminishing Covariance with Additive Variance Shell Noise, especially with larger but performs much worse over Identity Covariance with DKK Noise and Large Spherical Covariance with Additive Variance Shell Noise.
point, I2 as exp
in Figures 41
too low, such as with C
greater corruption. To strike balance, we select
Dimensiens, Dade
Prru Data Dimensions: 520, uption Percentage:
Comuptio Parcentage lage:
Erar
red Finor
-
601
1000
2000
3003
40300
3000
100
260
360
500
Oete
Dataa Stee: 500, Comuption Tarcentage: 0.10
Data Stow: 530, Dimensionnc 500
2.0
135
1.5-
1.5
1000
1.4
12
1.-
<00
600
02
64
Dinooitra
Can Up3eT
Figure 41: LRV Choice Of C: Identity Covariance, DKK Noise
48

---

Published in Transactions on Machine Learning Research (02/2025)
Olmensions: 500 Comuption Peroentage 0.30
Dimensions: 500, Conuution Percertage: 0.30
Ind Error
Iovic Lntr
Erer
5000
2000
3002
4000
3000
200
E313
EX0 542
Dimensionn
Data Saw 500. Cortupt Parcentage 0.30
Data Suor 500, Dimonsioro: 503
14-
10
12-
10-
02
LEmendons
Figure 42: LRV Choice Of C: Identity Covariance, In Distribution Noise
Dimenslors: 500, Comuption Sercentron: 6.10
Dimensions: 500, Cornuction Percentage 0.30
Iv Enrr
33.
Iv-i Lnter
In- krar
33
2226
3000
3022
200
420
0003
5:00
0S03 5000
Tata Siee 500, Com pilio Eneneetagee 5
Data Cpo- 120, Dimeedooe 100
Ul ora
Compaton
Figure 43: LRV CChoice Of C: Large Spherical Covariance, Additive Variance Shell Noise
49

---

Published in Transactions on Machine Learning Research (02/2025)
Emer
Dimeosions: 50C, Comuutar Pereetagge
Dimenstono: 500, Comippinn Peccentage: 0.10
28
1r--1 Oroe
25
24
325
18
1000
2000
2000
219
Data Size
Data Sime
Data Site: 503, Coruutiin Percentago: 0.10
Data Size: 500, Direndibns 500
compatinn
Figure 44: LRV Choice Of C: Large Diminishing Covariance, Additive Variance Shell Noise
50

---

Published in Transactions on Machine Learning Research (02/2025)
We additionally compare the weighting procedure of LRV that consider with one meant for general distributions discussed by Lai et. al. (2016). Rather than downweighting outliers, this alternate procedure completely prunes outliers by calculating point, μ', analogous to the coordinate wise median, finding a ball centered at y that contains 1 1-7 percentage of data points, and throwing away all points outside of this ball.
48 This general weighting procedure performs meaningfully worse than the Gaussian weighting procedure and degrades significantly worse with larger corruption across all of the distributions considered. However, achieves similar results data size increases. Since focus on the low data size regime and synthetic data with Gaussian inliers, only evaluate LRV with Gaussian-baset outlier downweighting.
Results are shown Figures 45
Dimensions, 500, Compptio Percentage: 0.10
Dimonsians: 500, Compption Percentage: 0.10
3003
220
20ta 5.00
Data Size: 500, Comuption Parcontage: 0.10
scoms
Data Size: 500, Dimendencs 500
125
650
Figure 45: LRV Gaussian Vs General Weighting: Identity Covariance, DKK Noise
51

---

Published in Transactions on Machine Learning Research (02/2025)
Erie Duto Sige
Erus Duko Sue
Dimensions: Porcertage:
500, Comupatin Parcentage:
2300
2003
4000
3000
305
333
DUG 582
D20
Ertor Comuptiinn
Data Size: 500, Comauttion Percentage: 0.10
Duta Size: 500, Dimons 500
150-
325
33.0
635
25
35-
LONEENGION
comppton
Figure 46: LRV Gaussian Vs General Weighting: Identity Covariance, In Distribution Noise
Fricy us Data zn
Eirr vs Dato Size
Percentage
22-
2>
12-
2000
3003
200
1000
920
1303 size
Data Sae 53C, Corpettin Petrentaree 0.10
Suel 500, Dimersionn 500
323
1-
2
Sror
23
01
Uiisansions
Gemaptton
Figure 47: LRV Gaussian Vs General Weighting: Large Spherical Covariance, Additive Variance Shell Noise
52

---

Published in Transactions on Machine Learning Research (02/2025)
Ereor Date Sa
Siep
S00, Compptio "Mrcentager
Cimerstons: Comuption Nercentage:
10-
35-
23-
2000
3000
5023
200
400
Deta 5.00
Data Sui
Data Slac: 500, Com Com-plion Parcantage: 3.10
Data Siet: 500, Dimersions:: 500
23
33
comption
Figure 48: LRV Gaussian Vs General Weighting: Large Diminishing Covariance, Additive Variance Shell Noise
53

---

Published in Transactions on Machine Learning Research (02/2025)
Eigenvalue Pruning Tail Threshold Here we explore the pruming routine in ev filtering_low_____ First, examine the parameter in the pruning threshold for ev_ lifftteiing_looe lo____oo______ Y weights the expectation that the Gaussian concentration inequality gives for how many points will surpass a certain value; larger values correspond to less aggressive pruning. We vary y in the set [0.5, 1, 2.5, 5, 10, 20, 50]. Results are shown in 52 We find that using values 7 that are too smalll result in significantly worse error. sample_mes mean because the pruning threshold is sensitive, performing significantly worse than all other choices of 7; determines all data to be outliers. We note that when all data is determined to be outliers, simply return sample_mean However, using reasonably sized results in mostly similar performance across distributions. Notably, larger values of tend to perform better over large diminishing covariance with additive variance shell noise, especially with larger n. We select = 5 throughout our experiments.
Figures 49 50
Setting 7 0.5 or 7= both achieve performance identical
Size
Dimerslons: 500 Compotion Percensace 0.30
Dimens one 500, Comruption Percentage: 0.10
ey tiornng ni oier
2-
1--
1000
3000
4033
100
480
Dute 9ize
Dlte 9:00
Dimceecions
Data 5 500. Corrugtion Parcentage 0.30
Data 5:0: 300. Dimansions: 500
20
15
10
700
compaton
Figure 49: Eigenvalue Pruning- Choice Of y: Identity Covariance, DKK Noise
54

---

Published in Transactions on Machine Learning Research (02/2025)
Dimendion: 500, Compption Parcartage: 0.10
Dimonsiers, 500, Compltoon Porcontage: 0.10
low
flteiin los
On Jufioriig ig Jou enar
2003
3000
40000
3000
223
D002 5iz0
300 500
Data Stey: 500, Comuption Percinlage: 3.10
Data Size 500, Dimeniors: 500
35
10
25
60
composs
Figure 50: Eigenvalue Pruning Choice Of 7: Identity Covariance, In Distribution Noise
Dimenslors: 500, Comuptiin Sercentron: 6.10
Dinensions: 500, Cornuction Percentage 0.10
17
Mu Jnc_ 2. Oncr
fuffters Jos so Lit
fltering inar
42-
0
2326
3000
3022
200
420
0003
0S03 5000
Tats Siee 500, Com piloo Bneneetaage 3
Dats Cpp- 120, Dimeedooe 100
34-
12
Comption
Figure 51: Eigenvalue Pruning Choice Of 7: Large Spherical Covariance, Additive Variance Shell Noise
55

---

Published in Transactions on Machine Learning Research (02/2025)
Emer
Dimeosions: 50C, Comuatirr Perrettage:
Dimenstono: 500, Comippiinn Peccentage: 0.10
Era
ey Viarnng inn n5 oner
54
12.
0
23
3.2
1000
2000
2000
219
Data Size
Data Sime
Data Site: 503, Coruptiio Percentags: 0.10
Data Size: 500, Dww.ndbnnnns 500
12
10-
22-
compatinn
Figure 52: Eigenvalue Pruning- Choice Of 7: Large Diminishing Covariance, Additive Variance Shell Noise
56

---

Published in Transactions on Machine Learning Research (02/2025)
We also explore ev __ffiterringlownn using two alternate pruning methods not explicitly based the Gaussian assumption the current one is: randomized pruning and fixed pruning.- Randomized Pruning removes points based on a random scaling of the largest deviation in the dataset. Define I as the largest deviation of a point projected onto the top cigenvector from the median of the projected points. Draw Z from the distribution on [) with probability density function 2x. Then, prune all points whose projected least TZ. This randomized pruning method derived from the mean estimation algorithı for unknown covariance distributions by Diakonikolas et al. (2017a). Fixed Pruming simply prunes the 0.57 percentage of points whose projection onto the top eigenvector is furthest from the median of the projected points at every iteration. This is identical to the pruning method in QUE_low_n, n, with projected deviations being used as 'outlier scores', instead of the quantum entropy scores used in QUE_low_" Results are shown in Figures 53
distance onto the top cigenvector
We find that both randomized and fixed pruning are able to match or slightly outperform the standard Gaussian pruning method. However, we note that unlike in QUE_low_n nn fixed pruning could potentially result in catastrophic crror. In particular, if corruption is uniformly distributed across O(d) orthogonal clusters, then ev =f_flterring_oww runs to return outlier, since it only prune in direction at once. But with fixed pruning, each iteration will prune too many outliers in each direction. We only evaluate Gaussian pruning to follow the conventions of Diakonikolas et al. 201 7
54
55
56
may take (
Demensions: 300 Comuption Parcentage 0.30
Dimensions: 300, Corrustion Parceriage: 0.10
Viarng, EAJ Sre
Dita Sba
Campttion
Siww: 500, Campption Percentage: 0.30
Date 500, 503
16-
12-
14-
80
66-
33.
20-
62
200
2003
Figure 53: Eigenvalue Pruning Pruning Method: Identity Covariance, DKK Noise
57

---

Published in Transactions on Machine Learning Research (02/2025)
Erg Dula Sige
Eist Dutu Sieo
Olmendions: 500 Comupoin Rercentage:
Dimendlars Compssion Porcentage:
eiliistng, anJ.wwdde Striine ca flndarr
5000
2000
3002
40C0
5000
200
E31 S4z0
EX0 542
Dota Size 560, Compption Rercentoge: 0.30
Data Siow: 500, Dimersions: 500
36
14-
10-
24-
13
56
33-
10
02-
230
cdmapton
Figure 54: Eigenvalue Pruning- Pruning Method: Identity Covariance, In Distribution Noise
Exoo vs Data Sine
Eroo vs Data Size Compption Percentage:
en So ine bo.. www. untrr
20.0
15-
22.0
33
340
1050
2028
4002
5020
302
230
302
412
322
0003 size
DSt 5000
aption
Twta Size: 920, Com gitti Procestige:
Data 520, Dirreniton 100
5.5
g
30-
400
800
Lemonsore
Compptin
Figure 55: Figenvalue Pruning-
Pruning Method: Large Spherical Covariance, Additive Variance Shell Noise
58

---

Published in Transactions on Machine Learning Research (02/2025)
Eror Data
Emeo Data 520, Coruiption Percentape:
Dimersions: Comuption Parcentage:
AL Jandon
25-
25
0
13-
13
2000
3003
4030
100
360
403
530
Octa 5.z
Data Sia
Data Siee: 500, Comppttio Nercentage: 0.10
Data Size: 500, Dimenseere: 500
50-
45
1.0
Figure 56: Eigenvalue Pruning- Pruning Method: Large Diminishing Covariance, Additive Variance Shell Noise
59

---

Published in Transactions on Machine Learning Research (02/2025)
Projected Gradient Descent Iterations Here we explore the number of iterations parameter, y for PGD. We choose values for in the set [1,5, 10, 15, 20). Results are shown in Figures 57
We find that low choices of result in significantly worse performance, while higher choices perform roughly equally. Notably, when y is set equal to 10, PGD performs much worse over Identity Covariance with DKK Noise, especially under large n, despite this choice of 7 performing among the best across of her distributions. This suggests that larger choices of may be necessary for PGD be robust across different corruption schemes. We note that the runtime of PGD increases approximately linearly with respect 7 there is a meaningful tradeoff when using larger values of y. We set 7 15 across experiments because it is the lowest y that performs among the best across the distributions tested.
58
59
60
Olmencions: 500 Comppoinn Porcentage 0.30
Dimensions: 500. Comussoo Percentage: 0.10
pg0-3
50
15-
Pot-15 Finer
pyyh21 more
25-
20-
100
Data Stz: 500. Compptin Percentoge: 0.30
Drmerscree
Data Sizo: 500, Dimonsions: 503
2.5
210
10-
05
Figure 57: Projected Gradient Descent Number Of Iterations Identity Covariance, DKK Noise
60

---

Published in Transactions on Machine Learning Research (02/2025)
Eroo ys Data Sue
Eroy us Dota Sie
Percentage:
Percentage:
roa-34 Crix
Pod-1
3003
2000
3000
4000
5000
209
Data Size
Jute Siee
Dimensions
Data Sipe. 500, Com-paaon Parcentage 3.10
Data Sec. 590. Dimensors, 500
10-
10
23-
35-
85
60
Camppa1
Figure 58: Projected Gradient Descent Number Of Iterations
Identity Covariance, In Distribution Noise
Sirrr Dake
Ensr Data 500, Comppioon Percantage:
Dimensions: Comuator Percentage:
ag5 10 kno
325
2.5
2000
3000
1000
218
303
530
Duto Sun
Data Six
Data Siee: 500, Comuuttonn Percentage: 0.30
Deta Slat: 500, Direndioos: 500
12-
compaton
Figure 59: Projected Gradient Descent Number Of Iterations ?: Large Spherical Covariance, Additive Variance Shell Noise
61

---

Published in Transactions on Machine Learning Research (02/2025)
Ener Data
Dimansiorn:
Percentage:
Cemppion Percentage:
375
23
ago kno
1g733 tm
20
125-
0
15
2.5
2000
3000
1000
300
530
Data Sue
Data Sux
Data See: Comuption Percentage: 0.30
Duta Slat: 500, Oerendions:: 500
12
25-
Oiressions
compaion
Figure 60: Projected Gradient Descent Number Of Iterations Large Diminishing Covariance, Additive Variance Shell Noise
62

---

Published in Transactions on Machine Learning Research (02/2025)
A.6 Robustness To Expected Corruption
Ertor vs Expected Computtin Data Sieec 500. Dineraion:: 500, Camuption caniagg:
Emeer Espacted Car
Date Size: 500. Dimension:: 500, Comuption Percontage: 0.20
sarrple IT rean Dror
cood medaar cear
16-
coard mnmmeedd maan
geometis moken
un zerpie
15-
fitm Siee
144
Lenor
agd Frrs
11-
0.25
comppton
(a) Identity Covariance DKK Noise
(b) Identity Covariance Subtractive Noise
Emer us Expacted Comuption
Date suew 500, Limetsioon 500, Parcentage: 0.20
Emor Corruption
25.0
20-
12.
10.2
15-
255
0.00
0.03
0.20
0.5
0.02
0.50
0.20
0.25
0.20
Expected
(c) Large Spherical Covariance Additive Variance
(d) Large Diminishing Covariance Additive Vari- ance Shell Noise
Shell Noise
Figure 61: Robustness To Expected Corruption: Error vs Expected Corruption T
We examine robustness to expected corruption, т. This is hyperparameter for ev__filternnglow__ QUE_low_n, PGD, lee_ role as a slack term in the filtering step. In QUE_low_n, 7 controls the mumber of points that are pruned in every iteration of the algorithm, but the mumber of iterations is unbounded. In lee de__valant_simple and T explicitly controls the amount of data that is pruned total. In PGD, t rols the space of feasible outlier weights. We evaluate error as expected corruption, T, varies from 0.01 to == 0.46 with true corruption fixed as = 0.20. Otherwise, the experiment sctup remains the same seen previously. As in Appendix(A.5 we replicate experiments over Identity Covariance with DKK Noise and with Subtractive Noise; Large Spherical Covariance with Additive Variance Shell Noise; and Large Diminishing Covariance with Additive Variance Shell Noise. These results are shown in Figure 61 with all estimators inchuded for reference. We include QUE_low_n with and without early halting.
and ciird__rimmdd_eee _mann In ev w__fittering__ow t only plays a soft
We find that most estimators perform nearly identically regardless of the choice of r, except for PGD, which performs nearly identically when 7 is an upper bound on true corruption 1) but degrades with underestimates of 17. QUE_low__ without early halting performs well throughout choices of T. With smaller choices of т, it will prune significantly less points at each iteration, but will run for more iterations until the corruption detection threshold is passed, while for larger choices of т, it will prune more points at cach itcration, but will run for less ilerations until the corruption delection threshold is passed. QUE_low_n with early halting, which is used throughout the real world experiments, sees degradation with underestimates of т, but identical performance with overestimates. ev ww_filtering_oon _o_____oww___________ also performs nearly identically regardless of the choice of 7, as expected by the soft dependency of the pruning threshold on 7. lee see_valant__smppe and trimmedd mean both degrade noticeably as increases over Identity Covariance data with Subtractive
63

---

Published in Transactions on Machine Learning Research (02/2025)
Noise and Large Diminishing Covariance data with Additive Variance Shell Noise; in both cases yielding error worse than sample_mean the more points they prane. Surprisingly, over Large Spherical Covariance with Additive Variance Shell Noise, ree_valintt_smpae nearly exactly matches the performance of PGD, except with slightly worse degradation with large overestimates of T.
A.7 Image Embedding Experiments
We evaluate algorithms on the problem of estimating the mean of embeddings of images generated by deep pretrained image models. As in the LLM experiment, we first examine the problem of mean estimation of image embeddings belonging to the same category, reporting LOOCV error. We then examine corrupted distribution where images belonging to one category are considered inliers and those belonging to another are considered outliers. We utilize set of images of cats and dogs from the CIFARIO dataset. (Krizhevsky 2009). We embed these images using 4 deep pretrained image models of varying embedding dimensions: ResNet-18, ResNet-50 (He et al.) 2015), MobileNet V3 (Howard et al. 2019), and EffecientNet BO (Tan & 2020). ResNet-18 has an embedding dimension of 512, MobileNet V3 has one of 960, Elficient B0 has one of 1280, and ResNet-50 has one of 2048.
Le
Common Category Images Here we examine LOOCV error vs data size on embeddings of images of cals. We vary data size from n 10 experiment, fixing expected corruption 17 0.1, employing the trace scaling heuristic and QUE_low_n, the halting heuristic on QUE_low_n, and averaging results over runs. We note that, as in the LLM experiments, employing the halting heuristic on ev wy_fittering_oov Jow_n does not improve performance. Results are shown in Figure 62
1000. Otherwise, experiments are run identically to the LLM
ev w__flttering_low_.
As in the LLM experiment, we observe that no algorithm significantly outperforms sample_mes mean, despite the nontrivial LOOCV error in each setting. As in the LLM experiment, ev_ filteringglow. tends to perform worse than other algorithms, which is unsurprising given its sensitivity to knowledge of the true covariance. Other robust mean estimation algorithms, including DUE_low_ny n, perform near identically to sample_mest mean.
Corrupted Images For the corrupted case, we draw data X P1 where the inlier distribution, P, consists of embeddings of images of cats, and the outlier distribution, Q, consists of embeddings of images of dogs. We fix data size = 1000 to focus on the z d and n < d regime. Otherwise, the experimental setup is idantical to in the LLM experiments. Results are shown in Figure 63
These results demonstrate similar trends to the LLM experiment. One key difference that woord_trimmed trimmedd_mean mean performs much worse than sample_meat here, compared to the LLM experiment where it tends to slightly outperform sample_mean. This suggests that naive pruning does not work well in this setting outliers are not obvious, reinforcing that this is difficult setting for robust mean esti- mation. Nonetheless, several robust mean estimators are able to perform well in this case. In particular, QUE_low_n is again the strongest performer, noticeably outperforming all other estimators and nearly matching good_sample_mean error across settings. Notably, this strong performance occurs even with n much less than d, with relative performance remaining the same even with n = 1000 and = 2048 in the ResNet-50 embedding case. Among the best estimators in the synthetic data case, PGD tends to perform similarly to sample_mean, except with large enough corruption across MobileNet V3 embeddings where it outperforms sample_mean LRV tends to perform similarly to sample_mes _man except under low corruption, where it noticably degrades; and ev ww_fittering_low iments, median Lettiinnoo___mssss____ means outperforms robust estimators that tend to perform better in the synthetic data cases. However, lee__ valiant_ poliant_simppl no longer performs near optimally, suggesting its sensitivity to distribu- tional assumptions, as expected due to its general poor performance over synthetic data experiments.
fails catastrophically throughout. As in the LLM exper-
Corruption vs Data Size We repeat experiments over the same corrupted data scheme but examine error vs data size. We fix true corruption 7) = 0.1, set expected corruption = n7, and vary data size n. We examine the performance of all estimators with data size ranging from = 100 to 5000. We additionally provide zoomed in plot, examining the performance of estimators excluding ev f_fffferrin_ooww..__ n, goord_median, and
64

---

Published in Transactions on Machine Learning Research (02/2025)
LOOCV Error vs Data Size
LOOCV Error vs Data Size
Dimensions: 512, Expected Corruption: 0.10
Dimensions: 960, Expected Corruption: 0.10
sample_mean
mean
coord_
...
coord trimmed mean
geometric,
9.75
lee valiaat_simpll
LOOCY trror
15.0
It
9.50
wv_fiteringglow
que low_n n
LOOEV
9.25
14.5
Pgd
9.00
14.0
8.75
8.50
200
400
1000
200
600
800
1000
(a) Net-11 18 Embeddings
(b) MobileNet V3 Embeddings
LOOCV vs
LOOCV Dimensions: 2048, Expected Corruption:
Dimensions: 1280, Expected Corruption:
14.50
14.25
16.5
LOOCV Erer
LOOCV Enter
16.0
13.75
13.25
13.00
15.0
12.75
200
400
600
800
1000
200
400
600
800
1000
Data Size
Data Size
(c) EfficientNet BO Embeddings
(d) ResNet-50 Embeddings
Figure 62: LOOCV Error on Cat Image Embeddings
mean which all fail catastrophically with data size from n 100 to 1000. Results
are shown in Figure 64
We find that the relative performance of algorithms remains similar across data sizes. Particularly, even with very large such as = 5000 and = 512 under ResNet-18 Embeddings, only QUE_low_n and median_of_mean _e_ans___s___eaa_ consistently outperform sample_mean, PGD, LRV, and lee_ valiant_simple tend to perform slightly worse than sample_men mean. ev filttering_lw___ fails catastrophically regardless of data size, though the error stabilizes with larger n. These results suggest that the weakness of robust mcan estimators over real world data distributions not just confined to the low data size regime. Yet. again, we find that QUE_low_n is the best performer, outperforming all other estimators and achieving near optimal performance throughout settings. Additionally, median_od of _mans does not show this same sensitivity to distributional assumptions as other estimators, and as in the synthetic data experiments, tends to perform near optimally with large enough n.
65

---

Published in Transactions on Machine Learning Research (02/2025)
Error V5 Corruption
Error Corruption
Data Size: 1000, Dimensions: 512
Data Size: 1000, Dimensions: 960
sample_rm mean
2.00
35
coord trimmedd_eee mean
1.75
30
geometric
Ice valiant, simple
1.50
of
fittering_oww__ que low_lo Pgd
2.0
1.5
good sample_mepeeeeeeeeeeeeeeeeeeesse mean
0.5
0.25
0.0
0.00
0.00
0.10
0.25
0.40
0.05
0.10
0.20 Corruption
0.35
Corruption
(a) Res 18 Embeddings
(b) MobileNet v3 Embeddings
vs Corruption
Size: 1000, Dimensions:
Error va Corruption Data Size: 1000, Dimeensons: 2046
semple, mevn firer
meesannelanncco Erro
10
mediin Enior Erior
OV fitering low treer
qUR_ioN Jon_r Brier
ento
meen Enor
15
1-
0.5
0.-
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.03
0.01
0.10
0.15
2320
C.75
0.30
0.35
0.40
Corruption
Corruption
(c) EfficientNet BO Embeddings
(d) ResNet-50 Embeddings
Figure 63: Error on Cat Image Emnbeddings Corrupted with Dog Image Embeddings
Enior vs Data Size Eimersions: 512, Corapttion Peccentage:
Dimensions: 512, comuptiin Percantage: 0.10
sarpll mean
modd_rredaan
1.4
median maans fror brtrr
1.4
fiternn Iox Bro qui lew Uneer
pgll Ericr
good oonple moat
1.2
0.1
0.4
200
432
stb
1000
Dalaa
Date Suo
(a) ResNet-18 Embeddings
66

---

Published in Transactions on Machine Learning Research (02/2025)
Dimersions, 960, Corruption CoruptiooForccetttt Porcentage 0.10
Ener Data Sizw
Dimensins: 960. Comuptio Parcantage: 0.10
35-
caerd meddin
parple, jrman
moan
geamantic
25.
20-
quejeN, beor
wwt.rrrrgggaoo Jouu Bro
pao Erier
good mean
0.8
1.5
0.4
300
200
450
000
1000
Data Lite
(b) MobileNet V3 Embeddings
Uimersions: 1280, Corruption Parcentage 0.10
Emorvs Data Size
1280, Currup Peroelapee 0.30
Cato
sempli monn brie
coord medin Lrer
coord mean
1.2
Hitering Cra-
pall Erice
good sample mean Erar
04
0.5
3000
2000
200
410
ado
1000
Data 1it
EfficientNet Embeddings
Uhmenssons: 2045, Corruption Percentage 0.1U
Data Size
Dimmtssions: 2048, Serruptinn Nertanlape: O
us
Ser pl_ monr bri
2.00
coord medion erter
coord tommed mean
gonnettie
130
Decr
3 1.5
palil Erist
good sample mean Enar
1.0
3000
2000
400
660
903
1000
TW's S3A
(d) ResNet-50 Emneedingg
Figure 64: Error Vs Data Size on Corrupted Image Data
67

---

Published in Transactions on Machine Learning Research (02/2025)
LOOCV Error vs Data Size Dimensions: 50, Expected Corruption: 0.10
LOOCV Error vs Data Size
Dimensions: 100, Expected Corruption: 0.10
sample_mean
mean
4.1
...
coord trimmed mean
geometric,
lee valiaat__impl
33
LOOCY Error
It
LOOCV Emor
ro_fittrinng_oee
que Jow_n n
Pgd
31
3.7
3.6
100
60
100
(a) 50 Dimensional Embeddings
(b) 100 Dimensional Embeddings
LOOCV vs Dimensions: Expected Corruption:
LOOCV
Dimensions: 300, Expected Corruption:
52-
5.1
LOOCV Error
LOOCV Ener
5.0
8,5
4.9
8,4
4.8
4,7
4.2
4.6
20
90
100
20
40
60
80
100
Data Size
Data Size
(c) 200 Dimensional Embeddings
(d) 300 Dimensional Embeddings
Figure 65: LOOCV Error on 'Pleasant' GloVe Embeddings
A.8 Word Embedding Experiments
We further evaluate algorithms on the problem of estimating the mean of non attention based embeddings of words. As in the LLM experiment, we first examine the problem of mean estimation over words belonging in the same category, reporting LOOCV error. We then examine a corrupted distribution where words belonging to one category are considered inliers and those belonging to another are considered outliers. We examine four different pretrained GloVe (Pennington al. 2014) models from GluonNLFU generating 50, 100, 200, and 300 dimensional embeddings. We utilize datasets of 100 pleasant words and 100 unpleasant words from Aboagye et al. 2023 The very limited data size available under this setting provides valuable real world test for robust estimators under low data size.
Common Category Words Here examine LOOCV error data size embeddings of "pleasant" words. Experiments are run identically to the LLM experiment, employing the trace scaling heuristic on ev ww_fittering_lo__ and QUE_low_n the halting heuristic on QUE_low_n, and averaging results over 5 runs. Results are shown in Figure 65
68

---

Published in Transactions on Machine Learning Research (02/2025)
Error vs Corruption
Error vs Corruption
Data Size: 100, Dimensions: 50
Data Size: 100, Dimensions: 100
sample_me mean
coord median
coord trimmed mean
1.-
geometric_ median
lee simpe
0.8
IV
W__fitering_oww
que que_lownn low_n
0.6
Pgd
good sample, mean
0.00
0.10
0.15
0.25 Corruption
0.30
0.40
0.00
0.05
0.20 Corruption
0.30
0.35
(a) 50 Dimensional Embeddings
(b) 100 Dimensional Embeddings
Corruption
Corruption
100, Dimensions:
1.2
1.0
0.0
Enaor
0.6
0.6
0.4
0.2
0.2
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
Corruption
Corruption
(c) 200 Dimensional Embeddings
(d) 300 Dimensional Embeddings
Figure 66: Error on "Pleasant" Embeddings Corrupted with 'Unpleasant! Embeddings
As in the LLM exporiment, we observe that no algorithm significantly outporforms sample_mean despite the nontrivial LOOCV error in each setting. Moreover, we observe that median mediann_o__mans me_ms consistently achieves error slightly worse than sample_mes mean, which is not seen in the LLM experiments, suggesting the algo- rithm's 's sensitivity to distributional assumptions. However, unlike in the LLM experiment ev_ w__fittering_oww does not fail rophicclyy here, instead nearly matching sample_ _eeeee______..__ This is not unexpected given ev wflffteringllwnnn and the trace estimate techniques sensitivity to distributional assumptions will some- time work including this case. LRV also tends to perform worse than other algorithms, though this gap is not as large as in the LLM experiment.
Corrupted Words For the corrupted case, we draw data X (1-1)))++ where the inlicr distribution, P, consists of embeddings of 'pleasant' words, and the outlier distribution, Q, consists of embeddings of 'umpleasant' words. This models a more extreme version of the case where ill-defined words may be placed in category, inducing bias. This is notable problem for word vectors, which do not take context into account Hu et al. 2016). The experimental setup is identical to in the LLM experiments. Results are shown
in Figure 66
While these results are different from the LLM experiment, they demonstrate similar trends. In particular, QUE_low n is again the strongest performer, noticeably outperforming all other estimators across the 200 and 300 dimensional cases, and never performing worse than sample_ mean in the 50 and 100 dimensional
mean
69

---

Published in Transactions on Machine Learning Research (02/2025)
LOOCV Error vs Data Size Dimensions: 50, Expected Corruption: 0.10
LOOCV Error vs Data Size
Dimensions: 100, Expected Corruption: 0.10
3.3
LOOCV Error
sample,
LOOCV Emor
3.8
coord median
311
geometric median
valiant simple
median of means
filtringg_at low que Jow_n
3.5
pgd
100
60
100
(a) 50 Dimensional Embeddings
(b) 100 Dimensional Embeddings
LOOCV
LOOCV
Dimensions: Expected Corruption:
Dimensions: 300, Expected Corruption:
5.0
5.5
4,9
5.4
LOOCV Error
4,8
LOOCV Ener
5.3
5.2
5.1
5.0
4,9
20
90
100
20
40
60
80
100
Data Size
Data Size
(c) 200 Dimensional Embeddings
(d) 300 Dimensional Embeddings
Figure 67: LOOCV Error on 'Unpleasant' GloVe Embeddings
cases. Notably, this strong performance occurs even with n much less than d. Unlike the LLM experi- ments, here QUE_low_n never approaches and 100 dimensional cases. dee_valiant valiant_simple, which tended to perform similarly to DUE_low_n and nearly match good_ srod__ample_maet Lealle__maaa_e in the LLM experiments, does not perform as well in this case. It always beats sample_mean but docs not come close to matching srrod_ssmple___m mean and performs similarly to other es- timators. Likewise, @oddaa_of__eeee even performs worse than sample_meat over very low corruption. Supported by synthetic data results, means and lee L___vvlant___mppe
_asp_s_mmeaaaaaeeee _man, and is beat by other estimators in the 50
does not perform as strongly here as in the LLM experiments and
this suggests the sensitivity of median_od mediann_o_mmeass in the LLM experiments, LRV tends to perform much worse than sample_ mean under low corruption and outperform sample_meat mean slightly with higher corruption; PGD tends to outperform sample_mes slightly; and coord_ podd__n coord_ tet___trim_mmmmeee __aan, and coometric_mediane tend to perform similarly or slightly worse than sample_mean. As in the LOOCV experiments, ev filterinng_ooww..o. simply matches sample_mess mean here.
distributional assumptions. As
Additional Experiments We porform additional experiments, swapping the roles of "plcasant" and 'un- pleasant embeddings. We report LOOCV error vs data size on embeddings 'umpleasant words Figure We report corrupted error vs data size on embeddings of 'umpleasant' words corrupted with "pleasant" words in Figure 68 We observe the same trends in the previous word embedding experiments.
70

---

Published in Transactions on Machine Learning Research (02/2025)
Error vs Corruption
Error vs Corruption
Data Size: 100, Dimensions: 50
Data Size: 100, Dimensions: 100
sample_men meen
1.0
coord median
coord trimmedd_meee. mean
1.0
geometric_ median
0.8
Ice valiant simple
median, of means
0.8
ev w__fiteringg_ow
que_low, Jow_n Pgd
0.6
0.4
good sample.mmaeeeee mean
good_sample_ntte
0.2
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
Corruption
(a) 50 Dimensional Embeddings
(b) 100 Dimensional Embeddings
Error
Error Corruption Data Size: 100, Dimensions: 300
Data Size: 100, Dimensions: 200
1.2
1.2
Eror
Enor
0.6
0.2
0.0
0.0
Corruption
Corruption
(c) 200 Dimensional Embeddings
(d) 300 Dimensional Embeddings
Figure 68: Error "Unpleasant' Embeddings Corrupted with "Pleasant' Embeddings
71

---

Published in Transactions on Machine Learning Research (02/2025)
A.9 LLM Experiment Ablations
Eigenvaluc Pruning Method Comparison We compare the performance of different pruning sub- routines for ev _filtering_low_n over selection of LLM experiments: LOOCV and Corruption Error over MiniLM and BERT embeddings. We evaluate Gaussian pruning, used throughout this paper, along with randomized pruning and fixed pruming, described in Appendix A..5 We retain the same conditions as in the original experiments, first scaling data utilizing the sample trace. We also include sample_men memple_meen and QUE_low_n n our plots for the sake of comparison, noting that QUE_low_n and ev filteringglov fixed pruning only differ in their method of scoring outliers. These results are shown in Figure 69 We notice that both randomized and fixed pruming methods do indeed perform better than the Gaussian prun- ing method. In particular, fixed pruning has the best LOOCV error MiniLM and matches the error of sample_mean mean over BERT, whereas Gaussian pruning fails dramatically. However, this performance does not translate into the corrupted case, where all three pruning routines lead to significant error compared to even sample_mean, except with large y where sample_mean error approaches that these methods. Additionally, as discussed Appendix [A.5] ev w__fiteering_lw_n with fixed pruning is not robust to noise distributions that require several runs of the algorithm to prune i.c. cases where noise lays in multiple ort orthogonal clusters. Notably, QUE [UE__ow_r outperforms all variations of ev_ filtering_low low__low_nh n in the corrupted data case, reinforcing the observation that the outlier detection method of QUE_low_n is more robust to distributional assunptions than that of ev fiilteringgoww__..
with
LOGCV Eror vs Data Sire Dimensions: 384, Expected Compuptin: 0. 0.10
LOOCV Error vs Data Siza
Dimensione: 76R, Expacted Carruption: C.10
ey filaiing ke
10.0
ev fitering ko n. random
or fitering food
good sample meenn
2.4
8.5
8.0-
2.2
200
Data
(a) LOOCV Error MiniLM
(b) LOOCV Error BERT
Error vs Coruuttion 400, Dimansions:
Error vs Comruption 400, Dimensions:
3.
14
30-
0.0
0.3
Compption
Cornuation
(c) Corrupted Error MiniLM
(d) Corrupted Error BERT
Figure 69: Eigenvalue Pruning Pruning Method: LLM Comparison
72

---

Published in Transactions on Machine Learning Research (02/2025)
LRV Weighting Procedure Here compare the two different weighting procedures for LRV described in Appendix A.5 Gaussian weighting, based on downweighting outliers, and general (non-Gaussian) weighting, based on completely pruning outliers. We evaluate these two methods over the same subselection of LLM experiments: LOOCV and Corruption Error over MiniLM and BERT embeddings. These results are shown in Figure 70 We notice that general weighting oulperforms Gaussian weighting LOOCV error, with this difference being especially noticcable across BERT embeddings. However, this performance increase is not seen in either corrupted case, where Gaussian weighting notably outperforms general weighting, except with small 17. This suggests that, least under low data size, LRV is not robust to general distributions, even using a general outlier weighting procedure.
LOGCY
LOOCY Erao
Dimensione: 3Bd, Expected Comuption: O. 0.16
Dimensione: 768, Expected Carruption: 0.10
2.35
sample maan
9.0
Iv. general
sample
2.45
2.40
8.0
0.6
2.30-
8.2-
2.15
50
100
150
200
250
300
350
400
20
300
150
200
250
350
400
Data Size
Data Size
(a) LOOCV Error MiniLM
(b) LOOCV Error BERT
Error vs Coruption
Error Corruption Dota Size: 400, Dimensions: 768
Data Size: 100, Dimansions: 384
2.0
1.0
3
2-
0.0
(c) Corrupted Error MiniLM
(d) Corrupted Error BERT
Figure 70: LRV Gaussian Vs General Weighting: LLM Comparison
73

---

Published in Transactions on Machine Learning Research (02/2025)
Additional Experiments We recreate the experiments in Section 5 over two different settings. First, we examine LOOCV Error over embeddings of the word field that correspond to the "field of study" definition rather than to the 'field of land" definition. These results are shown in Figure 71 Second, we examine corrupted embeddings X ~ (1 - + where inlier data, P. consists of embeddings of the word 'field" corresponding to the field of study" definition and outlier data, Q, consists of embeddings of the word "field" corresponding to the 'field of land' definition; inverting the inlier and outlier data originally examined. These results are shown in Figure 72 While the LOOCV error plots are not identical to the original experiment, corresponding to the expected differences in structure between the distributions of P and Q, we find the same overall trends across the 4 plots. We additionally observe the same overall trends for corrupted data which was consistently the best algorithm compared to the original experiment. However, breaks down for MiniLM here; al always performing notably alongside QUE_low_ for corrupled data originally, Supported by the general poor performance of lee to__vvllatt_smmpp over synthetic data experiments, this reinforces the unpredictable scnsitivity of lee_ deee__elint__ae__eeeee valiant_simple to distributional assumptions. QUE_low does not see any such degradation, performing near optimally across all cases, it does in the original LLM experiment.
worse than good_sample_mean
LOOCV Error vs Data Size
LOOCV Error vs Data Size
Dimensions: 384, Expected Corruption: 0.10
Dimensions: 512, Expected Corruption: 0.10
sample mean
3.75
coond_ median
coord trimmed mean
3.50
geometric median
lee, vea_velll__imppll
3,25
median of means
LOOCV Ereor
3.00
ev_ lfffffernngoowee
2.1
que Jow_nw__ n
LOOCV
Pgd
2.0
3.9
2.25
1.8
2.00
1.7
50
100
150
200
250
300
350
400
50
100
150
200
250
300
350
400
Size
(a)
MiniLM
(b) T5
LOOCV Error vs Data Size
LOOCV Error Data Size
Dimensions: 768, Expected Corruption: 0,10
Dimensions: 768, Expected Corruption: 0.10
10.50
10.25
13.5
10.00
LOOCV Enrr
LOOCV Error
13.0
3.50
9.25
12.0
9.00
11.5
8.75
150
200
250
300
Data Size
Data Size
(c) BERT
(d) ALBERT
Figure 71: LOOCV Error 'Field Of Study" Embeddings
74

---

Published in Transactions on Machine Learning Research (02/2025)
Error vs Corruption
Error vs Corruption
Data Size: 400, Dimensions: 384
Data Size: 400, Dimensions: 512
2.00
coord median
sample_men meen
2.00
coord trimmedd_meee....... mean
175
geometric_ median
1.75
Ice, valiant simple
median, of means
1.50
w__fiteringg_ow que_low_n Pgd
1.00
1.00
pond_sample_mee
0.75
0.75
0.25
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2 Corruption
0.3
0.4
(a) MiniLM
(b) T5
Error
Error Corruption
Data Size: 400, Dimensions: 768
Data Size: 400, Dimensions: 768
12
17.5
15.0
12.5
Enor
10.0
7.5
5.0
2.5
Corruption
Corruption
(c) BERT
(d) ALBERT
Figure 72: Error 'Field of Study" Embeddings Corrupted with 'Field of Land" Embeddings
75

---

Published in Transactions on Machine Learning Research (02/2025)
A.10 Dataset Generation
We generate a dataset of 400 sentences for each delinition of the word field using Chat accessed in June 2024. Attention based embeddings for the word field are extracted from these sentences for use in our LLM experiments. We used the following two prompts to obtain the sentences:
Field of Study
om running an experiment where ezamine embeddings of the word "field" with two different conterts. Please generate 400 unique sentences using the word "field" in contert with the following definition: 'a particular branch of study or sphere of activity or interest." Please return these sentences in the format of a JSON file.
Field of Land
I am running an ezperiment where ezamine embeddings of the word "field" with two different conterts. Please generate 400 unique sentences using the word "field" in contert with the sollowing definition: 'an area of open land, especially one planted with crops pasture, typically bounded by hedges or fences." Please return these sentences in the format of a JSON file.
Additional Prompts ChatGPT-4o did not produce the full 400 sentences in one go. To address this, we used the following additional prompts until we had generated the required number of sentences, and then manually combined the generated outputs. The prompt for 'field of study' sentences is slightly different, we originally observed that ChatGPT-40 would reuse the same field of study across numerous sentences. For Field of Study:
Please generate 100 more sentences. Do not repeot similar sentences "field" to refer to the same field of study multippe times.
For Field of Land:
Please generate 100 more sentences.
Tables Of Generated Sentences
We include the following tables of generated sentences.
Field Of Land Sentences:
Index
Sentence
The scarecrow stood tall in the middle of the field.
The deer were spotted grazing in the field at dawn.
The field was an ideal spot for stargazing.
He loved to watch the sunset over the field.
The field stretched out as far as the eye could see.
Sunflowers swayed in the field under the clear blue sky.
The field stretched out to the edge the forest.
The field was alive with the sound of chirping crickets.
They played hide and seek in the field, darting among the tall grasses.
He built small shed the edge of the field for storage.
The hot air balloon landed gently in the field.
She enjoyed picnicking in the field of wildflowers near her home.
Continued on next page
76

---

Published in Transactions on Machine Learning Research (02/2025)
Continued from previous page
Index
Sentence
13
She found an old, weathered barn at the edge of the field.
14
The field was fenced off to keep out wild animals.
They walked hand hand through the field of wildslowers, lost in conversation.
16
He loved the quiet solitude of the open field.
17
The field was perfect spot for birdwatching.
We had a picnic in the wide, open field.
19
The open field was covered in morning dew.
We watched the meteor shower from the field.
21
The open field was perfect for stargazing.
22
The field of strawberries was a patchwork of red and green.
The field was filled with the scent of blooming flowers.
24
Hikers followed the trail through the field of wild grasses, enjoying the solitude.
They played ultimate frisbee in the field.
The field of herbs was fragrant, each plant releasing its unique scent.
27
She found a hidden path that led to the field.
The field was sea of green during the spring.
She found a hidden path that led to the field.
30
Wildflowers grew abundantly in the field.
31 A lone tree stood in the middle of the field, providing shade.
32
The field was riot of color in the fall.
33
They harvested wheat from the vast field.
34
Deer grazed in the field dusk, their silhouettes blending with the shadows.
35
The field was vibrant green after the rain.
36
The field was riot of color during the summer.
37
The field was sea of gold during the harvest.
38
She enjoyed painting the landscape of the field.
39
He loved the feeling of the grass under his feet in the field.
40
We spotted deer grazing in the distant field.
41
The field was surrounded by rolling hills.
42
We walked through the field at sunrise.
43
The field was blankeled in snow during the winter.
44
The field was playground for the neighborhood children.
45
The field was sea of purple lavender in full bloom.
46
The field was blanketed in snow during the winter.
47
Cows grazed peacefully in the field enclosed by wooden fences.
48
The field, bordered by ancient oak trees, was serene spot for picnic.
49
She enjoyed walking through the field, picking flowers.
50
The field was blanketed with snow in winter.
51
He loved the peace and quiet of the open field.
52
The field was patchwork of different crops.
53
The field was sea of gold during the wheat harvest.
54 A scarecrow stood watch over the field.
55
He spent his afternoons walking through the field, lost in thought.
56
The field was divided into neat rows for planting.
57
The field was perfect spot for a family picnic.
58
Birds chirped happily in the field, scarching for insects among the plants.
59
The field was dotted with patches of wild grass.
60
She painted landscape of the field in her art class.
61
The field was dotted with hay bales after long day of harvesting.
Continued on next page
77

---

Published in Transactions on Machine Learning Research (02/2025)
Continued from previous page
Index
Sentence
62
He built a small fire pit in the middle the field.
63
He loved the smell of fresh-cut grass in the field.
64
She found a quiet spot in the field to read her book.
The field was a burst of color in the autumn.
The field of rye swayed in the breeze, creating waves of green.
The field was alive with the sound of crickets.
68
The field was a popular spot for local festivals.
69
He loved the smell of fresh-cut grass the field.
70
The field was favorite spot. for local photographers.
He loved to run through the field with his friends.
72
The field was alive with the sound of crickets.
We could see the farmhouse from across the field.
74
He loved the quiet solitude of the open field.
A scarecrow stood at the center of the field, arms outstretched.
76 We spotted a fox darting through the field.
The children flew paper airplanes the field.
They set up a makcshift basoball diamond in the field.
79 A herd of sheep grazed peacefully in the field.
80
They played a game of tag in the spacious field.
81
He loved to explore the field with his dog.
82 A gentle breeze rustled the leaves of the crops in the field. The field was covered in a blanket of fresh snow.
83
84
The horses galloped freely across the open field.
85
The dogs ran freely in the wide field.
86
The field of blueberries was a favorite spot for summer picking.
87
The field was a peaceful place to reflect and relax.
88
The scarecrow stood guard in the field, its tattered clothes fluttering in the breeze.
89 The field was place of peace and serenity. A gentle fog settled over the field in the morning. The field was home to several species of birds.
90
91
92
The field was ideal for an outdoor concert.
93
She enjoyed walking through the field, listening the birds sing.
94
The field was alive with the sound of crickets chirping at night.
95
The field was alive with the sound of crickets.
96
The field was perfect spot for a family picnic.
97
The field of corn rustled in the wind, creating soothing sound.
98
The field was quiet refuge from the busy city.
99 He built small bench at the edge of the field.
100 They played frisbee in the open field.
101 He loved the smell of fresh-cut grass in the field.
102 She enjoyed picnicking in the field with her friends.
103 The farmer worked tirelessly in the field to ensure good harvest.
104 A small stream ran along the edge of the field, providing water for the livestock.
105 She enjoyed walking through the field, picking flowers.
106
The community garden was set up in the field.
107 A tractor moved slowly across the field, plowing the earth for new seeds.
108
They had an Easter egg hunt in the field.
109 We could see the larmhouse from across the field.
Continued on next page
78

---

Published in Transactions on Machine Learning Research (02/2025)
Continued from previous page
Index
Sentence
110
A gentle breeze blew across the field.
111
The field stretched out to the horizon, seemingly endless.
112
The field buzzed with the sound of bees collecting nectar.
113
The field was a riot of color in the fall.
114
The field was home to a variely of wildlife.
115 Birds nested in the hedges surrounding the field, singing melodious tunes.
116
The field was dotted with hay bales, ready for storage.
117
The field was sea of gold during the wheat harvest.
118
The field was lush and green after the rain.
119
She loved to dance barefoot in the field.
120
The kids enjoyed a treasure hunt in the field.
121 The field of cherry blossoms was a sight to behold, petals drifting in the wind.
122
The farmer walked across the field, inspecting the growing wheat.
123 After the rain, the field was dotted with puddles reflecting the clouds.
124
Butterflics flitted about in the field, adding to its charm. She could see the field from her kitchen window.
125
126 We found quict spot in the field to relax.
127 They set up a campfire in the field.
128
The farmer plowed the field in preparation for planting.
129 She enjoyed walking through the field, picking flowers.
130
The field's soil was rich and fertile, ideal for planting.
131
The field was vibrant green after the rain.
132 The field was a peaceful place to escape to.
133
The field of sunflowers attracted bees with its abundance of pollen.
134
The field was blanketed in snow during the winter.
135 A small brook ran alongside the field, providing irrigation.
136 He watched the sunrise over the field from his porch.
137 She found a quiet corner of the field to meditate.
138 She found a hidden path that led to the field.
139 She picked wild strawberries in the field, their sweetness bursting in her mouth.
140
The large field was perfect, for flying kites.
141
The field of cotton was ready for picking, the fluffy bolls bursting open.
142
The field of rapeseed glowed bright yellow against the blue sky.
143
He loved to run through the field with his friends.
144
The field was canvas of colors in the spring.
145
The field was surrounded by a wooden fence.
146 Farmers plowed the field, preparing it for the next planting season.
147 He loved to watch the sunset over the field.
148
She could hear the distant sound of a tractor working in the field.
149 Children flew kites in the field, the colorful tails dancing in the wind.
150
The field was place of beauty and tranquility.
151 They had barbecue in the field last weekend.
152 The field of lettuce was irrigated regularly, ensuring crisp leaves.
153 She found a hidden trail that led to the field.
154 She enjoyed walking through the field, listening the birds sing.
155 A narrow path cut through the field, leading to the old barn.
156 She found a perfect spot in the field for her garden.
157
The field was sea of green grass in the spring.
158
Farm workers toiled from dawn to dusk, tending to the vast field.
Continued on next page
79

---

Published in Transactions on Machine Learning Research (02/2025)
Continued from previous page
Index
Sentence
159
The field of oats rustled softly as the wind passed through.
160
She loved to dance barefoot in the field.
161 A gentle mist rose from the field in the early morning.
162
The field of soybeans stretched for miles, a sea of green leaves.
163 The field was a place of peace and serenity.
164
The farmer surveyed the field, planning his next move.
165
The field was a sea of gold during the wheat harvest.
166
The field was patchwork of different crops, each thriving in the rich soil.
167 She spent her afternoons wandering through the field.
168
The wheat field swayed gently in the wind.
169
Sunflowers bloomed vibrantly in the field, their faces turning towards the sun.
170 The field was a perfect place for a family gathoring.
171
The field was alive with the sound of crickets.
172
The field was lush with green crops swaying in the breeze.
173
In the distance, a barn overlooked the sprawling field.
174
The field was sea of green during the growing season.
175
The field glistened with morning dew.
176 They picked strawberries in the field.
177 The field was a playground for the neighborhood children.
178 A herd of cows roamed froely in the field.
179 A flock of birds flew over the field.
180
The field was blanketed in wildflowers during the spring.
181 The field was full of life, even in the winter.
182 The field was a favorite spot for watching the stars at night.
183
Children played a game of soccer in the field behind the school.
184
The field was a perfect spot for birdwatching.
185
The field was peaceful place to escape to.
186
The kids enjoyed running freely in the open field.
187 He set up telescope in the field to watch the stars.
188
The children lay down in the field to watch the clouds.
189 The field was dotled with blooming llowers.
190 He found a perfect spot in the field for his garden.
191
The cows roamed freely in the open field.
192 The field of corn was ready for harvesting.
193
The field of daisies swayed gently in the wind, a sea of white petals.
194
The field provided a perfect. backdrop for photos.
195
The field was sea of gold during the harvest.
196 They held a yoga class in the field at dawn.
197 The field lay fallow, resting before the next planting season.
198 The sun set behind the distant field.
199 He could see the field from his bedroom window.
200 The field was vibrant green after the rain.
201 She ran through the field of tall grass, her laughter ringing out.
202 They set up a tent in the field for the event.
203
The field was favorite spot. for local photographers.
204 She enjoyed collecting wildftowers from the field.
205
They flew kites in the large, empty field.
206
The field was vibrant green after the rain.
207
The festival was held in the open field every summer.
Continued on next page
80

---

Published in Transactions on Machine Learning Research (02/2025)
Continued from previous page
Index
Sentence
208
The field was a haven for wilditowers.
209
The field of radishes was ready for picking, their bright red roots peeking out.
210
The field was lush with green crops swaying in the breeze.
211 They saw a fox darting across the field.
212
She found a quiet corner of the field to meditate.
213
Rows of corn stretched across the field, reaching up towards the sky.
214
The field was a playground for the neighborhood children.
215 She took a stroll through the field, enjoying the fresh air.
216 He loved the smell of fresh-cut hay the field.
217
The field was a patchwork of colors during the flower festival.
218 She spent her afternoons wandering through the field.
219
The field was covered in a blanket of wildflowers.
220 The pienic was set up in the field by the lake.
221 The field was sea of green during the growing season.
222 A bcautiful field of sunflowers stretched as far as the eye could see.
223
The scarecrow stood watch over the field, deterring hungry birds.
224 The horses galloped across the field, kicking up dust behind them.
225 The horses grazed peacefully in the field.
226 He loved the peace and quiet of the open field.
227 The field of tulips was a riot of color, rods, ycllows, and pinks blending togother.
228 The field was buzzing with activity during the harvest season.
229 He loved the quiet solitude of the open field.
230 She painted bcautiful landscape of the field.
231 He set up telescope in the field to watch the stars.
232 They wandered through the field of pumpkins, searching for the perfect one.
233
In the winter, the field was covered in a thick layer of snow.
234 Cows grazed peacefully in the field enclosed by wooden fences.
235
The farmers planted potatoes in the field closest to the farmhouse,
236 They set up a tent in the field, ready for a weekend of camping.
237
The field was a haven for wildflowers.
238 At sunset, the field glowed with golden hue, creating a picturesque scene.
239
The sheep roamed freely in the field, nibbling on fresh grass.
240
She enjoyed painting the landscape of the field.
241 He loved to explore the field with his dog.
242
The field was patchwork of different crops.
243 They saw a rainbow stretching over the field.
244 He found solace in the quiet field, away from the hustle and bustle.
245
The field was place of peace and serenity.
246 The field was expanse of green grass.
247 The field was haven for wildlife, including rabbits and deer.
248 He built small bench at the edge of the field.
249 A gentle breeze rustled the leaves in the field.
250 She collected wildflowers from the field to make a bouquct.
251 The field, surrouneed by rolling hills, was picture of serenity.
252 She found an old, weathered barn at the edge of the field.
253 The festival was held in the large, open field.
254
She spent her afternoons wandering through the field.
255
The field was playground for the neighborhood kids.
256
The field was vibrant green after the rain.
Continued on next page
81

---

Published in Transactions on Machine Learning Research (02/2025)
Continued from previous page
Index
Sentence
257
The field of cabbages was neatly arranged in rows, their heads forming a patch- work.
258
She found a quiet spot in the field to read her book.
259
Farmers tended to the field of pumpkins, ensuring each one grew plump and round.
260 The field was quiet refuge from the busy city.
261 They set up camp in the field, under canopy of stars.
262
The field was favorite spot for local artists.
263
The field was plowed into neat, straight rows.
264
The kids played catch in the large field.
265
The field was favorite spot. for local photographers.
266
The farmer rotated his crops to keep the field fertile.
267
We had family rcunion in the field.
268
He loved the smell of fresh-cut hay in the field.
269
The field of onions was harvested in the fall, bulbs dug up and stored for winter.
270
The field was blanketed in fog early in the morning.
271
The field provided ample space for the anmual county fair.
272
She enjoyed collecting wildflowers from the field.
273
She loved to dance barefoot in the field.
274
The field was riot of color in the summer.
275
The field was perfect spot for birdwatching.
276
She ran across the field, her laughter echoing in the open space. Butterflies fluttered over the wildftowers that dotted the field.
277
278
The field stretched out to the horizon.
279 They practiced their golf swings in the open field.
280
The field was perfect for game of cricket.
281
In the summer, the field was sea of golden barley ready for harvest.
282
The field was home to a family of rabbits.
283
The field was alive with the hum of insects.
284 The field was vibrant green after the rain.
285 The field was dotted with patches of wild grass.
286 We enjoyed a pienic lunch in the sunny field.
287 We lay on the blanket in the middle of the field.
288 She found a quiet spot in the field to read her book.
289 Children played soccer in the open field near the village.
290 The field was covered in a blanket of snow during the winter months.
291 The field was vibrant green after the spring rain.
292 He loved to explore the field with his dog.
293 Wildllowers dotted the field, adding splashes of color to the landscape.
294 They walked through the field of lavender, inhaling its sweet fragrance.
295 We had bonfire in the middle of the field.
296 He found quiet spot in the field to read his book.
297 The field was sea of green during the growing season.
298 The field of barley was ready for harvest, the grains turning golden.
299 The field was surronned by a white picket fence.
300 He built small shed at the edge of the field.
301
The field of sugar cane stretched to the horizon, its stalks swaying gently.
302
The field was burst of color in the autumn.
303
She enjoyed collecting wildflowers from the field.
Continued on next page
82

---

Published in Transactions on Machine Learning Research (02/2025)
Continued from previous page
Index
Sentence
304
He loved to watch the sunset over the field.
305
A beautiful field of sunflowers stretched as far as the eye could see.
306 She found an old, rusty plow at the edge of the field.
307 We watched the sunset from the edge of the field.
308 The dogs loved running around the open field.
309
The field of lavender was a haven for bees, buzzing busily among the flowers.
310
The field was riot of color in the fall.
311 Farmers worked diligently in the corn field.
312
The field was dotted with patches of wild grass.
313 Sheep roamed freely in the field, their woolly coats glistening in the sun.
314
The children played soccer in the field.
315 The field was favorite spot for local artists.
316 He built a small bench at the edge of the field.
317 The field was tapestry of colors in the spring, with various flowers in bloom.
318 We could see rabbits hopping in the field.
319 The field was sea of gold during the harvest.
320 She sat under the oak tree in the field, enjoying the shade.
321 Cattle grazed peacefully in the field, surrounded by rolling hills.
322
The field looked magical under the light of the full moon.
323 The field was dotted with balcs of hay.
324
The field was riot of color during the summer.
325 He set up picnic in the middle of the field.
326 The old oak tree stood alone in the field.
327 A scarecrow stood tall in the middle of the field, warding off birds.
328 The field was favorite spot for local photographers.
329 She spent hours wandering through the field, collecting herbs.
330 She enjoyed picnicking in the field with her family.
331 The field was home to a variety of wildlife.
332 She lay down in the field, gazing up at the clear blue sky,
333 She found a hidden trail that led to the field.
334
The field was full of lile, even in the winter.
335 The field was surrounded by a dense forest.
336
In the fall, the field turned golden hue as the crops matured.
337 A light. breeze swept through the open field.
338 He found a hidden trail that led to the field.
339 The path led us through vast field.
340 He built small shed at the edge of the field.
341 He found perfect spot in the field for his garden.
342 The field was haven for wildfloers.
343 The field was riot of color during the summer.
344
The scent of freshly cut grass filled the air as the field was mowed.
345 They walked through the field of wheat, the stalks brushing against. their legs.
346 Children around playing in the field all day.
347
The field was quiet refuge from the busy city.
348
The field was peaceful place to escape to.
349 He could see the field from his bedroom window.
350
The field was favorite spot for local artists.
351
The field was full of life, even in the winter.
352
A fence made of wooden posts and wire encireled the field.
Continued on next page
83

---

Published in Transactions on Machine Learning Research (02/2025)
Continued from previous page
Index
Sentence
353
The field was a burst of color in the autumn.
354
He loved to run through the field with his friends.
355
A tractor moved slowly across the field, tilling the soil.
356 She lay in the field, watching the clouds drift by.
357 The field was sea of green during the spring.
358 He loved the peace and quiet of the open field.
359
The field was patchwork of different crops.
360
The field was favorite spot. for flying drones.
361 Farmers harvested hay from the field, stacking neatly in bales.
362
She found a quiet corner of the field to meditate.
363
The field was sea of green during the spring.
364 Children played soccer in the field behind the school, their langhter echoing.
365
The tractor moved slowly across the plowed field.
366 The field was dotted with patches of clover.
367 He built small shed the edge of the field.
368 The field was place of beauty and tranquility.
369 He set up telescope in the field to watch the stars.
370
The field was used for growing sunflowers.
371 She enjoyed walking through the field, listening the birds sing.
372 The field of wheat stretched across the horizon, golden under the afternoon sun.
373
They planted variety of vegetables in the field.
374
She loved the scent of fresh earth in the field after it rained.
375
The field was peaceful place to reflect and relax.
376 They picked wildflowers from the edge of the field.
377 A lone (ree slood in the middle of the field.
378 He could the field from his bedroom window.
379 The field of hemp grew tall and strong, its fibers used for various products.
380 He loved the smell of fresh-cut hay in the field.
381 The field was buzzing with bees collecting nectar.
382 A rainbow arched over the field after the rain.
383 They planted rows of vegetables in the fertile field.
384 The field was peaceful place to reflect and relax.
385
The field of vineyards produced grapes for fine wines, rows of vines neatly trellised.
386 She found an old, weathered barn at the edge of the field.
387 They pienicked in the field of clover, enjoying sandwiches and lemonade.
388 She enjoyed painting the landscape of the field.
389 We took walk through the field at sunset.
390 They built a bonfire in the field, its flames lighting up the night sky.
391 The farmer's dog ran joyfully through the field.
392 The field was the perfect spot. for a family gathering.
393
She enjoyed picnicking in the field with her family.
394 The field was covered in a thick layer of frost.
395 The field of potatoes was ready for digging, the earth yielding its treasures.
396 He could see deer grazing in the field from his window.
397
The field was place of beauty and tranquility.
398
The field was perfect spot for a family picnic.
399
Rain nourished the field, ensuring bountiful crop for the season.
Continued on next page
84

---

Published in Transactions on Machine Learning Research (02/2025)
Continued from previous page
Index
Sentence
400
The field was haven for birdwatchers.
Table 2: Field Of Land Sentences
Field Of Study Sentences:
Index
Sentence
The field of bioethics addresses the ethical issues in biology and medicine.
He is an innovator in the field of software development.
The conference attracted top professionals specialized in an interesting field of study.
The field of cultural studies examines how culture shapes identity.
The field of immunology studies the immune system.
The field of anthropology studies human cultures and societies.
His expertise in the field of structural engineering is invaluable.
My favorite baseball movie is field of dreams.
The field of socio-cultural anthropology examines human societies and their customs.
10
The field of cultural anthropology studies human societies.
The field of operations research uses mathematical methods to make decisions.
12
The field evolutionary psychology explores the evolutionary origins of human behavior.
He is an authority in the field health policy and management.
He is a leading expert in the field Torensic anthropology.
Her studies in the field dance theory are intriguing.
He has made strides in the field of molecular genetics.
His work in the field of plasma physics highly regarded.
The field of computational sociology uses computational methods to study social phenomena.
19
His studies in the field of marine biology are fascinating.
20
The field of telecommunications is rapidly advancing.
21
The field of supply chain management is vital for global commerce.
He decided to pursue career in the field of computer science.
23
The field of biotechnology holds great promise for the future.
24
She has background in the field of political science.
25
Hl is a leading researcher in the field of evolutionary genetics.
26
The field of digital forensics investigates cybercrines.
27
The field of transpersonal psychology explores spiritual and transcendent. pects of the human experience.
28 He is a pioneer in the field of machine learning.
29
Her research in the field of music cognition explores how the brain processes music.
30
The field of sociology looks al how societies function.
31
Her research in the field of computational chemistry is groundbreaking. Her expertise in the field of infectious discases informs public health policies. Her expertise in the field of environmental sociology addresses human interac- tions with the environment.
32
33
34
The field of optics studies light and its interactions.
Continued on next page
85

---

Published in Transactions on Machine Learning Research (02/2025)
Continued from previous page
Index
Sentence
35
The field of biogeography studies the distribution of species across geographical arcas.
36
He received an award for his contributions to the field of engineering.
Her passion for the field of public health is evident.
38
The field of marketing explores consumer behavior and advertising strategies.
39
She is passionate about her work in the field of social work.
The field of actuarial science asscsses financial risks.
41
The field of hydrology studies the distribution and movement of water on Earth.
42
She has received accolades for her work in the field of artificial intelligence.
43
The field of health informatics improves patient care through data,
44
The field of neuroinformatics combines neuroscience and data analysis.
45
She is highly respected in the field of architectural history.
46
He is a pioneer in the field of digital humanities.
47
The field of sports medicine focuses on athletes' health and performance.
The field of chemical engineering involves the creation of new materials. 48
49
He is a leading voice in the field of peace and conflict studies.
50
He is a prominent figure in the field of environmental sociology.
51
The field of sports medicine helps athletes recover from injuries.
52
The field of industrial design merges function with aesthetics.
53
The field of geriatric medicine focuses on elderly care.
54
The field of behavioral economics blends psychology and economics.
55
Her research in the field of climatology addresses global warming.
56
His research in the field of evolutionary biology is groundbreaking.
57
Her research in the field of computer graphics enhances visual simulation tech- niques.
58
The field of health informatics uses technology to improve healthcare delivery.
59 He has deep interest in the field of computational neuroscience. He is a renowned figure in the field of machine learning. The field of medical anthropology explores the intersection of culture and health.
60
61
62 The field of acoustics studics sound and its properties.
63
Her research in the field of gerontology focuses on aging.
64
The field of molccular gastronomy explores the science behind cooking.
65
She is advancing knowledge in the field of developmental psychology.
66
The field of physical therapy helps people recover mobility.
67
The field of social epidemiology examines health disparitics.
68
The field of artificial intelligence presents many ethical questions.
69
She is a trailblazer in the field of behavioral neuroscience. Her expertise in the field of neuroimaging enhances brain research.
70
71
She is an innovator in the field of fashion design.
72
His work in the field of artificial intelligence has been widely recognized.
73
He has made siguificant strides in the field of computational neuroscience.
74
She is making waves in the field of renewable energy.
75
The field of computational neuroscience models neural systems and behavior.
76
His studies in the field of immunology are groundbreaking.
77
He is a leading scholar in the field of information science.
78
He is a leading figure in the field of aerospace engineering. The field of digital sociology explores the impact of digital technologies on society.
79
Continued on nexl page
86

---

Published in Transactions on Machine Learning Research (02/2025)
Continued from previous page
Index
Sentence
80
Her expertise in the field gerontology addresses aging.
The field of linguistics helps us understand language structure.
The field of library science organizes and manages information resources.
Her expertise in the field of data science is highly sought after.
The field of literary criticism involves analyzing and interpreting texts.
He is a pioneer the field of disaster risk reduction.
86
She is an influential figure in the field of gender studies.
The field of cognitive anthropology studies cultural variations in cognition.
The field of agronomy deals with crop production and soil management.
He is a leading ligure in the field quantum information science.
90
He has published extensively in the field of theoretical physics.
The field of health informatics combines healthcare and IT.
92
Her research in the field of psychopharmacology examines the effects of drugs
on behavior.
93
He has authored several books in the field of history.
94
The field of international law governs legal relations between states.
95
The field of ethology examines animal behavior in natural environments.
96
The field of social psychology studies how individuals influence each other.
97
The field of political economy studies the relationship between politics and economics.
98
The field of anthropology examines human societies and cultures. Her expertise in the field of marine archaeology uncovers submerged history. Her work in the field of human-computer interaction designs user-friendly in- terfaces.
99
100
101 He is well-known in the field of quantum physics.
102 He has dedicated his carcer to the field of atmospheric sciences.
103
She has distinguished career in the field of comparative literature.
104 The field of cultural studios examines cultural phenomena.
105 Her work in the field of artificial intelligence is innovative.
106 He is a notable expert in the field of space exploration.
107 Her research in the field of cognitive anthropology explores cultural cognition.
108
Her work in the field of cognitive psychology studies mental processes.
109
He is a well-known expert in the field of civil engincering.
110 The field of behavioral economics explores how psychology impacts economic decisions.
111 The field of consumer psychology studics consumer behavior and decision- making.
112
The field of artificial intelligence is growing rapidly.
113
The field of aerospace science investigates flight and space.
114
She has published numerous papers in the field of environmental science.
115 She has a keen interest in the field of film studies.
116 His innovations in the field of electrical engineering are impressive.
117 Her research in the field of mobile computing enhances smartphone technology.
118
The field of computational economics applies computational methods to eco- nomic analysis.
119 The field of marine chemistry studies the chemical composition of oceans.
120 His work in the field of marine biology groundbreaking.
121
Her innovations the field of textile science are notable.
122
The field of consumer psychology explores why people buy things.
Continued on next page
87

---

Published in Transactions on Machine Learning Research (02/2025)
Continued from previous page
Index
Sentence
123
The field of evolutionary psychology explores human behavior.
124
The field of computational biology uses data to understand biology.
125
He has extensive experience the field of quantum computing.
126
The field of neuroscience delves into the workings of the brain.
127
The field of veterinary science focuses on animal health.
128
She has a strong foundation in the field of theoretical physics.
129
The field of music therapy uses music to improve mental health.
130 She has a profound impact on the field of forensic science.
131 His work in the field of linguistics has redefined language theories.
132 Her interest in the field of space exploration began in childhood.
133
The field of industrial design creates functional and aesthetic products.
134 His studics in the field of artificial intelligence are influcntial.
135 The field of educational psychology enhances leaching methods.
136
He is a notable figure the field of mechanical engineering.
137 The field of goophysics examines the physical propertics of the Earth.
138
Her work the field of urban planning promotes sustainable urban develop- ment.
139
The field of medical imaging develops techniques to visualize internal organs.
140
She has been working in the field of bioengineering for over decade.
141 He has background in the field of developmental psychology.
142
The field of musicology delves into the study of music.
143 He is a renowned scholar in the field of classical studies.
144 The field of computational archaeology uses computer models to study archae-
ological data.
145 He is an authority in the field of cybersecurity. Her contributions to the field of immunology have been invaluable.
146
147
The field of mathematical biology applies mathematical models to biological
processes.
148
Her research in the field of neuroscience is highly respected.
149
The field of sports psychology helps athletes improve performance.
150
The field of environmental engineering soeks sustainable solutions.
151
The field of social neuroscience investigates the neural basis of social behavior.
152
The field of social geography studies the spatial distribution of social phenom- ena.
153
The field of landscape architecture focuses on designing outdoor spaces.
154
The field of psycholinguisiccs invostigates the relationship between language and the mind.
155
He is a trailblazer the field of genetic counseling. The field of materials science investigates the properties of materials. 156
157
The field of computational linguistics develops algorithms for natural language processing.
158 He has background in the field of political sociology.
159
The field of semiotics analyzes signs and symbols in communication.
160
The field of planctary science explores the formation and evolution of plancts.
161 Her research in the field of robotics engineering advances automation technol- ogy.
162 He is a specialist in the field of aerospace engineering.
163
Her research in the field of cultural psychology investigates cultural influences cognition.
Continued on next page
88

---

Published in Transactions on Machine Learning Research (02/2025)
Continued from previous page
Index
Sentence
164
She has published numerous papers in the field of quantum mechanics.
165
He is an expert in the field of agribusiness management.
166
The field of robotics seeing remarkable advancements.
167
The field of entomology studies insects and their behaviors.
168
The field of literary criticism analyzes literary works.
169 Her expertise in the field of transportation engineering is crucial for infrastruc-
ture projects.
170 The field of pharmacology investigates how drugs affect the body.
171
She received an award for her work in the field of environmental science.
172 She is working on a project in the field of urban planning.
173
Her research in the field of endocrinology has been transformative.
174
She is conducting groundbreaking work in the field of bioinformatics.
175
The field of veterinary medicine cares for animal health.
176
Her work in the field of evolutionary psychology examines psychological traits.
177 She has made a name for herself in the field of textile engineering.
178
The field of forensic science applies scientifie methods to criminal investigatiios.
179 The field of quantum optics studies the behavior of light and matter the quantum level.
180 He is renowned in the field of human-computter interaction. The field of biotechnology holds great promise for the future. The field of paleoclimatology reconstructs past climate conditions.
181
182
183
His contributions to the field of economics have been groundbreaking.
184 He has deep interest in the field of roboties.
185 The field of musicology analyzes music history and theory.
186
Her expertise in the field of forensic anthropology aids in criminal investiga- tions.
187
The field of computational genetics analyzes genetic data using computational methods.
188
Her expertise in the field of legal studies is unmatched.
189 He is a specialist in the field of cardiology. She has made significant strides in the field of biotechnology.
190
191 Her contributions to the field of environmental law are significant.
192 He is a key player in the field of financial technology.
193 Her research in the field of computational neuroscience is influential.
194 She is interested in the field of cognitive science.
195 He has a background in the field of telccommunication..
196 He is a respected authority in the field of clinical research.
197 Her discoveries in the field of biochemistry have been revolutionary.
198 Her research in the field of artificial life simulates biological processos in com- puter models.
199 She has a strong background in the field of political science.
200
The field of bibliometrics analyzes academic publication patterns.
201 Her contributions to the field of educational technology are noteworthy.
202
The field of meteorology studies weather patterns and forecasting.
203 Her work in the field of financial engineering optimizes investment strategies.
204 The field of bioethics navigates moral issues in medicine.
205
The field of environmental economics studies the economic impact of environ-
mental policies.
206 She has a strong foundation in the field of human resources. Continued on next page
89

---

Published in Transactions on Machine Learning Research (02/2025)
Continued from previous page
Index
Sentence
207
He has a background in the field of peace studies and confiict resolution.
208
The field of organizational behavior studies how individuals and groups behave within organizations.
209 He has made significant contributions to the field of renewable energy.
210
The field of forestry studics the management of forests and natural resourees.
211 The field of agricultural science seeks to improve food production.
212
The field of photonics involves the study of light gencration and manipulation.
213
He is a recognized expert in the field of computational physics.
214
He is an innovator in the field of genetic research.
215 The field of computational biology uses computational methods to analyze biological data.
216
Her rescarch in the field of human-computer interaction improves user experi- ence.
217
The field of molecular biology examines the building blocks of life.
218
His research in the field of telecommunications has advanced the industry.
219
He is researching climale change within the field of environmental studies.
220
The field of historical linguistics studies language change over time.
221
The field of aerospace medicine focuses the health of pilots and astronauts.
222
The field of educational sociology examines educational institutions and pro- cesses.
223 She has dedicated her career to the field of education.
224
Her expertise in the field of public health is widely recognized.
225
She is advancing the field of clinical psychology.
226
The field of visual arts encompasses various creative disciplines.
227
She is a renowned expert in the field of pediatric medicine.
228
He has a strong background in the field of systems engineering.
229
The field of astrophysics seeks to understand the universe.
230
Her work in the field of bioacoustics explores animal communication through sound.
231 He is a renowned expert in the field of computational linguistics.
232 He has a profound impact on the field of digital markoting.
233
The field of marine science explores oceanic systems.
234
The field of materials science involves studying the properties of materials.
235
The field of bioinformatics combines biology and computer seience.
236 The field of environmental chemistry studies chemical processes in the environ- ment.
237
The field of cognitive development explores how thinking processes evolve over time.
238
The field of toxicology studies the effects of chemicals on living animals.
239
Her research in the lield of climate science is groundbreaking:
240 Her contributions to the field of computational fluid dynamics are substantial.
241
The field of fluid dynamics studies the behavior of liquids and gases.
242
The field of petrochemical engineering deals with petroleum products.
243
The field of ergonomics designs equipment for efficiency.
244
She is a key figure in the field of digital humanities.
245 He has background in the field of educational leadership.
246
He is a pioneer in the field of nanotechnology.
247
The field of geneties explores the inheritance of traits.
248
He has deep interest in the field of computational linguistics.
Continued on next page
90

---

Published in Transactions on Machine Learning Research (02/2025)
Continued from previous page
Index
Sentence
249
The field of political sociology studies political institutions and behavior.
250
The field of paleontology uncovers the history of life on Earth.
251
The field of astrophysics reveals the wonders of the cosmos.
252
The field of cognitive science examines how we think and learn.
253
He has a prolific career in the lield of synthetic biology.
254 The field of quantum computing is still its infancy.
255 Her career in the field of art history has been illustrious.
256
He is making significant contributions to the field of microbiology.
257
He has deep understanding of the field of financial mathematics.
258
The field of criminology examines the causes of crime.
259
The field of computational linguistics combines language and computing.
260
He is a leader in the field of pharmacology.
261 The field of actuarial science helps manage financial risks.
262 The field of environmental economics addressos the impact of economic activity
on natural resources.
263 Her work in the field of cognitive neuroscience investigates brain function.
264
Her research in the field of behavioral ecology examines animal behavior.
265
Her research in the field of nutrition has led healthier eating guidelines.
266
The field of developmental linguistics studies language acquisition in children.
267
The field of gerontology explores aging and its effects on individuals and soci- etics.
268
The field of dialectology studies regional differences in language. She has dedicated her life to the field of humanitarian aid.
269
270
The field of information technology is constantly changing.
271
He is an expert in the field of acoustical engineering,
272
The field of developmental psychology studies human growth and development.
273
She is advancing the field of artificial intelligence.
274
The field of climate science studies weather and climate change.
275
He is a thought leader in the field of sustainable development.
276 The field of evolutionary ecology examines the adaptation of organisms to their
environments.
277 He has made notable contributions to the field of computer engineering.
278 She is a distinguished rescarcher in the field of plant biology.
279
Her research in the field of conservation biology protects biodiversity.
280 The field of computational chemistry models chemical structures and reactions.
281 His work in the field of artificial neural networks is groundbreaking.
282 Her work in the field of atmospheric science predicts weather patterns.
283 The field of synthetic chemistry creates new compounds.
284 The field of archaeology uncovers the secrets of ancient civilizations.
285
The field of chemistry explores the properties of matter.
286 Advances in the field of medicine have improved patient outcomes significantly.
287 He has a deep understanding of the field of artificial intelligence.
288 Her contributions to the field of visual perception are significant.
289 He has made siguificant contributions to the field of behavioral economics. Her expertise in the field of environmental sociology addresses human- environment interactions.
290
291 He is a respected voice in the field of climatology.
292
The field of visual arts encompasses painting, sculpture, and more.
293
He has made significant advances in the field of cognitive robotics.
Continued on nexl page
91

---

Published in Transactions on Machine Learning Research (02/2025)
Continued from previous page
Index
Sentence
294
The field of cyber security is critical in today's digital world.
295
The field of geology studies the Earth's physical structure.
296
The field of educational psychology applies psychology to educational settings.
297
The field of behavioral genetics investigates the genetic basis of behavior.
298
He is a pioneer in the field of computational photography.
299
He has a background in the field of social psychology.
300 She has a strong interest in the field of sociology.
301 Her work in the field of psychology has been groundbreaking.
302 He has deep understanding of the field of environmental microbiology.
303
The field of cognitive neuroscience studies the biological basis of coguition.
304
She is a prominent researcher in the field of computer vision.
305
He is an expert in the field of social network analysis.
The field of computational linguistics develops algorithms for natural language 306 processing.
307 The field of ergonomics designs equipment to improve human use.
308
He is an expert in the field of urban sociology.
309 She chose to specialize in the field of bioinformatics.
310
The field of educational psychology applies psychological principles to educa- tion.
311 The field of artificial intelligence is evolving rapidly.
312 The field of developmental biology examines the growth of organisms.
313
The field of occupational therapy helps people perform daily activities.
314 The field of neuropsychology studies the brain-bchavior relationship.
315 He is a leader in the field of sustainable agriculture.
316 Her expertise in the field of digital authropology explores online cultures.
317 He is a leader in the field of renewable energy.
318
The field of economics encompasses a wide range of topics.
319 He has notable career in the field of emergency management.
320 The field of population genetics investigales genelic varialion within popula- tions.
321 The field of cultural heritage management preserves historical artifacts.
322
The field of computational genomics analyzes genetic data using computational methods.
323 The field of cultural sociology examines cultural terns and practices.
324 He is conducting research in the field of renewable energy.
325
The field of biomedical informatics combines healthcare and data science.
326
The field of astrophysics explores the mysteries of the universe.
327
The field of cryptography focuses on securing communication.
328 The field of computational physics uses numerical methods to study physical
phenomena.
329 She has a degree in the field of marine ecology.
330 Her work in the field of museum studies enhances cultural preservation.
331 The field of environmental toxicology studies the effects of pollutants.
332 The field of media studies examines how media affects society.
333 He is a prominent figure in the field of data analytics.
334 He has a background in the field of industrial psychology.
335 Her research in the field of child psychology is pioneering.
336 His contributions to the field of artificial intelligence are notable.
337 The field of biomedical engineering innovates healtheare technologies.
Continued on next page
92

---

Published in Transactions on Machine Learning Research (02/2025)
Continued from previous page
Index
Sentence
338
He is a leading expert in the field of developmental economics.
339
He is a recognized authority in the field of bioinformatics.
340
The field of neuroeconomies combines neuroscience, psychology, and economics.
341
He is a pioneer in the field of artificial life research.
342
She is an authority in the field of biomedical engineering.
343
The field of archaeology uncovers the mysteries of ancient civilizations.
344
The field of computational social science uses data to study social phenomena.
345
She has published extensively in the field of medieval literature.
346 research.
Her work the field of digital humanities bridges technology and humanities
347
She is an authority in the field of network security.
348
He has deep understanding of the field of nanomaterials.
349 He is an authority in the field of digital anthropology.
350 The field of game design creates interactive entertainment experiences.
351 Her research in the field of evolutionary linguistics explores language evolution.
352 He is a leading researcher in the field of human rights law.
353
The field of forensic science is crucial for solving crimes.
354 He is a leading expert in the field of urban ecology.
355 The field of marine biology studies ocean ecosystems. Her innovative ideas have reshaped the field of urban design.
356 The field of biomedical sciences advances medical knowledge.
357
His career in the field of nanotechnology is flourishing.
358 She is a leading expert in the field of genetics.
359
The field of data science is transforming industries worldwide.
360
The field of genetic engineering is hot topic in scientific circles.
361 He is well-versed in the field of cultural anthropology.
362
The field of ethnomusicology studies music within cultural contexts.
363 The field of psychometrics measures psychological traits and abilities.
364 Her work in the field of game theory has practical applications in economics.
365
She is a thought leader in the field of educational technology.
366
The field of psychology studies the human mind and behavior.
367
The field of linguist offers many fascinating areas of study.
368
The field of public policy shapes governance and society.
369 Her research in the field of linguistics has garnered international acclaim.
370 The field of international relations examines global politics.
371 Her contributions to the field of artificial intelligence are substantial.
372 He is a key player in the field of international relations.
373 The field of ecological economics integrates ecology and economics for sustain- able development.
374 The field of sociology examines social behavior and institutions.
375 He is considered a pioneer in the field of nanotechnology.
376 He is an authority in the field of risk management.
377 He is a recognized authority in the field of rehabilitation engineering, The field of cognitive science integrates psychology, neuroscience, and linguis- tics.
378
379 His expertise in the field of supply chain management is invaluable. The field of computer vision develops algorithms for interpreting visual data. 380
381
She is a thought leader in the field of environmental law.
382
The field of cybersecurity is essential for protocting information systems.
Continued on nexl page
93

---

Published in Transactions on Machine Learning Research (02/2025)
Continued from previous page
Index
Sentence
383
He is a respected scholar in the field urban planning.
384
The field of biophysics combines biology and physics principles.
385
The field of climate modeling predicts future climate changes. Her work in the field of neuroethics addresses the moral implications of neuro- science.
386
387
Her insights in the field of strategic management are highly valued.
388
She is exploring now techniques in the field of digital art.
389
The field of educational psychology helps improve teaching methods.
390
Her work in the field of social work helps vulnerable populations.
391 The field of artificial intelligence is continuously evolving.
392
He is highly respected the field of electrical engineering.
393
He has deep understanding of the field of molecular biology.
394 Her expertise in the field of computational chemistry aids drug discovery.
395 The field of urban sociology explores the dynamics of cities.
396
Her studies in the field of strategic management help businesses thrive.
397
He has made significant contributions to the field of game development.
398
Her work in the field of computational neuroscience models neural processes.
399
The field of medicine requires years of rigorous training.
400
The field of educational technology enhances teaching and learning through
technology.
Table 3: Field Of Study Sentences
94


Reviews:
[{"review_id": "udnI2zDdAq", "reviewer": "Reviewer_UoCd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths :\n\n- Comprehensive Empirical Study: The paper provides a detailed empirical evaluation of existing robust mean estimation methods, emphasizing practical performance in the high-dimensional, low-data regime.\n- Introduction of Algorithm Variants: The proposed modifications to existing algorithms enhance their practical applicability in scenarios where assumptions like Gaussianity may not hold.\n- Real-World Applicability: The paper includes experiments on meaningful real-world datasets, making the contributions more relatable and actionable for practitioners.\n- Reproducibility: The accompanying Python library is clean and scalable, providing an excellent resource for further research.\n\nWeaknesses :\n\n- Writing and Clarity: \n  * The paper lacks a strong and clear message, with convoluted writing in several sections. For instance, the contributions related to new algorithm variants are insufficiently detailed and not well-highlighted in the structure. For an example of a benchmark article that is ambitious in its scope while still managing to maintain clarity, the authors may refer to Descending through a Crowded Valley — Benchmarking Deep Learning Optimizers (Schmidt et al., 2021).\n  * The Background section is verbose, with the first three paragraphs easily condensed into one succinct paragraph. The sixth paragraph is unclear and needs elaboration.\n\n- Figures: \n  * The figures are exported as pixel-based graphics, making them difficult to read when zoomed in. Switching to vector-based graphics is recommended.\n  * Redundant labelling (e.g., y-axis and legends both reading \"error\") reduces readability.\n  * In Figure 2, the authors should explore cases where the actual corruption level deviates from the expected corruption level.\n  * Instability in some plots, such as the explosion of lee_variant in Figure 5 under high corruption, needs to be analyzed and discussed.\n\n- Link to Related Work: \n  * The connection between differential privacy and robust statistics is mentioned but not explicitly explained. An important heuristic is that robust estimators are more amenable to privatization via noise addition, and that private mechanisms (i.e. for which it is hard to test membership) are naturally robust. Suggested references include:\nDifferential Privacy and Robust Statistics (Dwork et al., 2009)\nFrom Robustness to Privacy and Back (Asi et al., 2023).\n\n- Testing Framework: The focus on real-world data is valuable, but additional experiments on synthetic data deviating from Gaussianity are necessary to fully characterize the algorithms' robustness, and to understant the failing situations of each algorithm.\n\nTerminology and Typos: \n\n- Page 2: The term \"isometric covariance\" should likely be replaced with \"isotropic covariance\" for clarity and consistency with standard terminology. To be clear, the only isometric (i.e. orthogonal) covariance matrix is the identity matrix, so what is written is not wrong, yet the formulation is confusing.\n- The statement \"this can be shown to concentrate tightly\" should specify the concentration sense (e.g., with high probability).\n- The explanation of the trace of the covariance matrix (\"where Tr(Σ) is the trace of the covariance matrix\") appears in a confusing order and should be clarified.\n- The reference to Lee & Valiant (2022) for the scaling of Gaussian variance with the trace of the covariance matrix is unnecessary; this is a basic property of Gaussian distributions.\n\n- Title: The title does not reflect the empirical nature of the work. A clearer alternative could be: Empirical Evaluation of Robust High-Dimensional Mean Estimation Techniques.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on the challenge of robust mean estimation in high-dimensional settings with limited data, where traditional algorithms often fail to achieve near-optimal performance. The authors experimentally evaluate various robust mean estimation techniques and find that, with certain modifications to outlier pruning, these methods can achieve similar performance to the sample mean of uncorrupted inlier data, even with very low data sizes. The authors test these methods on synthetic Gaussian data, and real-world data, including embeddings from large language models. The paper also contributes a Python library for robust mean estimation, promoting reproducibility and encouraging further research.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Improve Writing and Structure (Critical):\n  * Rewrite sections for clarity and conciseness, especially the Background section, condensing the first three paragraphs and expanding on the sixth paragraph.\n  * Better highlight the new algorithm variants, including detailed motivation for their introduction.\nFigures:\n\n- Use vector-based graphics for better readability.\n\n- Fix formatting issues, such as redundant labels in figures.\n\n- Expand Figure 2 to study mismatches between expected and actual corruption levels.\n\n- Discuss and analyze instabilities in plots, such as the explosion of lee_variant in Figure 5.\n\n- Discuss Related Work:\n  Explicitly explain the link between differential privacy and robust statistics, incorporating the references:\nDifferential Privacy and Robust Statistics (Dwork et al., 2009)\nFrom Robustness to Privacy and Back (Asi et al., 2023).\n\n- Testing Framework:\nConduct experiments on synthetic data with non-Gaussian distributions to evaluate algorithm robustness under varying distributional assumptions.\n\n- Fix Terminology and Typos\n\n- Revise Title:\nUpdate the title to better reflect the paper’s empirical evaluation focus, such as Empirical Evaluation of Robust High-Dimensional Mean Estimation Techniques.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aOb50wXWLi", "reviewer": "Reviewer_khgm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the paper is easy to read and well-written. The proposed eigenvalue bound is simple but useful in their modification in existing methods for the considered problem setting. \nNumerous experiments are conducted, and they shows extensive comparisons between the methods. Overall, I feel this paper is survey-like paper about empirical study of the robust mean estimate in high-dimension and low data size, which is of interest in ML community. \n\nWeaknesses.\n1. Although experiments on the estimators are sufficiently conducted (which I enjoyed), I think overall contribution regarding theoretical view is insufficient. For instance, could the authors provide reasoning or intuition behind why QUE generally performs better than others in their experiment settings? \n2. It is difficult to distinguish the curves in Figure. A better visualization would be helpful as I think experiments are main contribution. Is it possible to provide enlarged version of some portion of figures? or in the top left figures, are curves at n=20 necessary? it is also shown in the top right figure.\n3. For synthetic experiments, despite lots of corruption scenarios are considered, it only provides Gaussian for base data distribution. Including experiments with other data distributions like Poisson, Multinomial, or mixture of Gaussians would strengthen the empirical analysis.\n\nQuestions.\n\n1. Isn't the sample mean a maximum likelihood estimator for the mean of Gaussian? Perhaps, this is the reason why the sample mean yields best performance on uncorrupted experiments? If so, it would be good to relate this with the empirical findings.\n2. Isn't **good_sample_mean_error** evaluated on uncorrupted data? If it is, in the bottom left Figure 3, how does LRV outperform the good_sample_mean_error in high-dim? I could not understand how we get a better result with corrupted data than uncorrupted data.\n3. In Section 5.1, \"LOOCV under the sample mean represents a valuable baseline for comparison as it demonstrates the minimum error to be expected across this data set.\", why would the sample mean perform with minimum error?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies robust mean estimators in high-dimension and low data size settings. The authors survey numerous robust mean estimators and summarized the methods. For some estimators, the authors correct parameters and steps based on their theoretical results so that the estimators perform also well in relatively low data size settings. They extensively compare those robust mean estimators on Gaussian synthetic data under uncorrupted and corrupted settings, as well as on real-world dataset other than Gaussian using LLMs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to Weaknesses and Questions above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kwBlCiU2Lz", "reviewer": "Reviewer_DdfH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n---------------\n+ The paper performs extensive experiments over a wide range of settings.\n+ It studies many different methods and proposes their own sensible versions, which seem to work very well\n+ The real data experiments deviate from the simple simulation settings and are quite interesting\n+ Authors provide code and a Python library\n\nWeaknesses\n---------------\n+ The paper is relatively long and hard to read in places.\n+ Notation is sometimes not explained\n+ The applications, though interesting, do not necessarily help with clarity", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper analysis different mean estimation approaches under contamination in the case of relatively low sample sizes and in particular when the number of dimensions exceeds the number of observations. They analyse several scenarios and different contamination regimes always assuming a samples from a Gaussian distribution that are contaminated by varying distributions Q. They summarize and include various algorithms and also add their own versions of some of those algorithms in their experiments. Finally they end with interesting real-world experiments using neural nets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper is definitely interesting and discusses an important topic. It's biggest problem from my side is the readability. This is in part due to the language used; sometimes\nsentences do not seem to make sense grammatically, as well as a lack of explanations/definitions. I give some examples below, but another careful reading should be done to identify these sentences. Due to the nature of the paper which summarizes many different concepts, readability and clarity of the language used is crucial here. Letting it proofread by a third person might be helpful.\n\n\nGeneral: \n------------\n+ In general, I am satisfied with the range of included methods. For general robust estimation, it is also possible to use MMD minimization, as, e.g., in this paper https://arxiv.org/pdf/1912.05737. They discuss the mean estimation of a Gaussian distribution in Section 4. The authors do not need to follow this suggestion, but if it is possible to include it, I would be extremely curious how this MMD estimation would perform in this benchmark.\n+ You should also explain the  \\Omega, O notation, maybe even in the \"key notation\" table. In general make sure that you define the symbols used for a smoother reading experience. \n+ Brackets around the citations seem to be missing. For instance at the end of page 2, I would write \"the data to be adversarially corrupted from X (Huber (1946);...).\n\nIntroduction: \n------------\n+ \"data lies in a very high-dimensional setting\"  --> maybe change to something like \"data lies in/is part of high-dimensional spaces\"\n+ \"However, in the corrupted setting, we find that some of the theoretical improvements, after some adjustments, are the best performers signficantly besting the sample mean.\" --> Maybe improve this sentence, it was very confusing for me upon first reading. \n+ \"We also consider some cases of unknown covariance, but as straight-forward adaptations of these approaches towards estimating covariance further stresses the need for data size n as a function of d, we mostly leave this scenario for future work.\" This sentence is very confusing to me, it might make sense to split it up.\n\nBackground\n----------------\n\"For many years in high dimensions, practitioners were faced with either potentially large error (e.g., on order of η √ d) in using the sample mean or other generalizations of the median\" --> other? Again seems to be a bit weird.\n\n\nSection 5: \n------------\nI am not familiar with embeddings, and so Section 5 was somewhat hard to read. As the first part of the paper is interesting to statisticians like me that are likely not very familiar with this topic, it could make sense to give more details here. These details should concern in what space the means are taken and what this mean should exactly represent: What is an in-context embedding? What is an embedding dimension? Does one simply map a word to Rq where q is the embedding dimension?\n\nSection 6: \n------------\n+ I would not use \"catastrophic failure\"; it seems a bit unscientific but also does not really convey what the problem is exactly (it could be that the method breaks down or just performs badly). Clearer would be something like \"low performance\" or even \"abysmal performance\", if the emphasis should be on that.\n+ \"QUE_low_n begins to perform better with much less data size than ev_filtering_low_n, but this is unsurprising given the rapid convergence of the inlier top eigenvalue to 1.\"\nI am confused, should it not be QUE and ev_filtering here instead of QUE_low_n and ev_filtering_low_n?", "broader_impact_concerns": "No broader Impact concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All of the reviewers were in favor of accepting this paper, and I agree with them.
Decision: Accept (poster)