KAIROS: Scalable Model-Agnostic Data Valuation
Authors: Jiongli Zhu
Parjanya Prajakta Prashant
Alex Cloninger
Babak Salimi
Venue: NeurIPS 2025 poster ()
Keywords: data valuation
mmd
model-agnostic
data-centric
Primary area: general_machine_learning
TL;DR: A principled and scalable method for model-agnostic data valuation

Abstract: Data valuation techniques quantify each training example's contribution to model performance, providing a principled basis for data cleaning, acquisition, and selection. Existing valuation methods remain inadequate: \emph{model-based} techniques depend on a single fitted model and inherit its biases, while \emph{algorithm-based} approaches like Data Shapley scale poorly due to their need to train multiple models. Recent work has proposed model-agnostic alternatives based on Wasserstein distance between the training set and a clean reference set, but exact computation is expensive and approximations often misrank examples. We introduce KAIROS, a model-agnostic framework that values examples by their contribution to the Maximum Mean Discrepancy (MMD) between the training set and a clean reference distribution. Unlike Wasserstein methods, MMD admits a closed-form solution that requires no approximations and is scalable to large datasets. Additionally, KAIROS enables efficient online valuation: adding a new batch of $m$ examples requires only $O(mN)$ computation to update all scores, compared to $O(N^2)$ in prior work where $N$ is the training set size. Empirical evaluations on noise, mislabeling, and poisoning benchmarks show that KAIROS consistently outperforms state-of-the-art baselines in both accuracy and runtime. On ImageNet, KAIROS achieves up to 15 $\times$ speedup over the fastest baseline while maintaining superior data valuation quality. Our results demonstrate that model-agnostic methods can match or exceed model-based approaches in performance while scaling to large datasets.

KAIROS: Scalable Model-Agnostic Data Valuation
Jiongli Zhu", Parjanya Prajakta Prashant', Alex Cloninger, Babak Salimi
University of California San Diego
jiz1430ucad. pprashant@ucsd. edu, oclooingerrbucsd.dde asalimi Stalimi@uccddco leccsd.ett edu
Abstract
Data valuation techniques quantify each training example's contribution to model performance, providing a principled basis for data cleaning, acquisition, and se- lection. Existing valuation methods remain inadequate: model- bbased techniques depend on a single fitted model and inherit its biases, while algorithm based ap- proaches like Data Shapley scale poorly due their need to train multiple models. Recent work has proposed model-agnostic altermatives based on Wasserstein dis- tance between the training set and clean reference set, but exact computation is expensive and approximations often misrank examples. We introduce KAIROS, a model-agnostie framework that values examples by their contribution the Maxi- mum Mean Discrepancy (MMD) between the training set and a clean reference distribution. Unlike Wasserstein methods, MMD admits closed-form solution that requires approximations and is scalable to large datasets. Additionally, KAIROS enables efficient online valuation: adding a new batch of examples requires only (mmN) computation to update all scores, compared to O(N²) in prior work where N the training set size. Empirical evaluations on noise, mislabeling, and poisoning benchmarks show that KAIROS consistently outperforms state-of-the-ard baselines in both accuracy and runtime. On ImageNet, KAIROS achieves up 15 x speedup over the fastest baseline while maintaining superior data valuation quality. Our results demonstrate that model-agnostic methods can match or exceed model-based approaches in performance while scaling to large datasets.
Introduction
The performance and behavior of AI systems significantly depend on the training data. The quality of data affects accuracy [63] robustness [41] and safety [9]. Data valuation methods, which quantify each training sample's contribution model performance, are therefore essential for developing -egh-performacc and reliable models. Existing valuation methods broadly fall into two categories. Model-based techniques, such as influence functions [33] and TracIn [47], evaluate the effect of individual data points on single trained model, so their scores depend on the specific training run and hyperparameter settings 3T Algorüthn-bused methods, e.g., Data Shapley [22], estimate each point's marginal contribution by averaging over many retrains, which becomes computationally infeasible for modern large-scale datasets (2 27 Neither category provides valuations that are both consistent and tractable on -xxmmee datasets.
Commercial and legal pressures amplify the need for model-agnostic data valuation methods. For large models, training datasets come from two primary sources: web-crawled data and purchased high-quality datasets. Web-crawled data noisy, containing duplicates (38] and potential poison- ing [25), making it essential to identify and filter harmful examples before training. High-quality licensed datasets, such as those from The Times (53 and Shutterstock (57), are expensive, requiring careful assessment of their value. Since training frontier models takes months and costs millions
These authors contributed equally to this work.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Wasseratiin LoO Rank
1000
Wasserstein LDD Rank
1000
1000
Accumulsted Runtiree (5)
KAIRDS
750
750
750
400
500
500
MMD LOO
500
200
250
250
250
0
500
1000
500
100
Wasserstein
Sikkhonn Dual Potential Rank
MMD Influence Rank
Batches
Dual Prttentiaal Rank
[LAVA]
(KAIROS)
(a)
(b)
(c)
(d)
Figure I: Comparison of Wasserstein- and MMD-based influence methods. (a) Leave-one-out (LOO)
ranks versus dual potentials for unregularised optimal transport. (b) LOO ranks versus Sinkhorn dual
potentials (LAVA). (c) LOO ranks versus MMD directional derivatives (K ATROS), which lie almost perfectly on the diagonal. Among the top-100 100 points, LAVA overlaps with true LOO rankings by 60% while KAIROS achieves 99% overlap. (d) Online runtime: KAIROS scales linearly with number of batches, whereas LAVA grows quadratically.
of dollars companies typically train only once. This means data valuation must occur before training begins, ruling out model-baser approaches. From the legal perspective, the EU AI Act Article 10 [ requires training data to be "relevant, sufficienlly representative, and to the best extent possible, free of errors," with mandated assessment of data availability, quantity, and suitability. Criti- cally, these requirements apply to the data itself, not to any specific model, making model-agnostic valuation essential for regulatory compliance. This need has motivated model-agnostic techniques such as LAVA 30 that value data before model training.
In this work, introduce KAIROS, scalable framework for model-agnostic data valuation. Each training example receives a distributional influence score: the change induces in statistical diver- gence between the empirical distribution P and trusted reference Q when that example is removed. To avoid prohibitive leave-one-out out-out retraining costs, we define this score as the directional derivative of the divergence with respect to infinitesimal up-weighting. For popular divergences like Wasserstein-L 1, computing this derivative exactly is infeasible. Existing methods approximate it using Kantorovich dual potentials or its entropically-regulare
Sinkhorn variant, in LAVa (30). We show these approximations produce rankings that drift from the true I.00 ordering, over-valuing some points and under-valuing others (Figures Ta Tbp.
KAIROS instead uses Maximum Mean Discrepancy (MMD) as the divergence metric. We derive a closed-form directional derivative for MMD that matches LOO rankings (Figure Ic) and naturally
incorporates label information. Our main contributions include:
1. We provide closed- form influence function for MMD with O(1/N2) error from true leave-one-out valuations.
2. Computing influence scores requires model training or iterative optimization, taking only (m)N time and O(N) memory for batch size m. This enables efficient valuation for web-scale data, especially in the online setting (Figure
3. We establish that KAIROS satisfies symmetry and density-separate axioms, ensuring fair rankings for points that contribute equally and clear separation low- and high-quality data.
The framework extends to conditional MMD kernels, allowing KATROS to incorporate label information for supervised tasks.
5. Extensive experiments on label-noise, feature- and back-door benchmarks demonstrate that KAIROS (i) more accurately detects corruptions, (ii) preserves accuracy when pruning low-value points and sharply degrades it when removing high-value points, and (iii) runs up 50x faster than prior methods LAVA and KNNSHAPLEY. Code available at Github
2 Related Work
Data valuation methods broadly fall into three categories: model-based, algorithm based, and model- agnostic approaches. Model- based methods include influence functions [33] 24, 8. trajectory based approaches [47 4 68 ; and post-hoe kernel approximations like TRAK 46 and DAVINZ 77 These methods produce model-specific valuations that may vary across different models trained on the same data. Algorithm-baset methods, particularly Data Shapley [22]. define value in terms of marginal contributions to specific learning algorithms, with various approximations developed to avoid the

---

prohibitive cost of training 2N models [28 54 21 68 35 67]. For bagging models, Data- BOB and 2D-O0B [3 61] provide efficient alternatives using out-of-bag estimation. Algorithm-agmn agnostic approaches like LAVA 30 quantify contributions based on dataset distance but suffer from poor performance on label error detection [29], computational complexity of O(n²), indeterministic approximations, and memory issues, which SAVA [32) partially addresses through batching strategies for Wasserstein computation. A detailed literature review is provided in Appendix A
3 Data Valuation via Distributional Influence
We cast data valuation in the framework of distributional sensitivity analysis. As in standard settings, given training set Dimin = {x yt) ) <=1 drawn i.i.d. from a noisy distribution Q, and Ji=1 ) from clean target distribution P. For a candidate point (z,y), we ask: how much does it contribute to the distance between P and Q? Concretely, we quantify the value the change in distributional distance when the empirical training distribution is infinitesimally perturbed toward the Dirac mass at (x., y). Since this influence depends only on P, Q, and distributional distance d( P, Q), it defines algorithm-agnoste valuation score.
validation set D {(r,, y4)
The remainder of this section formalizes this idea ($3.T) derives a closed-form score using MMD (3 (3.2). extends it to labels using a conditional extension of MMD ($ 3.3 and presents an efficient algorithm for its computation ($ (3.4..
3.1 Distributional Distances
In the absence of a learning algorithm, the distributional distance between P and Q serves as a surrogate for the -rain-validation risk gap. Several distance measures such as Total Variation, Wasserstein distance (65 66 and MMD (33 have been shown upper bound the absolute difference between training and validation losses 42 51 12]. We therefore value a training point by (the negative of) its contribution the distance measure [30]. Specifically, define this contribution by
the influence function (37] of the distributional distance.
Definition (Distributional influence). Let d -M M -R be Gaateau--diffrreniiable on the space
M of probability measures. The influence of a point x is
I,(x; d(P, (1 - s)Q + eδ.) d(P.Q)
+
where 6g denotes the Dirac measure
For finite samples, the influence function approximates the leave-one-out valuation with an error of Q11/ngan ) (64 Chapter 20, p. 291].
Choosing a distance. A distance d suitable for data valuation must (i) admit a tractable influence formula and (ii) upper bound the train-validation loss gap. f-divergences form important family of distance measures, which include popular metrics such as Bellbacck-eibler (KL) divergence [4. 21. However, they require density-ratio estimation (unstable in high dimensions 600 and are not well-defined whenever the support of P extends beyond that of Q. Integral Probability Metries (IPMs) 44 58) avoid these pitfalls by not requiring density ratios in their definition or computation. Given two distributions P and Q, the IPM is defined d( P, Q) = sup [E]]]]] [o]/]] for suitable function class F, where f is called the critic function that aims to distinguish between P and Q. IPMs encompass various distance measures including Wasserstein-1 65]. MMD 23]. and Total Variation Distance [44].
IPM Influence decomposition.
Substituting the IPM into the influence definition (Eq. gives
IFP (2; P,Q) sup [EP//] - E E(1-e))+ed, [f]] - supp [PPlf] - Eolf]]
(2)
2-+0+
Bolf: fr] Eplf -
-
1-
(i critic-shift term
(ii) point-gap term
Let f* and f denote the optimal crities before and after up up-weighting a point by an infinitesimal mass e. The critic-shifi term in the IPM influence function depends on x and is generally intractable. If the difference (J² f*) decays as O(ga) for some > 1, the critic-shift term vanishes and Equation (3) collapses to the simpler point-gap term. However, this decay need not hold for all
IPMs; in particular, it does not hold for -
39 p. 39-40]. In the Wasserstein-L I case, f
3

---

corresponds to a Kantorovich potential, which is non-umique [66 559 16 Discarding the critic-shift
term therefore yields influence values that are -on-deterministic
Current model-agnostic data-valuation methods like LAVA (30] rely on this simplification, retaining only the point-gap term for the Wasserstein-1 metric. This introduces two significant issues. (i) Since the dual critic is non-unique, the resulting influence values can vary arbitrarily, violating determinism. (ii) To enhance scalability, LAVA replaces the exact Wasserstein-1 objective with its Sinkhorn-regularized counterpart; however, this introduces a bias of order O(dv log(1/v)). where d is the data dimensionality and is the regularization strength [20]. Correcting this bias is computationally expensive. and even as V + O, the neglected critic-shiff term remains unresolved, causing the resulting influence scores to deviate substantially from the true leave-one-our rankings, the stated objective of the method (Figure Ib).
3.2 Closed-form Influence via MMD
We address these challenges by using MMD 23
IPM with closed-form expression for the distance. We derive closed-form influence function that can be computed directly without first computing the distance itself. This section focuses on the marginal distribution of features. We incorporate label information in subsection between them is defined as:
3.3 Given two distributions P and Q, the MMD distance
MMD(P,Q) sup (fx=e(((x)) Ex-o(f(=))) = 1 - pe ma 1illsS
(4)
where μup = Ez-r(o(x)] and no Ee- [6(x)] are the kernel mean embeddings in the RKHS
H. and the kernel k(z, x²) ) = (o(x), (xx')) (23]. While computing I|wp pell directly can be
challenging, the squared distance admits a closed-form expression via the kernel trick:
HIMD E,,'' (ake, z²)] + Ex,='=((h(e, x') P., (kk((,, z')].
(5)
This enables a closed-form computation of the influence with respect to MMD via the chain rule. Proposition 1. The influence function for MMD as the distance metric is, up to additive and positive multiplicative constants, given by
IF (z; P. P,Q)= = (k(oo [x]] BR- [k((z,,)))
(6)
The full derivation is provided in Appendix B.I
Note that for downstream tasks such as feature error detection, backdoor attack identification, or ranking the most and least valuable points, only the relative rankings are important (35 671; hence, the additive and multiplicative constants in the influence function do not affect the outcome. Henceforth, we use I-( to denote the rescaled version Dimin, the unbiased
of IF (. P. Q), omitting P and Q for brevity. For training point sample estimate of its JIMD-based influence is given by:
Phaa
(x:)
ka
val 225)
k(a
faaja xx:);
Royal
Pirain
where the first term approximates Eg'= p [k(2", .x))] using validation data and the second term approx- imates Egrn [k(x', =x) using the training data excluding z,;
Properties. Beyond closely approximating leave-one-our scores 664, our MMD-based influence enjoys two nice properties: (i) Symmetry ensures that points making the same marginal contribution to the MMD (the MMD estimate) receive identical influence, yielding fair rankings; (ii) Density- sepuration guarantees the existence of a global threshold that cleanly partitions regions where the
validation distribution dominates (P > Q) from those where the training distribution dominates
(Q > P), enabling high-accuracy in detecting noisy samples and backdoor attacks.
Proposition 2 (Symmetry). Let Dirain and Dval be finite samples from distributions Q and P. respectively. for all subsets S C Dinin (01,2zh
(x.H) - MMD(D"", SS) = MMD(D ,SU fa, MMD(D MMD(D"" SSU SS
then [x = (z2).
See Appendix B. 2 for proof.

---

Proposition 3 (Density Separation). Let P and Q be two probability distributions on X C R.. For any > 0 and > O. there exists a Gaussian isotropic kernel k such that for IF(x) Bg [k(,,=)]] R [K( x') x')}, the following holds
[F(x) > 0, P(''' Q(²)) 2 eV E 3(2,r). [F) < <0, P('') Q(z') -e B(a,r)
where ((x,r) {z':
||>
=|| <}
In Appendix [.3 we give full proof of Proposition 3 and we also present a real-world example illustrating its practical effect, where the MMD-based influence scores for clean versus corrupted points are perfectly split by a near-zero threshold, whereas the Wasserstein- based scores exhibit substantial overlap and admit no such clean cutoff.
3.3 Capturing Feature-Label Correlations with MCMD
Marginal MMD over P(X) and Q(X) effectively detects covariate shift but overlooks label-specific coruptions- as flipped labels, back-door triggers, or concept drift- that alter P(Y) (X) without changing P(X). Directly operating the joint distribution (X, Y) using a product kernel often reduces sensitivity to label noise: in high-dimensional feature spaces, the geometry of X dominates the kernel distances, causing small perturbations in labels Y to barely shift the joint embedding, and diminishing the test's 's power as dimensionality grows [49 [0]. We therefore retain marginal MMD for detecting feature- anomalies, and augment it with the expected value of Maximum Conditional Mean Discrepancy (MCMD) [52] 45), a conditional extension MMD denoted as E-MCMD. This hybrid criterion enables KATROS to identify both covariate and label anomalies within a unified, kernel-based theoretical framework.
Definition (Maximum Conditional Mean Discrepancy (MCMD) 52 conditional distributions P(Y and Q(Y (X) at point is: (CMDP.0(z) == 11 (x) Hy (*))
The MCMD between
where - dP(y)z)) and Q (x)x(x)= (x)
((y) dQ(l)zz) are known the conditional
kernel mean embeddings.
To aggregate this measure across covariate space, we take the expectation of MCMD(.) over the training distribution i.e. E-MCMD(P, Q) = E.-.q/MCMDr:.((tt)) We derive the influence for E-MCMD via Definition
Proposition 4. The influence function for F-MCMD as the distance metric is, up to additive and positive multiplicative constants, given by
IFF-MMMM (2, y P,Q) - [| kyy)
(8)
The full derivation is provided in Appendix B.4 Henceforth, we use IF (F, y) to denote the rescaled version of IFFMMMMM ((x, y); P, Q), omitting P and Q for brevity. We can simplify this 1}). For categorical labels, consider the
expression when Y is categorical (Y E (0, 1,......,
feature map dby) ey E C and the kernel k(y,y') { y'} where by is the y-th standard basis vector. For this kernel and mapping function, we have "V1x (r) [P(O)x)..... P(C - [/=)]T
which is simply the probability vector for each class conditioned on Therefore:
IF cond (r. u) - (x) evll2
P(yl (P(w 1)2.
(9)
For the finite sample case, X can be estimated by a classifier trained on D that returns Ply]z). The finite sample estimator for the conditional influence is:
(z.a((..)) (z.y)=
Σ
P(y' x))² + (P(yl=) - 1)a.
(10)
V'y
Combined Net Distance and Influence. Finally, we integrate both marginal and conditional
discrepancies into a single "net" distance:
where À > 0 balances the two terms. The overall influence of (x,y) on due is simply (11)
deei (P, Q) = (1 - A) MMD(Px, Qx) + A F-MCCDD(P.Q)
IF (2, v) (1 A) IF(x) À 1l y).
(12)
5

---

Generalization Error Bound. We establish theoretical link between the net discrepancy dnet and downstream model performance. Under mild regularity assumptions, the expected cain-validation loss gap is bounded above by the sum of the marginal MMT) and conditional F-MCMD. Conse- quently, removing a point which decreases this distance provably tightens the out-of-distribution error bound for any learning algorithm. The influence is first-order approximation of the effect of removing point, suggesting that removing points with large influence scores could decrease the out-of-distribution error. A concise statement follows, with the complete theorem and proof in Appendix B.5
Theorem 1 (Bounding transfer loss (simplified)). Let (X, .x) and (Y, dy) be compact metric spaces and = x x V. Let L Z - IR be a continuous loss function. Then, for some constant c,
B(r,)) (4 w)] & ((x,))-((((,, + o( (MMD(PX,Qx) Qx) Ex-Qx [MCMDP.(=e))
3.4 Batch Computation and Streaming Updates
Modern ML pipelines often require continuous data valuation include language models trained on fresh web crawls or recommender systems processing new user interactions. Recomputing influence scores from scratch after each update is prohibitively expensive, scaling as ((N2) with dataset size. We address this with a two-phase approach. The offline initialization scans the initial training and validation sets once, computing and caching three quantities per training point: the average training kernel similarity A, the average validation kernel similarity B, and the label residual R, The online update then processes each incoming batch efficiently. For new points, we compute their statistics from scratch. For existing points, we update only A; using kernel evaluations between old and new points. This approach maintains accurate valuations in streaming settings without quadratic recomputation.
data arrives. Examples
Offline Algorithm. In the standard offline setting, we assume access to the entire dataset and classifier trained on D that provides predicted probability vectors g. P(ylzz; for training points. For each training point (Iis yi) E Dirain, we precompute three key quantities:
A
M
tainn
.I), B.
val k(a
R.
D:
train
||
(13)
Prrain
Pvval
1=1
Using these precomputed values, the feature influence is II(;;; B: -A, (from Equation
and the conditional influence IF (x yi) = Rg (from Equation 100 Therefore, the final net influence score is:
iF ne y1) = (1 A)(B: A.) + XR.
(14)
The time complexity for this algorithm is O + Thrain nva)), which simplifies to O(n since typically Auain Teaal
Online Algorithm. In the online setting, process data in batches. At time t, receive new batch of size m, bringing the total dataset size to n4+1 = n4 + m. We update the influence scores for both existing points and the new batch. While naive approach of recomputing everything from scratch would require O time, our elegant influence expression allows computation in
Onn: m + m²)
For existing points (z:, yi) where i S n,, the terms B; and R, remain unchanged. The training kernel average A, can be efficiently updated as:
10th
Me+1 ( A
(m.
Σ
train k
Note that this update requires computing only new kernel evaluations per existing point, not Re+1+ For new points (Inetii ti.+i) where i 1,..., m, we compute all three quantities from scratch using Equation (3). The final influence scores are then computed as IF (n;,, yi) = for points. This update procedure runs in O(,, m) for the old data and O(m²) for the new batch resulting in total complexity of O(n + m²). Detailed algorithm provided in Appendix C
(1-AR

---

KNNShapley
DataOOB
DVRL
LAVA
KAIROS
% Covered Corrupted Data
CIFAR-10
STL-10
IMDB
AG News
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.0
1.0
(a)
(b) % Inspected Data
(c)
(d)
Figure 2: Feature noise detection results.
KNNShapley
DataCOB
DVRL
LAVA
KAIROS
Covered Corrupted Data
CIFAR-10
STL-10
IMDB
AG News
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
(a)
(b)
Inspected Data
(c)
(d)
Figure 3: Label noise detection results.
4
Experiments
We implement KAIROS custom data valuation method in the OpenData Val benchmark (29 benchmark for evaluating data valuation methods. We evaluate all methods on three key applications: (1) detecting noisy data, mislabels, and mixture of them, (2) detecting malicious data injected by data poisoning attacks, and (3) data pruning by removing data with the lowest values. We also test removing data with the highest values to thoroughly examine the effectiveness of all data values, including low and high. In addition, we test the runtime of methods with varying data sizes, in offline and online settings. Experiments are repeated five times with different random seeds, and we report the mean (colored regions denote the standard deviations). We include additional experiments on the hyperparameter and kernel choice, noisy validation set, and million-scale data in Section
Datasets. We evaluate on four widely used datasets, including CIFAR-10 (34], STL-10 [0]. IMDB (43) and AG News (77). to cover both image and text modalities. In most experiments. we simulate limited clean-data availability by using 10000 noisy training examples and 300 clean validation examples, with held-out test of 10000 clean samples. For the smaller STL-10 dataset, we scale down to 3700 training, 300 validation, and 1000 test examples.
Baselines and parameepr We compare KAIROS with four satt--of-the-at art data valuation methods with different mechanisms: LAVA [0). DATAOOB [36.. DVRL [76), and KNN SHAP- LEY (27). For KAIROS, we the Gaussian kernel bandwidth to the median of all pairwise distances and fix the balancing factor in Equation (12) to 0.03. See details in Appendix D
Noise and Mislabel Detection. In this experiment, we introduce noise into 20% of the data. Following 22 30 73 72), we inject feature noises by adding white noise to the images and randomly replacing words with other words for texts, and introduce label noises by randomly changing the labels of corrupted samples to other classes. Figures 2 and present the performance of different data valuation methods in identifying corrupted samples stemming from two distinct sources of noise: feature perturbations and mislabels. Each curve plots the cumulative fraction of corrupted data recovered as a function of the percentage of training data inspected.
KAIROS consistently achieves strong performance across both noise types and all datasets. In the feature noise setting (Figure 2 it ranks noisy samples more effectively than all baselines on all datasets, especially in the early inspection phase. In particular, on AG News, all methods except KAIROS are close to the diagonal, meaning that they perform similarly to assigning random values
7

---

KNNShapley
DataOOB Poison Frog
DVRL
LAVA
KAIROS
% Covered Corrupted Data
Badnet
Style
LISA
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
(a)
(b)
Inspected Data
(c)
(d)
Figure 4: Malicious data detection results.
KNNShapley
DataOOB
DVRL
LAVA
KAIROS
STL-10 (Lowest)
STL-10 (Highest)
IMDB (Lowest)
IMDB (Highest)
0.95
Test Accuracy
0.96
0.80
0.70
0.70
0.60
0.92
0.85
0.50
0.90
0.60
0.0
0.1
0.2
0.0
0.1
0.2
0.0
0.1
0.2
0.0
0.1
0.2
(a)
(b)
Removed Data
(c)
(d)
Figure 5: Effect of removing the least valuable (a,c) and the most valuable (b,d) data points on test accuracy.
to all data points. In the presence of label noises (Figure 3 our method remains competitive and outperforms most baselines, particularly on AG News and STL-10.
Overall, while some methods show competitive performance specific datasets or noise types, such as LAVA on feature noises and DATAOOB on label noises, they fail to generalize well across both feature and label noise scenarios. In contrast, KAIROS archives top-1 detection accuracy in 6 (out of 8) scenarios, and stays top-2 for the remaining 2 cases, indicating its versatile performance noise detection.
Malicious Data Detection. We evaluate robustness under adversarially crafted poisoning attacks following (30 We test Badnet (25] and poison frogs (55) attacks CIFAR-10, and clean label style attack [88 and LISA 26 on AG News. For CIFAR-10 attacks, we inject 3% malicious data; for AG News attacks, inject 10% poisonous data. Model-based techniques are not applicable for LISA fine-tuning of LLMs due to the computational expense of training hundreds of models.
Across the four scenarios in Figure LAVA underperforms scenarios (a) and (c), while DATAOOB struggles with (b.c) and cannot handle (d). KNNSHAPPLY, DATAOOB, and DVRL show less competitive performance than KAIROS across all cases. LAVA achieves comparable performance with KAIROS only in detecting poisoned fine-tuning data but is significantly less effective for Badnet and style attacks.
Point Removal. Figure 5 (a-d) shows how test accuracy changes when removing the least or most valuable training data identified by each method. We use 20% mislabeled data. Good valuations should cause large accuracy drops when removing high-value value data and small drops (or increases) when removing wow-value data.
In both scenarios, our method yields the most desirable behavior. When pruning the least-valuable data (Figure (a. c)), on STL-10, KATROS, DATAOOB, and KNNSHAPLEY increase the accuracy by similar amount, while the values obtained from DVRL and LAVA do not help. On IMDB data, all methods except DVRL keep the test accuracy when removing 30% of the data. For both datasets. when discarding the most valuable data (Figure 5 (b, d)). KAIROS results in more significant accuracy drops than all baselines. This indicates that K KAIROS gives both meaningful low and high values to the data, while most baselines only effectively identify low-valued data.
(b,

---

Effect of Validation Sample Size. In practice, valida- tion sets are often small due to expensive labeling. To understand how many validation samples are needed to ob- tain reliable data values, we conduct noise detection with varying validation set sizes. In particular, to better com- pare the convergence of our method (based on MMD) and LAVA (based on Wasserstein), we test with 20% feature noises on IMDB data, the scenario where LAVA performs relatively well. To measure effectiienesss, detection accuracy, defined by the percentage of correctly identified corrupted data among the 20% least-valued data points. As shown in Figure 6 the variance of KAIROS, indicated by color regions, shrinks more quickly with growing validation set size than that of LAVA. Although LAVA benefits from increased validation set, it requires 9K validation samples to reach an accuracy of 0.77. In contrast, KAIROS only requires 30 samples to achieve this, This implies that MMD is more robust and reliable compared to Wasserstein under finite samples.
Deleciinn Accurcyy
1.00
KNNShapley
0.75
0.50
DVRL
0.25
LAVA
0.00
KAIRDS
10"
107
10
Validation Size
Figure 6: Label noise detection accuracy under varying validation set sizes.
adopt the
Offline Runtime. To understand the scalability of dif- ferent methods, we measure the runtime of methods on CIFAR- 10 data with label noise. We vary the training set size and keep the static validation set size 300. As shown in Figure 7 KAIROS, LAVA, and KNNSHAPLEY are similarly efficient, and are significantly faster than DVRL (10x) and DATAOOB (100x). Although DVRL's runtime grows slowly with training samples, as shown in previous results, performs poorly in most applications, making it not practically applicable. DATAOOB's ' high runtime results from training models on various bootstraps.
KNNshapley
Rusttime 14)
aaraD
100
DVRL
100
LAWA
KAIROS
5000
10000
15000
Training Samples
Figure 7: Offline runtime comparison.
Online Runtime. To show the adaptability of KAIROS in the online setting, we split 10000 of CIFAR- 10 data into 100 batches, each containing 100 samples, and feed them
Accumullata lated Buntimm (4)
-000
KNNShapley
in a streaming way. We measure the accumulated time taken to conduct data valuation after each batch update. KAIROS adopts Algorithm for value computation and updates, while LAVA and KNNSHAPLEY have no direct adaptation to this setting, thus have to re-calculate the values when a new batch comes in. DATAOOB and DVRL are omitted for this experiment as they take more than 8 hours to complete the experiment, meaning that they are not practical to use in this setting. As shown in Figure KAIROS is significantly faster than LAVA and KNNSHAPLEY in the online setting. The speedup becomes more significant when more batches are included. The speedup reaches 28x compared KNNSHAPLEY, and 50x compared to LAVA when all batches are fed.
200
LAWA
KAIROS
50
@BBatches
Figure 8: Online runtime comparison.
Large Scale Experiment on ImageNet. To evaluate KATROS at scale, we compare it to the scalable variant of LAVA, SAVA, on ImageNet (17), which contains 1.28M training images and 1000 classes. We extract features with ResNet-50 encoder and run all methods on a single NVIDIA A100 (40GB VRAM). We report the runtime and the AUC of the "% inspected data vs. % corrupted data covered" curve (also shown in Figures and 3 under both feature-noise and label-noise settings.
Method
Runtime
AUC
Method
Runtime
AUC
KAIROS
7m56s
0.869
KAiROS
7m52s
0.861
SAVA
1h58m
0.817
SAVA
1h58m
0.484
Table 1: Feature noise detection on ImageNet.
Table 2: Label noise detection ImageNet.
Our results show that KATROS significantly outperforms SAVA both efficiency and effectiveness. The efficiency gain of KATROS comes from its closed-form solution, which enables batch-based GPU acceleration and avoids expensive Sinkhorn computations. By contrast, SAVA (and lAVA) require computing pairwise conditional Wasserstein distances between P(z y) across all class pairs, leading to 1000x computations. This accounts for over 80% their runtime, making KAIROS 2 more suitable for large-scale applications.

---

Summary. Across all tasks and datasets, KAIROS consistently achieves performance gains over sfate-of-the-ar baselines. effectively ranks data under both natural and adversarial data corruptions and noises. The runtime experiments demonstrate the advantageous efficiency of KATROS compared baselines, especially in the practical online and large-scale settings.
Conclusions, Limitations and Broader Impacts
We introduce KAIROS, scalable data valuation framework that uses Maximum Mean Discrepancy to compute closed-form influence functions for detecting feature noise, label corruption, and backdoors. KAIROS achieves up to 50x speedup over existing methods with O(mNV complexity in online settings, making it practical for web-scale deployment while maintaining faithful leave-one-out rankings. Our approach provides theoretical guarantees through symmetry and density separation properties and offers model-agnostic influence scores that enable transparent data quality assessment, fairness auditing, and regulatory compliance without requiring model retraining. Current limitations include the use of fixed kernels and fixed balancing coefficient for all tasks. Future work should focus on learned kernels, efficient approximate methods, and regression extensions.
Acknowledgments and Disclosure of Funding
This research was supported by NSF award IIS-23401 24, NSF CISE CIF award 2403452, and NIH grant #54HG012510, The views, opinions, and findings presented are those of the authors and do necessarily represent those of the NSF or NIH.
10

---

References
[1] Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union L 2024/1689, 12 July 2024, July 2024. CELEX 32024R 1122411689 1899.
[2] Syed Mumtaz Ali and Samuel D Silvey. A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society: Series (Methodological), 28(1):131-142, 1966.
[3] Mohammad Mohammadi Amiri, Frederic Berdoz, and Ramesh Raskar. Fundamentals of task- agnostic data valuation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 9226-9234, 2023.
[4] Juhan Bae, Wu Lin, Jonathan Lorraine, and Roger B Grosse. Training data attribution via approximate unrolling. Advances in Neural Information Processing Systems, 37:66647-666666 2024.
[5] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jen- nifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79:151-175, 2010.
[6] Felix Biggs, Antonin Schrab, and Arthur Gretton. Mmd-fuse: Learning and combining kernels for two-sample testing without data splitting. Advances in Neural Information Processing Systems, 36:75151-75188, 2023.
[7] Yiling Chen, Yiheng Shen, and Shuran Zheng. Truthful data acquisition via peer prediction. Advances in Neural Information Processing Systems, 33:1 13:18194-18204 2020.
[8] Sang Keun Choe, Hwijeen Ahn, Juhan Bae, Kewen Zhao, Minsoo Kang, Youngseog Chung, Adithya Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, et al. What is your data worth to gpt? l1m-scale data valuation with influence functions. arXiv preprint arXiv:2405.1399 13954, 2024.
[9] Zachary Coalson, Juhan Bae, Nicholas Carlini, and Sanghyun Hong. if - guide: Influence function-guided detoxification of llms. arXiv preprint arXiv:2506.01799 2025.
[10] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
[11] Ben Cottier, Robi Rahman, Loredana Fattorini, Nestor Maslej. Tamay Besiroglu, and David Owen. The rising costs of training frontier ai models. arXiv preprint 2rXiv:2405.21015. 2024.
[12] Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamoniy. Joint distribution optimal transportation for domain adaptation. Advances in neural information processing systems, 30, 2017.
[13] Ian Covert, Chanwoo Kim, Su-In Lee, James Y Zou, and Tatsunori B Hashimoto. Stochastic amortization: unified approach accelerate feature and data attribution. Advances Neural Information Processing Systems, 37:4374-4422 2024.
[14] Imre Csiszár. Information-type measures of difference of probability distributions and indirect observation. studia scientiarum Mathematicarum Hungarica, 2:229-318, 1967.
[15] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
[16] Eustasio Del Barrio and (ean-Michel Loubes. Central limit theorems for empirical transportation cost in general dimension. 2019.
[17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEFE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.

---

[18] DLA Piper. EU AI Act's ban on prohibited practices takes effect: First compliance milestones, 2025. Accessed 10 May 2025.
[19] Damien Garreau, Wittawat Jitkrittum,, and Motonobu Kanagawa. Large sample analysis of the median heuristic. arXiv preprint arXiv:/707.07200 2017.
[20] Aude Genevay, Lénaic Chizat, Francis Bach, Marco Cuturi, and Gabriel Peyré. Sample com- plexity of sinkhorn divergences. In The 22nd international conference on artificial intelligence and statistics, pages 1574-1583. PMLR, 2019.
[21] Amirata Ghorbani, Michael Kim, and James Zou. A distributional framework for data valuation. In International Conference on Machine Learning, pages 3535-3544. PMLR, 2020.
[22] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International conference on machine learning, pages
PMLR, 2019.
[23] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
[24] Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308. 03296, 2023.
[25] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv: 1708. 06733, 2017.
[26] Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Tekin, and Ling Liu. Lisa: Lazy safety alignment for large language models against harmful fine-tuning attack. Advances in Neural Information Processing Systems, 37: 17:105521- 2024.
[27] Ruoxi Jia, David Dao, Boxin Wang. Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas J Spanos, and Dawn Song. Efficient tusk-specite data valuation for nearest neighbor algorithms. arXiv preprint
2019.
[28] Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J Spanos. Towards efficient data valuation based on the shapley value. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1167-1176. PMLR, 2019.
[29] Kevin Jiang, Weixin Liang, James Y Zou, and Yongchan Kwon. Opendataval: unified benchmark for data valuation. Advances in Neural Information Processing Systems, 36, 2023,
[30] Hoang Anh Just, Feiyang Kang, Jiachen Wang, Yi Zeng, Myeongseob Ko, Ming Jin, and Ruoxi Jia. Lava: Data valuation without pre-specified learning algorithms. arXiv preprint WxXi:23355.00054 2023.
[31] Karthik Karthikeyan and Anders Søgaard. Revisiting methods for finding influential examples. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 9877-9885, 2022. arXiv:2111 11.004633.....
[32] Samuel Kessler, Tam Le, and Vu Nguyen. Sava: Scalable learning-agnostic data valuation. arXiv preprint arXiv:2406.0 22400.0011 1300 2024.
[33] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions.
In International conference on machine learning, pages 1885-1894. PMLR, 2017.
[34] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[35] Yongchan Kwon and James Zou. Beta shapley: a unified and noise-reduced data valuation framework for machine learning. arXiv preprint 1 2021.
[36] Yongchan Kwon and James Zou. Data-oob: Out-of-bag estimate as simple and efficient data value. In International conference on machine learning, pages 18135-18152, PMLR, 2023.
12

---

[37] John Law. Robust statistics- the approach based on influence functions, 1986.
[38] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Buus Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proc. ACL, pages 8424 8445, 2022.
[39] Cyril Letrouit. Lectures on quantitative stability of optimal transport. 2025.
[40] Jiashuo Liu, Tianyu Wang, Peng Cui, and Hongseok Namkoong. On the need for a language describing distribution shifts: Illustrations on tabular datasets. Advances Neural Information Processing Systems, 1115111--1448, 2023.
[41] Xiaoqun Liu, Jiacheng Liang, Muchao Ye, and Zhaohan Xi. Robustifying aalige- large language models through clean data curation. arXiv preprint arXiv:2405.19358. 2024.
[42] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pages 97-105. PMLR, 2015.
[43] Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142-150,
2011.
[44] Alfred Müller. Integral probability metrics and their generating classes of functions. Advances applied probubility, 29(2):429-44 4433, 1997.
[45] Junhyung Park and Krikamol Muandet. A measure- measure-theoretic approach kernel conditional mean embeddings. Advances in neural information processing systems, 13:21247-21255 2020.
[46] Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclere, and Aleksander Madry. Trak: Attributing model behavior at scale. arXiv preprint arXiv:2303. /8/86, 2023.
[47] Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. In Proc. NeurlPS, pages 19920-19930, 2020.
[48] Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. Mind the style of text! adversarial and backdoor attacks based on text style transfer. arXiv preprint 1rXiv:2110.07130 2021.
[49] Aaditya Ramdas, Sashank J Reddi, Barnabas Poczos, Aarti Singh, and Larry Wasserman. Adaptivity and stattitsiionsstaitttiss tradeoffs for kernel and distance based high dimensional two sample testing. arXiv preprint arXiv: 150800050 2015.
[50] Sashank Reddi, Aaditya Ramdas, Barnaáás Péczos, Aarti Singh, and Larry Wasserman. On the high dimensional power of linear-time two sample test under mean-shift alternatives. In Artificial Intelligence and Statistics, pages 772-780. PMLR, 2015.
[51] levgen Redko, Amaury Habrard, and Mare Sebban. Theoretical analysis of domain adaptation with optimal transport. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML. PKDD 2017, Skopje, Macedonia, September 18-22, 2017, Proceedings, Part II 10, pages 737-753. Springer, 2017
[52] Yong Ren, Jun Zhu, Jialian Li, and Yucen Luo. Conditional generative moment-matching networks. Advances in Neural Information Processing Systems, 29, 2016.
[53] Reuters. Time, openai sign multi-year content deal, June 2024. Published 27 Jun 2024.
[54] Stephanie Schoch, Haifeng Xu, and Yangfeng Ji. Cs-shapley: class-wise shapley values for data valuation in classification. Advances in Neural Information Processing Systems, 35:34574-
34585, 2022.
[55] Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Du- mitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. Advances in neural information processing systems, 31, 2018.
13

---

[56] Lloyd S Shapley et al. A value for n-person games. 1953.
[57] Sherwood News. Ai deals make up third of shutterstock's revenue by 2027, June 2024. Accessed 10 May 2025.
[58] Bharath K Sriperumbudur, Arthur Gretton, Kenji Fukumizu, Bernhard Schölkopf, and Gert RG Lanckriet. Hilbert space embeddings and metrics on probability measures. The Journal of Machine Learning Research, 11:1517-1561, 2010.
[59] Thomas Staudt, Shayan Hundrieser, and Axel Munk. On the uniqueness of kantorovich potentials. SIAM Journal Mathematical Analysis, 57(2):1452-1492, 2025.
[60] Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation: A com- prehensive review (statistical experiment and its related topics). RIMS Kokyuroku (Research Institute for Mathematical Sciences, Kyoto University), 1703: 10-31, 2010.
[61] Yifan Sun, Jingyan Shen, and Yongchan Kwon. 2d-oob: Attributing data contribution through joint valuation framework. arXiv preprint 2xXv::2488.03572, 2024.
[62] Sebastian Shenghong Tay, Xinyi Xu, Chuan Sheng Foo, and Bryan Kian Hsiang Low. Incen- tivizing collaboration in machine learning via synthetic data rewards. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 9448-9456, 2022.
[63] Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: Improving Ilm pretraining via document depluiiiit and diversification. Advances in Neural Information Processing Systems, 36:53983-5399999 2023.
[64] Aad W Van der Vaart. Asymptotic statistics, volume Cambridge university press, 2000.
[65] Cédric Villani. Topics in optimal transportation, volume 58. American Mathematical Soc., 2021.
[66] Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2008.
[67] Jiachen T Wang and Ruoxi Jia. Data banzhaf: robust data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics, pages 6388-6421. PMLR, 2023.
[68] Jiachen T Wang. Prateck Mittal, Dawn Song, and Ruoxi Jia, Data shapley in one training run. arXiv preprint 2:Xx::2406.11111,, 2024.
[69] Jiachen T Wang, Tianji Yang, James Zou, Yongehan Kwon, and Ruoxi Jia. Rethinking data shapley for data selection tasks: Misleads and merits. arXiv preprint M:Xiv:2455.03877 2024.
[70] Tim Wibiral, Mohamed Karim Belaid, Maximilian Rabus, and Ansgar Scherp. Lossval: Efficient data valuation for neural networks. arXiv preprint arXiv:2412.041588 2024.
[71] Zhaoxuan Wu, Yao Shu, and Bryan Kian Hsiang Low. Davinz: Data valuation using deep neural networks at initialization. In International Conference on Machine Learning, pages 24150-24176. PMLR, 2022.
[72] Ziang Xie, Guillaume Genthial, Stanley Xie, Andrew Y Ng, and Dan Jurafsky. Noising and denoising natural language: Diverse backtranslation for grammar correction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pages 619-628, 2018.
[73] Ziang Xie, Sida I Wang, Jiwei Li, Daniel Lévy, Aiming Nie, Dan Jurafsky, and Andrew Y Ng. Data noising smoothing in neural network language models. arXiv preprint arXiv:1703.0257. 2017.
[74] Xinyi Xu, Shuaiqi Wang, Chuan Sheng Foo, Bryan Kian Hsiang Low, and Giulia Fanti. Data distribution valuation. Advances Neural Information Processing Systems, 37:2407-2448. 2024.
14

---

[75] Xinyi Xu, Zhaoxuan Wu, Chuan Sheng Foo, and Bryan Kian Hsiang Low. Validation free and replication robust volume-based data valuation. Advances Neural Information Processing Systems, 34: 10837718888 2021.
[76] Jinsung Yoon, Sercan Arik, and Tomas Pfister. Data valuation using reinforcement learning, In International Conference on Machine Learning, pages 10842-10851. PMLR, 2020.
[77] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
[78] Xiao Zhou, Yong Lin, Renjie Pi, Weizhong Zhang, Renzhe Xu, Peng Cui, and Tong Zhang. Model agnostic sample reweighting for out-ofddistriuttonn learning. arXiv preprint orX19 2023.
15

---

A Related Work
Model-based Data Valuation A common approach for data valuation is the leave-one-out (LOO) score, i.e., the change in performance when a point removed from training, but this is computa- tionally expensive. Influence functions 33
approximate these scores using second-order derivatives, though they remain intractable for modern deep models. Recent advances have made influence functions scalable large models (4 g and others trace test loss the training trajectory to attribute influence 47 68). TRAK [46] takes a post-hoc approach, approximating the model as a kernel machine trace predictions back to training examples, while DAVINZ [71] uses neural tangent kernel approximations to predict influence at initialization. Other approaches include reinforcement learning to learn data values [76], prediction from noisy labels [3], and dynamic self- weightigg
mechanisms within loss functions [70].
Algorithm- based Data Valuation Unlike model-based methods that track influence during or after training, algorithm-based approaches define data value in terms of a specific learning algorithm and utility function. A major line relies on Shapley values [56], which uniquely satisfy fairness axioms such symmetry and null-value. Data Shapley [22] defines point's value as its average marginal contribution to model utility across all subsets, but exact computation requires training 2 models. Efficient approximations [28). closed-form solutions for k-NN [27] class-wise extensions (54), and distributional variants (21) have been proposed. Retraining multiple models can be avoided through gradient- and hessian based approximations [68] However, the reliability of shapley methods is sensitive to the utility function [69]. Other work relaxes the efficiency axiom to improve robust- ness 35 67. For bagging models, Kwon al. 36 show that out-of-bag estimates yield effective approximations.
aggooitii- Data Valuation Algorithm-agnoste valuation methods operate without knowl- edge of the learning algorithm. Just et al. [30] propose LAVA, which quantifies the contribution of point based on its contribution to the dataset distance. However, their method has several limitations: [29]; (ii) their approximations are not always aligned with the leave one out valuation and can be indeterministic [66] (see Figure 1b and (iii) the method
(i) it performs poorly in identifying label errors
computationally expensive (even approximate form, has O(n²) complexity)
15 The distance
must be computed over the entire dataset even when valuing single point (iv) memory complexity (n²)) To address memory bottlenecks in LAVA for large datasets, Kessler al. [32) propose
SAVA, a batching strategy for Wasserstein computation, though other challenges remain unresolved.
Dataset Valuation Another line of work focuses on valuing entire datasets rather than individual points, typically in settings with multiple data providers where fair compensation is desired 77 These methods use distance-based metrics such as mutual information (7). MMD (62), MMD 744, and volume 775 to assess utility. However, they do extend naturally to individual valuation due to their reliance on large datasets, inability to capture point-level interactions, and lack of influence estimation. Unlike these methods that compute distances between entire datasets. our approach quantifies individual datapoints' contributions to distributional distance through influence functions. This value not equal the distance between the individual point and the reference dataset.
B Proofs
B.1 Proof of Proposition
1
Proposition I The influence function for MMD as the distance metric is, up to additive and positive multiplicative constants, given by
IF FFMD (z; P,Q)
x)] Eermqlk(ae x))]
Proof. We first derive the influence function for MMD then apply the chain rule to obtain the result for MMD.
16

---

d (P, (1 c6G)
MMD (z; P.Q)
de
c=0
d
1p - ((1 20ll-E)11 E)12@ + =0(x)))
de
d de
[[|up pell ll + 20(up-
uQ: -Q -(x)) + + + 6(x))
-2uup pQ; pQ x(x))
2 (-(ur, (pr, x(x)) (ut, uQ)
2 (Exrr [k(a'' x)] E R [k(2' ; =)])
2 (E P.Q [k(2', x²)] + E x xx))))
Now, applying the chain rule:
IF MMD (z; P,Q) IF MMD= (z;; P.Q)
2MMD(P,QQ
Ignoring terms independent of obtain the simplified expression:
TF ("; P, P)Q)== F x]] Ev (k(('',=)))
(15)
B.2 Proof of Proposition 2
Proposition
2 Let Dimin and Dual be finite samples from distributions Q and P, respectively. for all subsets $ Dirain Araiainn ,x
MMD(D SU (raiinn)) DMD(D"d ,S) MMD(D SS (2)) (rinn)))) MMD(D
then (attain) IF (zttrann)
Proof. Consider = Dtrain \ {a:train ( ..train.. J. We have,
MMD(Dval S (2;rainn) MMD(Dva) S) MM S (xf (x)rain)) MMD
Adding MMD(D S) to both sides gives
MMD(D S (atrain)) MMD(Dva) S (aztain))
Squaring both sides,
2 (D U (a.tran)) = MMD3 (D S U (2.rrinn) )). MMD
Using the finite-sample estimator for MMD,
MMDD (D T)
Σ k.
k( ayaa)
Σ
k( , train)
n
|T|2
kIeT
Recal
h( o
trainy
nvl
LET where T S train or I SU (attainn. Therefore, substituting the finite-sample estimator into.
MMD 2 (D SU U (rtiinn)) }) MMD (D ,SU (x) (raaa )).
17

---

After canceling common terms and using [T] Atrain 1,
Thaal
Aerain
k(a%
2
train)
k(x frain Iz
traln)
Alyall 192
Mtrain
11,,
2
2
Atrain
k(x 2.
val )
k( 2 zerain)
vll
Atrrin
mmm
1=1,475
Therefore,
IF [x (2())
B.3 Proof and Example of Proposition
The core idea is that a threshold can be used to distinguish points where P locally dominates Q versus those where Q dominates P. based on their relative densities in a neighborhood around each point. For accurate local density estimation, the kernel bandwidth a (e.g., a Gaussian kernel) should be chosen sufficiently small. This result holds in the infinite sample regime where expectations are exact. the finite sample case, this separation may not hold strictly, as small values of a can induce high-variance estimates.
Intuition:
Proposition 3 Let P and Q be two probability distributions on X' S R For any E > 0 and r 0, there exists a Gaussian isotropic kernel & such that for (x) = Ex'n (M(x,,))) Ex'n xk(,,x))), the following holds
[F(x)= >0, 0, iF (137--Q04 Q(x)) 2 eV B(z,r), [F())) <0, (f((x') Q(x') S - Vx' E B(r,r),
where B(x, r) (z': || x|| <r).
Proof. Let
M(xxx'= (2T02) exp( - || 2'113 )
202
then
[F))) = (P(x') (((d))))(r,tt k(1, dx' JI'EB(z,r)
(P-Q)
(Px,))((zxx))
J''R
('B((r)
Since
- Q(x')= 2 for x' B(x,r) and IP- Q1 outside,
f(x) o/
k(xxx') [ JE'Enaa k(xxx') cA (1 A) (E+ 1)A-1, 1,
(a'EN(Z))))
where
A= / k(xxx') dr J'EE((..))
Note that,
J B(u.r)
Mx,xd)dx'
Pr (|| x=
[r) < < <=3)= PT([[[]] Pr( 2 2 r2).
Let W =
X7, E(W] no² and Var(W) 2no By Chebyshev's inequality, for
? no?
2no Pr(W 2 19) < (r2 - E(WI)2 (r - 2)
Var(W)
hence
2n 1
(r²
2)²
18

---

MMD
Wasserstein
Frequency
200
noisy
Frequency
200
noisy
clean
clean
100
100
0
0
0.055-0.000 0.000
0.025
-50
Influence
Influence
Figure 9: Influence distribution obtained from influence based on MMD (KAIROS) and Wasserstein
(LAVa)).
Choose V > 0² > 0 so that
2n (r² no 222
1
E+1
Then (c 1)4 11 0, implying IF(x:) > O.
The o" gives
(F(x)) < ((e + 1JA 1) <0 for the other case where P(z') - Q(x')()))))=)))))))))
-c. -
Thus, there exists Gaussian kernel with bandwidth a² that satisfies the desired separation.
Example 1 (Density Separation). We compare the influence distributions obtained from MMD and Wasserstein, computed on the CIFAR-10 sample data, to show the density separation property. As shown in Figure 9 MMD-based influence exhibits near-zero optimal threshold that separates the noisy and clean points almost perfectly. On the other hand, the Wasserstein based influence for noisy and clean data is entangled, and there is no threshold which cleanly separates the two.
B.4 Proof of Proposition
Proposition 4 The influence function for E-MCMD, up to additive and positive multiplicative constants, given by
IF (FF-MCMD((Z.u); = =1//f/x(=) -(u)ll
(16)
Proof. Consider the perturbed distribution Q: (1 - = c with corresponding marginal s)Qx + zdg. Since Qe y x) the perturbed conditional embedding point is given by
(x) <lv).
Therefore,
MCMDDp(4') |1 H
if #
MCM)
(17)
if =
R-MCMD(P, Q.) can be written as:
Ex=(1=))0x++6_ MMMD (X)
= =11-) (EE-Qxx (C) p,q,
= (1 - -) (Ex-ax (M)) + + (x) - d(y))
19

---

Therefore we have,
Bx-(1-e)xxtodd MMMD (X) - Ex~ox (MMM)).(X))
lim
6-+0-
(1- ( (1- - Ex~ex pg
lim
c-10-
(1) -0)) - ell
lim --
(x) mb)ll + Ex~qx [MCMD::(()]
Ignoring terms independent of æ and
IF IFE-MCMD((2. v): = -11xf1x(2)) -
B.5 Proof of Theorem [
We present the full statement of Theorem D bellow, followed by a detailed proof.
Theorem I (oundiigg transfer loss). Let (X, dx) and (Y, dy) be compact metric spaces, and let kx, by be universal kernels on N with RKHS Hx. Hy. Equip Z = X y with the tensor-product kernel whose RKHS is
H. = xx Hx
Let P. Q P(2) huve marginals Px. Px.Qx= ((X) and conditionals P(. Q(: | x)) E P(Y). Let
L: 2 + R be continuous loss function. Since H is endowed with universal kernel, H is dense in
C(Z). Let
L'EH { (1m ('r + sup || sup L' v'(x,)))lz, + I| L'
|||||
inf
zex vEy
Then
((r.))~QL(E, & B(ep)) w)] + ||-||- (MMDx Qx) + Ex~Qx [MCMDP.Q(t)]
Proof. Fix 0. Since H is dense in C(Z), choose L' H with || L'll 30 S.. Then
Eq [ EQ|L'I + Eq L - L'), Ep[L] = [L'] | + Ep L - L'],
[4[4] - Bp [4] = (Kq L'] - Ep(L() + (BQlL - L' - Ep(L L')).
Since P and probability measures,
EQ[L L' Er L7l < IL - L'lloc Se.
By the law of total expectation,
Eq ['] = Ex~qx E L' L'xx, v)]. Ep[L'] = Ex (a) L' (z, v)..
Add and subtract Ex-Q x E P to obtain
Bo [P[[T.] A B,
where
A = Ex~Qx E L' Ey- L'r, 1)).
B Ex-Qx E Pay 1/ T. w) Ex Bayw (24,
Define fz)(x) Ey-p m) (' v) E Mxx Then
B = En (x) PE
20

---

so
[B] & ll MMDx Qx) = ||'|| 2 MMDx((x..2)
For each æ, let (x,, :) E Hy. Then
Q(-1() (z, y) By-P((a) v)1 - 11, ,,,, MCMDP..(c))
hence
|A| < Rx [ MCMDPo(() || (3)) Bn-Qx (MMMDpQ(x)-
Combining the bounds for A, B, and the approximation term,
E & Ep[L] + I' 11x MMDx (xx, Qx) x) + Ex~qx (Z)] tE.
Taking the infimum over L' E H. yields the desired result.
C Influence Computation and Update for Streaming data
Algorithm
I
provides the detailed algorithm for computing the influence the online setting.
Algorithm 1 Online update for influence
Input: Current size n4, variables A(t), B(t), R( E IR" validation size rivat, batch size m, incoming batch data (Xueww E Rmadd yuow Rmxx)), balancing factor X.
Output: Updated ((t+1)
g((+1) g(t+)), R(++1), p(++1) E R
I: Rt+1 f- Th m
2: compute new E BRu Knewoold Rmxme K Knew,new Rm
// kernel sub-matrices // update kernel means for old points
3: io 1,...,,777 T4 do
4: A(t+1) - m ((m - 1) + Ez= Kiy
B B((+1) + B(())
6: (((+1) f- R((0)
7: end for
8: R.eww
|| Marin 1...., do
Straio
(1 1...., (.....,m) m)
I/ residuals for new points update kernel means for new points
9: for
10:
((41))
Knew,ol old + + K new 1)
ne+1
11: B(t+1) nmal
2jmr k(2) val xn
train)
12: R (1+1)
mitii < R.n.x
13: end for
14: for L..... 14+1 do
15: DI(++1] <- (1((+1)) - ((+11) + -1-
|| net influence
16: end for
D Additional Experiments and Detailed Settings
Detailed description of baselines and choice typer-parameters. We compare with the follow- ing baselines: LAVA 30 is model-agnostic method that uses the Wasserstein-1 dual potential as an influence proxy: DATAOOB (36) trains a bootstrap ensemble and assigns each point its average out-of-bag loss to capture its contribution; DVRL. [76] learns per-sample importance weights via reinforcement learning and uses those weights directly as data scores; and KNNSHAPLEY [27] computes exact Shapley values tion for efficiency. To avoid the prohibitive runtime of DATAOOB without sacrificing accuracy, we decrease its number of bootstraps from 1000 to 100. All baselines run with their default settings in OpenData Val 229
nearest-neights neighhor proxy model, exploiting its closed-fomm solu-
Following 19 23 we adopt the "median heuristic" to set the kernel bandwidth to the median of pair-wise distances. Given the large number samples, we estimate the median on 10000 sampled pairs. In practice, as shown in experiments, sampled bandwidths work well across various datasets and scenarios. The balancing factor determined by aligning the scale of the two components net influence (Equation (22), when both feature and label noise exist.
21

---

Comparison with Sample Reweighting Methods We compare KAIROS with MAPLE [78), model-based sample reweighting method. Unlike KAIROS, MAPLE does not compute data values but learns reweighting functions for specific training objectives. While applicable to corrupted label detection, KATROS offers broader applicability across tasks such as detecting harmful fine-tuning data and data poisoning attacks. MAPLE is not model-agnostic and inherits the limitations of model-based approaches discussed in Section A
We evaluate both methods on CIFAR-IO feature noise and label noise detection. For MAPLE, we use target labels as group labels since no explicit group labels are available. We report the AUC of the fraction of covered corrupted data versus the fraction of inspected data.
Table 3: AUC for CIFAR-10 feature noise and label noise detection tasks.
Method
AUC Feature Noise
AUC Label Noise
Data OOB
0.727
0.784
KNN Shapley
0.723
0.751
LAVA
0.837
0.529
KAIROS (Gaussian)
0,857
0.791
MAPLE
0.347
0.828
Maximum possible
0,900
0,900
MAPLE performs well label noise detection but poorly feature noise detection. While reweighting methods can be effective for specific corruption types like label noise, KAIROS provides consistent performance across diverse tasks including feature noise, label noise, adversarial attacks, and harmful fine-tuning detection.
Selection Bias Detection We evaluate KAIROS on selection bias where subgroups are under- represented in training data. We use the ACS Income dataset from WhyShift |? J. which predicts income from demographic attributes. This dataset exhibits geographic bias where some states like Puerto Rico (PR) under-represented while others like California (CA) are rerreeepe.eted. Liu et al. [40] show that models trained predominantly on CA data fail to generalize to PR.
We simulate selection bias by creating training set with 80% CA and 20% PR samples (1000 total) and balanced validation set with 50% CA and 50% PR samples (300 total). We evaluate how well KAIROS and baselines identify points from the under-represented group (PR). Data valuation methods should assign high influence to under-represented samples. We report the AUC the fraction of under-representte samples detected versus the top & fraction of data chosen.
Table 4: Selection bias detection performance on ACS Income dataset.
Method
AUC
Data OOB
0.326
KNN Shapley
0.494
LAVA
0.546
KAIROS
0.855
Maximum possible (theoretical)
0.900
The results show that KAIROS considerably outperforms both model-agnostic and model-based baselines in identifying under-represented samples. For data redundancy, KAIROS rankings remain stable when datasets contain duplicates. We verified this by duplicating the training dataset up to 10 times and confirming that relative rankings are preserved.
Comparison with additional Model-based Methods We compare KATROS with TRAK [46 and LOGRA 88. recent model-based attribution methods for large models. TRAK trains multiple models while LOGRA optimizes influence functions for efficiency. Both require model training, while KAIROS is model-agnostic and training-free.
22

---

We evaluate on CIFAR-10 10 feature noise and label noise detection using the experimental setup from Section inspected data.
4
We report the AUC of the fraction of covered corrupted data versus the fraction of
Table 5: AUC comparison on CIFAR-10 corruption detection tasks.
Method
Feature Noise
Label Noise
Data OOB
0.727
0.784
KNN Shapley
0,723
0.751
LAVA
0.837
0.529
KAIROS
0,857
0.791
TRAK
0.638
0.743
LOGRA
0.565
0.663
Maximum possible (theoretical)
0.900
0.900
KAIROS achieves the highest AUC on both tasks, outperforming TRAK and LOGRA particula
Effect of kernel choice order to study the effect of kernel choice, we conduct additional experiments using polynomial kernel of degree 2 on the feature-noise CIFAR-10 task. We report the AUC of the fraction of covered corrupted data vs the fraction of inspected data.
Table 6: AUC for detecting feature noise in CIFAR-10
Method
AUC
Data OOB
0.727
KNN Shapley
0.723
LAVA
0.837
KAIROS (Gaussian)
0,857
KAIROS (Polynomial)
0.856
Maximum possible (theoretical)
0.900
The results show that KAIROS with polynomial kernels achieves nearly identical performance to Gaussian kernels (0.856 vs 0.857) and outperforms all baselines.
Estimating P(y(æ) from Validation Data The validation set is typically small (300 samples in our experiments), yielding noisy P(y|z) estimates. However, this provides sufficient signal to identify corrupted training points. After removing these points, we train on the cleaned, larger training set to obtain better P(y(z) estimates and higher accuracy.
We demonstrate this on CIFAR-I0 with feature noise. A model trained only on validation data (300 samples) achieves 72% test accuracy. Training on the full training set after removing the bottom 20% identified by KAIROS achieves 88.5% test accuracy. Even selecting just the top 300 training samples (matching validation size) gives 87.3% accuracy, demonstrating that KAIROS improves sample quality beyond what the small validation set provides. This shows that while the validation classifier has limited accuracy, it successfully identifies low-quality training data, enabling substantial improvements when training on the cleaned set.
Robustness to noise in the validation set KAIROS, similar to LAVA, assumes access to a small sample from the (clean) reference distribution. Further, model-based methods value data based on performance on the validation set which is generally assumed to be representative of the test distribution.
However, in practice, these small samples of the reference set may have some noise. We test the robustness of KAIROS and other baselines experiments by adding noise to the validation set for the feature noise CIFAR- 10 task. We consider two settings where we randomly corrupt 3% and 7% of the validation samples respectively.
noisy reference set. We conduct additional
Dataset Licensing Information.
The license terms for each dataset used in this work follows:
23

---

Table 7: Robustmess to Noisy Reference Set AUC Performance
Method
No Validation Noise
3% Validation Noise
7% Validation Noise
Data OOB
0.727
0.711
0.711
KNN Shapley
0.723
0.705
0.705
LAVA
0.837
0.635
0.601
KAIROS
0.857
0.857
0,856
CIFAR-10 34
released under the MIT License.
https://ww..c..torontoooooo
cifar: htm
STL-10 [0]: no explicit license is provided. http://atta.sssssfffrrr..ee http:///ccaaoorrrddarrrrrttateaa
IMDB
subject to IMDb's non-commercial terms use. https://dataastts.inddb imdwssss com/
AG News 77
provided for research and non-commercial use. http::/wwee
di http://:w...mpp......... it/
gulli/AG_ of news articles.hi html
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We state our contributions and scope in Section] |
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
25

---

Justification: We provide intuitions and examples in the main body of the paper. We include the complete proofs the appendix, and refer them in the main body of the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Descriptions in the main body of the paper (Section 3.4 are sufficient for reproducing our approach and experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
26

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification: We share the anonymized Github repository containing the data and code in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https//ccppblccc/uuuddeeeoo Coaee//ddeeeebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We describe the experiment settings in detail in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We repeat experiments with different random seeds and show the mean and standard deviation in the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
27

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe the sufficient information on the computer resources in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttpp/://eerppss..cpppbtt//t/t
Answer: [Yes]
Justification: We reviewed the NeurIPS Code of Ethics and make sure the paper conforms with it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential broader impacts of this work in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
28

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper has such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We credit all existing assets used in the paper, and include the license informa- tion (Sections 4 and D)
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
29

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide documentations for the released anonymized code (Section 4
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
30

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this work does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
31


Reviews:
[{"review_id": "3lfCp2zTXV", "reviewer": "Reviewer_g4SV", "summary": "The paper proposes a framework for assessing the value of individual training examples without relying on a specific model. It introduces a closed-form influence function based on Maximum Mean Discrepancy, which approximates leave-one-out utility and avoids the computational overhead of retraining models. KAIROS offers theoretical guarantees such as symmetry and density separation, and supports efficient online updates. Empirical evaluations show that KAIROS performs competitively across various tasks, including noise detection and data pruning, and achieves notable runtime improvements in the online setting.", "questions": "None.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n- The paper introduces a novel model-agnostic data point valuation method that addresses a major weakness of the previous LAVA approach: it introduces a bias of order O(dv log(1/v)) that may cause inaccurate ranking.\n- The method leverages Maximum Mean Discrepancy and the kernel trick to derive a closed-form influence score, providing both conceptual clarity and practical simplicity.\n- The approach is supported by theoretical guarantees. It is proved that the MMD provides an upper bound of the validation loss, and the method guarantees fairness and density separation.\n- The effectiveness of the approach is supported by experiments as well.\n- In the online setting, the method achieves significantly higher computational efficiency due to its closed-form structure.\n\n**Weaknesses:**\n- The performance of KAIROS might depend on kernel choice. \n- Although KAIROS is highly efficient in the online setting, its computational complexity in the offline setting is comparable to that of LAVA.", "was_revised": true, "final_justification": "I maintain my original evaluation.", "rebuttal": "We thank the reviewer for their thorough review and strong support of our work. We appreciate your recognition of our theoretical analysis, empirical validation, and computational efficiency.\n\n**W1: The performance of KAIROS might depend on kernel choice.**\n\nThank you for this question. Any MMD-based method depends on kernel choice, and we acknowledge this consideration for practitioners. To address concerns of sensitivity to the kernel, we conducted additional experiments using a polynomial kernel of degree 2 on the feature-noise CIFAR-10 task. We report the AUC of the fraction of covered corrupted data vs the fraction of inspected data (we will include full plots in the final paper, but images are not allowed in rebuttals):\n\n**Table 1: AUC for detecting feature noise in CIFAR-10**\n\n| Method | AUC |\n|--------|-----|\n| Data OOB | 0.727 |\n| KNN Shapley | 0.723 |\n| LAVA | 0.837 |\n| KAIROS (Gaussian) | **0.857** |\n| KAIROS (Polynomial) | 0.856 |\n| Maximum possible (theoretical) | 0.900 |\n\nThe results show that KAIROS with polynomial kernels achieves nearly identical performance to Gaussian kernels (0.856 vs 0.857) and outperforms all baselines. Further, as mentioned in Section 5, adopting learned kernels that adapt to specific tasks is a promising direction for future research.\n\n**W2: Although KAIROS is highly efficient in the online setting, its computational complexity in the offline setting is comparable to that of LAVA.**\n\nThank you for recognizing KAIROS's superior efficiency in online settings, where we achieve O(BN) complexity compared to LAVA's O(N²) (where B is batch size and N is dataset size). Regarding the offline setting, while both methods have O(N²) theoretical complexity, there are important distinctions: LAVA achieves O(N²) only through Sinkhorn approximation, which introduces bias and influence that may be non-deterministic (Section 3.1). In contrast, KAIROS provides exact MMD influence computation without approximation while maintaining the same complexity. *Furthermore, our empirical results (Figure 7) demonstrate that KAIROS achieves faster actual runtime than LAVA even in the offline setting.*"}, {"review_id": "EDxhct1XJm", "reviewer": "Reviewer_UAd2", "summary": "The paper proposes a model-agnostic method for data valuation MMD-based distribution influence score. The method is efficient due to the closed-form expression to calculate data value based on the kernel trick and is compatible with streaming batch implementation.", "questions": "* How does this method compare against more recent data attribution methods for large models like Trak, Logra? \n* How robust is the method to distribution shift between training and validation set? Distributional measures may have issues in this scenario where there is no clean reference available. How robust is the method  to a noisy or biased reference set?\n* How to choose the kernel and bandwidth in practice for different datasets? How to balance $\\lambda$? Is the median heuristic always a sensible choice?\n* How would the method perform on more nuanced issues like selection bias, concept drift, or data redundancy?\n* How does this work differ from and related to \"Data Distribution Valuation\", which also uses MMD for data valuation? https://arxiv.org/abs/2410.04386", "limitations": "The experiments could be improved. There needs to be a comparison with more recent and competitive baselines such as https://arxiv.org/abs/2406.01130. Larger models and more realistic datasets need to be evaluated to show the robustness of the proposed method.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros\n* The paper shows the deficiency of other methods for approximating LOO is a valuable contribution. \n* The method seems computationally efficient and streaming compatible. \n\nCons\n* Experimental performance is good, although the evaluation is fairly basic and could be improved.\n* The method still depends on training a classifier for label conditional MMD so is not technically model-free.\n* The method is limited to classification tasks and assumes a clean reference dataset is available to compute MMD", "was_revised": true, "final_justification": "I appreciate the authors response. Based on the additional experimental results and comments, I raise my score to accept.", "rebuttal": "We thank the reviewer for their constructive and insightful feedback, and appreciate their positive evaluation of our theoretical analysis and computational efficiency.\n\n**W1 & Lim1: Experimental performance is good, although the evaluation is fairly basic and could be improved. There needs to be a comparison with more recent and competitive baselines such as SAVA.**\n\nThank you for the feedback! SAVA was primarily built to address the memory consumption issue of LAVA, making it scalable to larger datasets. Therefore, we compare KAIROS with SAVA on ImageNet, which has 1.28M training samples and 1000 classes. We use a ResNet50 encoder to extract feature representations from ImageNet, and we use an A100 GPU with 40GB VRAM for computing data values. We report the AUC of the fraction of covered corrupted data vs the fraction of inspected data (we will include full plots in the final paper, but images are not allowed in rebuttals):\n\n**Table 2: KAIROS and SAVA on Imagenet**\n\nFeature Noise Detection\n| Method | Runtime | AUC |\n|--------|-----|-----|\n| KAIROS | **7 min 56 s** | **0.869** |\n| SAVA | 1hr 58 min | 0.817 |\n\nLabel Noise Detection\n| Method | Runtime | AUC |\n|--------|-----|-----|\n| KAIROS | **7 min 52 s** | **0.861** |\n| SAVA | 1hr 58 min | 0.484 |\n\nOur results show KAIROS outperforms SAVA in both efficiency and effectiveness. Since SAVA is built based on LAVA, similar to the results in Figures 2 and 3, KAIROS outperforms SAVA in both experiment settings, especially in label noise detection.\n\nThe efficiency improvement stems from KAIROS's closed-form solution, making it more suitable for batch-based GPU acceleration compared to Sinkhorn computations. In contrast, SAVA (and LAVA) needs to compute pairwise conditional Wasserstein distances between P(x|y) for every pair of classes, requiring $\\frac{1000\\times 999}{2}$ computations, thus leading to a high runtime. Specifically, the computation of pairwise conditional Wasserstein distances takes more than 80% of the overall runtime.\n\n**W2: The method still depends on training a classifier for label conditional MMD so is not technically model-free.**\n\nOur method is model-agnostic with respect to the training data being valued. We never train models on the training set. We only train a single classifier on the validation set (3% of training size in our experiments) to estimate P(y|x). This preserves the core benefits of model-agnostic approaches:  1) avoiding multiple model retrainings (unlike Data Shapley or Data-OOB) and 2) data preprocessing before training the downstream model. Further, this makes our approach significantly faster than model-based baselines (Figure 7, page 9).\n\n**W3a: The method is limited to classification tasks.**\n\nThe conditional component (E-MCMD) currently targets classification, but our framework applies broadly to many other tasks which do not have a target (y), such as image poisoning attacks and harmful LLM fine-tuning data detection (Figure 4). The marginal MMD component works for any data type. We acknowledge the classification limitation for labeled tasks and identify regression extension as future work (Section 5).\n\n**W3b & Q2a: This method assumes a clean reference set. Distributional measures may have issues in this scenario where there is no clean reference available. How robust is the method to a noisy or biased reference set?**\n\nThank you for this important question. For data valuation, one needs to measure utility with respect to something. In our case, we assume access to a small sample from the target distribution, which we call the validation (reference) dataset. This assumption is shared with LAVA. Further, model-based methods also value data based on performance on the validation set which is generally assumed to be representative of the test distribution.\n\nWe test the robustness of KAIROS and other baselines to a noisy reference set. We conduct additional experiments by adding noise to the validation set for the feature noise CIFAR-10 task. We consider two settings where we randomly corrupt 3% and 7% of the validation samples respectively.\n\n**Table 3: Robustness to Noisy Reference Set - AUC Performance**\n| Method | No Validation Noise | 3% Validation Noise | 7% Validation Noise |\n|--------|---------------------|---------------------|---------------------|\n| Data OOB | 0.727 | 0.711 | 0.711 |\n| KNN Shapley | 0.723 | 0.705 | 0.705 |\n| LAVA | 0.837 | 0.635 | 0.601 |\n| KAIROS | **0.857** | **0.857** | **0.856** |\n\nOur results show that KAIROS is robust and maintains original performance in the presence of noisy reference sets. Moreover, KAIROS outperforms both model-based and model-agnostic baselines across different noise levels.\n\n**Q1: How does this method compare against more recent data attribution methods for large models like Trak, Logra?**\n\nThank you for this question. TRAK and LOGRA are model-based attribution methods that require training models (TRAK often trains multiple models, while LOGRA provides optimizations to make influence functions more efficient for large models). In contrast, KAIROS is model-agnostic and requires no training.\nWe test KAIROS against these methods on feature-noise detection and label-noise detection tasks using CIFAR-10 with the same experimental setup described in the paper. Since we cannot share plots in the rebuttal, we report the AUC of the curves below. We will include the full plots in the final version.\n\n**Table 4: Brief Description of Experiments - AUC Performance Comparison**\n| Method | Feature Noise | Label Noise |\n|--------|---------------|-------------|\n| Data OOB | 0.727 | 0.784 |\n| KNN Shapley | 0.723 | 0.751 |\n| LAVA | 0.837 | 0.529 |\n| KAIROS | **0.857** | **0.791** |\n| TRAK | 0.638 | 0.743 |\n| LOGRA | 0.565 | 0.663 |\n\nFor both experiments, KAIROS achieves the highest AUC compared to baselines, including TRAK and LOGRA, particularly in the feature noise detection setting.\n\n**Q2b: How robust is the method to distribution shift between training and validation set?.**\n\nThank you for this question. KAIROS is designed for distribution shift between training and validation distributions. The method measures how training points from noisy distribution Q contribute to distance from clean validation distribution P. All our experiments involve such shifts where training sets and validation sets come from different distributions.\n\n**Q3: How to choose the kernel and bandwidth in practice for different datasets? How to balance λ? Is the median heuristic always a sensible choice?**\n\nWe agree that kernel and bandwidth selection is an important practical consideration. The median heuristic has proven effective across all our experimental tasks and is widely adopted in MMD literature [2-4]. We also experiment with a polynomial kernel of degree 2, which demonstrates excellent performance. Please refer to the response to W1 of reviewer g4SV for detailed experimental results. For the balancing parameter λ, we set it to align the variance of the feature and label components in Equation 12 (see Appendix D). While the median heuristic works well in practice, we acknowledge that adaptive kernel selection could be beneficial for some applications and identify it as future work (Section 5). We will add results comparing different kernel choices to the revised paper.\n\n**Q4: How would the method perform on more nuanced issues like selection bias, concept drift, or data redundancy?**\n\nThank you for suggesting this direction. We explore selection bias where subgroups are under-represented in training data. We use the ACS Income dataset from WhyShift [1], which predicts income from demographic attributes. This dataset exhibits geographic bias where some states like Puerto Rico (PR) are severely under-represented while others like California (CA) are over-represented. Liu et al. [1] also show that models trained predominantly on CA data fail to generalize to PR.\nWe simulate selection bias by creating a training set with 80% CA and 20% PR samples (1000 total), while our reference validation set contains balanced 50% CA and 50% PR samples (300 total). We evaluate how well KAIROS and baselines identify points from the under-represented group (PR). Data attribution methods should assign high influence to under-represented samples. We report the AUC of the fraction of under-represented samples detected vs top k fraction of data chosen.\n\n**Table 5: Selection Bias Detection Performance**\n\n| Method | AUC |\n|--------|-----|\n| LAVA | 0.546 |\n| KNN Shapley | 0.494 |\n| Data OOB | 0.326 |\n| KAIROS | **0.855** |\n\nThe results show that KAIROS considerably outperforms both model-agnostic and model-based baselines. We will add these results to the final version of our paper.\n\nFor data redundancy, KAIROS rankings remain stable when datasets contain duplicates. We empirically verified this by duplicating the training dataset up to 10 times and confirming that relative rankings are preserved, demonstrating robustness to redundant data.\n\n**Q5: How does this work differ from and related to Data Distribution Valuation?**\n\nData Distribution Valuation values entire distributions (or datasets) by computing MMD between each dataset and a reference (or uniform average of all datasets). Our work addresses a fundamentally different problem: quantifying individual datapoints' contributions to the MMD between training and validation distributions via influence functions. This measures how removing a single point changes the distributional distance, not the MMD between that point and the reference distribution. We will include a detailed comparison in the appendix of the final version.\n\n[1] Liu et al. \"On the need for a language describing distribution shifts: Illustrations on tabular datasets.\" NeurIPS 2023\n\n[2] Shekhar et al. \"A permutation-free kernel two-sample test.\" NeurIPS 2022\n\n[3] Gretton et al. \"A kernel two-sample test.\" JMLR 2012\n\n[4] Doran et al. \"A Permutation-Based Kernel Conditional Independence Test.\" UAI 2014"}, {"review_id": "j7sqBGds5r", "reviewer": "Reviewer_od2D", "summary": "The authors consider the problem of model-agnostic data valuation via distributional influence function which are able to approximate the leave one out error with  error inversely proportional to squared number of train samples. As against the previous work on Kl-divergence or integral prob metrics like Wasserstein-1 distance which doesn't admit a tractable influence function, the authors consider MMD which they show admits a closed form influence function bypassing calculating the distributional distance explicitly. Besides they also sow that the influence function is symmetric for fair rankings and also the density separation property which means there exist a single global threshold which can separate the validation distribution domination region from the train one. They further adapt it for the corruptions in the label space by also considering maximum conditional mean discrepancy and consider the net discrepancy as the convex sum of MMD for input space and MCMD for labels, thereby the resulting influence function also becomes a convex sum, which upper bounds the train val loss gap. They also show a theoretical link between this net discrepancy and the model performance as a measure of generalizationl. They also show its efficient to use in the online setting since the order of complexity of calculating the influence function at any time as number of samples seen till now times the batch size. The authors further verify its effectiveness empirically on the small-scale classification datasets (including both image and text datasets).", "questions": "Please see the weaknesses section.", "limitations": "Yes, they have mentioned in Section 5.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1.) The contribution of using the MMD metric and thereby coming up with a closed form influence function which follows desired properties and is efficient to implement is nice since the prev works on KL or the intergal prob metrics like wasserstein-1 were all using some approximations and for which the critic shift term is ignored in the prior work. \n2.) The analysis is easily adaptable to label-space corruptions which is also interesting and useful. \n3.) The connection of the net-discrepancy to generalization is also an interesting insight.\n3.) The empirical results show that their method can effectively rank noise samples and is better than the other baselines for both feature noise and label space corruptions, especially when the percentage of inspected data is low. Their method also works under the adversarial attacks as shown in figure 4.\n4.) The analysis on the test accuracy when lowest/most valuable points are reomoved on the STL/IMDB datasets also advocates for the effectiveness of their method. \n\nWeakness:\n1.) It would have been better if the authors have covered a bit large-scale datasets (like ImageNet).\n2.) Also, there are works like FSR [2] which doesn't require clean reference set although they dynamically update it based on the model. But it might be interesting to see a comparison since it was also proposed for the corrupted label setup (although I am unsure this might be a fair comparison or not, but just wanted to mention comparison with methods on these lines as other model agnostic sample reweighting methods like MAPLE [1].\n\n[1] Model Agnostic Sample Reweighting for Out-of-Distribution Learning. ICML 2022.\n[2] Learning Fast Sample Re-weighting Without Reward Data. CVPR 2021.", "was_revised": true, "final_justification": "I would like to maintain my current rating. Although the authors have done a good job on comparing with MAPLE, it would have been better if the effectiveness of their method can be demonstrated on a bit larger datasets to actually understand the performance difference where they could have tried finetuning stronger pretraining models.", "rebuttal": "We thank the reviewer for their valuable review and positive assessment of our contributions. We are glad the reviewer appreciated the novelty of our closed-form MMD-based influence function, its theoretical properties, and empirical effectiveness, particularly under noisy and adversarial conditions.\n\n**W1: It would have been better if the authors have covered a bit large-scale datasets**\n\nThank you for your suggestion! The current baselines in the paper are challenging to run on larger datasets, due to runtime (model-based methods) or memory consumption issues (LAVA). However, we conduct additional experiments comparing KAIROS with SAVA (a scalable version of LAVA) on the ImageNet dataset, which contains 1.28M training examples and 1000 classes. We use a ResNet50 encoder to extract feature representations from ImageNet, and we use an A100 GPU with 40GB VRAM for computing data values.\n\nOur results demonstrate that KAIROS achieves better performance and efficiency on this large-scale dataset. In particular, when facing a large number of classes, the pair-wise conditional Wasserstein distance computation in SAVA becomes expensive, leading to a 15x runtime compared to KAIROS. Please refer to our response to W1 & Lim1 of Reviewer UAd2 for the detailed results and analysis.\n\n**W2: On differences and comparison on sample reweighting methods like FSR and MAPLE**\n\nThank you for suggesting this comparison. FSR and MAPLE are model-based sample reweighting methods that differ fundamentally from KAIROS. Unlike KAIROS, they do not compute data values but rather learn reweighting functions for specific training objectives. While they can be applied to corrupted label detection as you mentioned, KAIROS offers broader applicability across tasks such as detecting harmful fine-tuning data and data poisoning attacks. These methods are also not model-agnostic and inherit the limitations of model-based approaches that we discussed earlier.\nTo provide a comprehensive evaluation, we conducted experiments comparing KAIROS with MAPLE on CIFAR-10 feature and label noise detection tasks. We use the target labels as group labels for MAPLE since no explicit group labels are available for our datasets. We report the AUC of the fraction of covered corrupted data vs the fraction of inspected data (we will include full plots in the final paper, but images are not allowed in rebuttals):\n\n**Table 6: AUC for CIFAR-10 feature noise and label noise tasks**\n\n| Method | AUC Feature Noise | AUC Label Noise |\n|--------|-----|-----|\n| Data OOB | 0.727 | 0.784 |\n| KNN Shapley | 0.723 | 0.751 |\n| LAVA | 0.837 | 0.529 |\n| KAIROS (Gaussian) | **0.857** | 0.791 |\n| MAPLE | 0.347 | **0.828** |\n| Maximum possible (theoretical) | 0.900 | 0.900 |\n\nThe results agree with the reviewer’s intuition. MAPLE performs the best on label noise detection but has very poor performance on feature noise detection. This illustrates that while reweighting methods can be effective for specific corruption types like label noise, KAIROS provides consistent performance across diverse tasks including feature noise, label noise, adversarial attacks, and harmful fine-tuning detection. We will include this comparison in the revised paper."}, {"review_id": "6pSE8AFVhO", "reviewer": "Reviewer_cX5q", "summary": "This paper tackles the problem of data valuation, which, in the context of this paper, is the problem of determining how influential a datapoint is to the performance of a ML model. This paper takes a \"model-agnostic\" approach, and does not consider any specific model. Instead, the paper measures the distance of the training data $Q$ to an idealized, clean test distribution $P$ using an integral probability metric (IPM). The intuition is that if the IPM is smaller, then *any* model will do better on test data. Specifically, the paper computes an influence function (IF): the influence of the point $x$ is the derivative evaluated at $\\epsilon=0$:\n$$\n\\frac{d}{d\\epsilon} IPM((1-\\epsilon)Q + \\epsilon\\delta_x, P)\n$$\nPrevious work (LAVA) has taken this approach as well. But the paper notes that these approaches use IPMs that are ill-suited for the task in that (1) they require approximations to be made to the IPM computation, and (2) the IPM computation is not unique. The paper recognizes that if the maximum mean discrepancy (MMD) is used as the IPM, neither of these problems apply. The paper shows how the IF of the MMD can be computed efficiently as more and more data is added to the test set, which is not true of previous work in this space. The paper also provides separate IF calculations for the un-labeled setting (i.e., the data are just covariates $x$) and the labeled setting (the data are covariates $x$ and labels $y$). The paper runs experiments to see if the proposed method can discover corrupted data and identify points that are/aren't valuable for model accuracy. In most cases, the proposed method outperforms existing methods, and where it does not outperform other methods, its accuracy is very close.", "questions": "1. Why is the algorithm practical in the labeled setting? In particular, why would someone use this algorithm if they already have access to (an estimate of) $P(y|x)$?\n2. Why does the labeled setting not seem to depend on $Q$? Or is the dependence a little hidden?\n3. Why do we want model-agnostic data valuation?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "# Strengths\n\n- The paper is generally well-written\n- The overall idea of this paper is very similar to previous work. But, by providing careful analysis of the mathematical details behind previous work, the paper is able to recognize where improvements can be made. I think this is a great type of contribution to the field.\n- The idea of doing model-independent data valuation is fairly new, and I think this paper makes a good contribution to it, and thus derives an algorithm with some potentially interesting properties in the space of data valuation\n- In an experiment (Fig 8), the proposed method is much faster than similar competitors as the number of times data is added grows large. In fact, the experiment is so compelling that it made me realize I wasn't following how LAVA (previous work) worked! The paper mostly summarizes the main insight of KAIROS over LAVA as not needing particular approximations due to their choice of IPM. But this figure makes it seem like there are additional insights that allow KAIROS to run in constant time. I would really highlight the compute time differences as well.\n- This is minor, but the paper restates its theoretical results in the appendix before proving them -- this makes reading and checking them **so much** easier.\n\n# Weaknesses\n\nOverall I think this is a pretty solid paper. My two main issues with it are its overall motivation and the practicality of the labeled side of the algorithm. I see the other issues raised below as less important.\n\n**Overall motivation**\n\nThe introduction argues that model-agnostic data valuation is really important. I didn't follow the arguments here. The paper cites a growing legal need to have auditable algorithms. I agree with this claim. It then seems to immediately conclude that \"[to bridge] the gap between commercial pressures and regulatory demands has spurred interest in new classes of *model-agnostic* valuation techniques\". I wasn't following this logic. Why do we need model-agnostic techniques? It seems like legal or commercial data purchasing decisions are going to be made with a particular model (e.g., I believe the OpenAI v. New York Times legal case cited earlier in the introduction is about a specific model -- ChatGPT). So why do we need model-agnostic tools here? I think some more time should be dedicated to this argument. Without a clearer rationale, I still think the paper has some academic interest as a technical improvement over existing work. But I don't see a clear impact of the paper.\n\nA smaller point about motivation comes from Figure 1. Fig 1 notes that previous work (LAVA) seeks to approximate the exact leave-one-out (LOO) difference under the Wasserstein-1 metric when datapoints are dropped out. KAIROS seeks to estimate the exact LOO in terms of the maximum mean discrepancy (MMD). Fig 1 shows that KAIROS does a much better job approximating the MMD LOO differences than LAVA does approximating the Wasserstein-1 LOO differences. Why is this a sign that KAIROS is doing a good thing? Maybe we actually wanted to approximate the Wasserstein-1 LOO differences, and LAVA is as close as we can get to doing that. Here's a (kind of facetious) version of this: I have a fabulous new algorithm that's faster than KAIROS *and* does a better job of approximating its distance. The algorithm is \"return the number 5,\" and the distance that it approximates is \"5\".\n\n**Practicality of algorithm**\n\nIt's not clear to me that the label-dependent version of the algorithm (E-MCMD, where we observe not only covariates $x$ but also labels $y$) is practically viable. I have two concerns:\n\n1. In practice, when applying to labeled data $(x,y)$, the method requires an estimate of the true distribution $P(y\\mid x)$. But isn’t the whole reason I want to acquire more data in a supervised setting (i.e., where we have labels $y$) is because we **can’t** estimate $P(y|x)$ very well?  So what’s the practical value of this method? This holds double in settings where we’re trying to use IF’s to detect corrupted data — if one can fit a good model for $P$ with the corrupted labels, why bother trying to clean the labels?\n2. The equation given for E-MCMD in the generic case (Eq 8) and then specialized to classifiers (Eqs 9 and 10) don't seem to depend on the distribution of the training data $Q$. How can it be that this method gives reasonable estimates without a dependence on $Q$? I would imagine that if $Q=P$, we should get very different estimates of data value than if $Q$ is wildly different from $P$.\n\nTo be clear, I don't see any such issues with the label-free version of the algorithm.\n\n\n\n**Theoretical claims**\n\nA few somewhat minor issues about the strength of the theoretical claims:\n1. \"[theorem 1 shows that] pruning or down-weighting points with large influence scores provably tightens an out-of-distribution error bound\" -- I don't agree with this summary of Thm 1. Thm 1 shows that decreasing the MMD and E-MCMD decreases an out-of-distribution error bound. But this isn't the same thing as removing high influence points. A high influence point means that, if we think of attaching a weight $\\epsilon$ to each datapoint, there is *some* decreased $\\epsilon'$ such that using weight $\\epsilon'$ would decrease MMD / E-MCMD. Pruning a datapoint is equivalent to taking $\\epsilon' = 0$. But these aren't the same thing.\n2. \"Symmetry [from Proposition 2] ensures that, for finite samples, points making the same marginal contribution to the MMD receive identical influence, yielding fair rankings\" -- this seems to imply that Prop 2 shows KAIROS will give equal scores to datapoints that contribute the same amount. I don't think this is what Prop 2 shows. Prop 2 shows that the finite sample estimate of the IFs correctly ranks finite sample contributions to the *finite sample estimate* of the MMD. Why should a practitioner care that the finite sample estimate $\\widehat{MMD}$ will be correctly reflected by IF? $\\widehat{MMD}$ is noisy and could be completely erroneous; I would argue that a practitioner would only at all care about decreasing the actual MMD. And Prop 2 says nothing about IFs preserving the order of the actual MMD.\n\n**Unclear notation**\n- $n$ and $N$ are used throughout the introduction and abstract without definition.\n- I don't think $k$ was ever stated to be the kernel of anything; it's just said it's a \"kernel\"\n- \"we use IF() to denote the rescaled version of [influence functions]\" but this scaling wasn't defined -- most of these are kind of obvious, but I actually don't know what this scaling is supposed to be.\n\n\n**References**\n\nThere were a few places where I thought the references could have been better done:\n\n1. Line 105 cites the entirey of a textbook for a result -- please cite a specific result / page / theorem. I tried looking briefly through chapter 20 (source of the only mention of influence function in the index) because I wasn't aware of this general of a result. I couldn't find anything that jumped out.\n2. A key result that makes the issues with LAVA clear is just referenced as [57], which is an entire book on optimal transport. This doesn't help readers understand where this result came from, unless they want to read the entire textbook!\n3. The whole paragraph from lines 120-126 needs more clear references for some of its claims. E.g., where does the fact that the $\\alpha > 1$ decay doesn't hold for Wasserstein-1 come from? I think this is a really key paragraph that has most of the theoretical insights driving the paper, so I really think it's important to get this part right.", "was_revised": true, "final_justification": "After reading the replies and the other reviews, I still think this is a strong paper that will do well at NeurIPS; I've kept my score as a 5 and have more confidence that this is a well-researched and interesting paper!\n\nI specifically appreciate the proposal to add an experiment showing exactly how a noisy $P(y|x)$ is good enough for KAIROS to do a good job; I think this will be a good addition to the paper. I also think the proposed rewordings and clarifications will strengthen the paper as well.\n\nBest,\ncX5q", "rebuttal": "We thank the reviewer for their detailed review, insightful comments, and positive evaluation of our paper. We appreciate that the reviewer recognized our novel mathematical analysis, computational efficiency and empirical results.\n\n**W1 & Q3: Why do we need model-agnostic techniques? Why would legal and commercial decisions be made with respect to model-agnostic techniques?**\n\nThank you for this question. We clarify the commercial and legal motivations for model-agnostic data valuation.\n\n**Commercial incentives**: When training large models, two important reasons why data valuation would be important:\n - Large models are often trained on noisy web-crawled data containing duplicates [10] and data poisoning attacks [11]. Therefore, it is essential that the data is cleaned before training.\n- At the same time, companies also pay substantial costs for high-quality datasets (OpenAI-Time Magazine, Shutterstock deals [1][2]). In this case, it is crucial to be assess the value of data being bought.\n\nWhy model-agnostic? *Model training takes months and costs millions of dollars [3], so it is reasonable to assume the largest models are trained only once (or there is a large cost to train a model again). This means data valuation must happen before training begins.* Traditional methods like Data Shapley are impractical because they require training multiple models. Other model-based approaches too require at least one training run before they perform data valuation. Therefore, companies need model-agnostic approaches for data valuation.\n\n**Legal incentives**: EU AI Act Article 10 [4] requires that training data be \"relevant, sufficiently representative, and to the best extent possible, free of errors and complete\" and the Act mandates assessment of \"the availability, quantity and suitability of the data sets that are needed\". Importantly, these requirements apply to the data itself, not to any specific model trained on it. This data-centric approach makes sense because datasets are commonly shared and reused across different AI applications and organizations. Model-agnostic methods are therefore essential for regulatory compliance.\n\n**W2: Why is KAIROS approximating MMD LOO better than LAVA approximating Wasserstein LOO a good thing?**\n\nThanks for this thoughtful question. Model-agnostic methods define influence based on distributional distances, and both Wasserstein and MMD are valid choices. Our key contribution is providing a method that faithfully approximates its stated objective. LAVA faces inherent challenges with Wasserstein approximation due to the non-unique dual potentials and regularization bias, leading to deviations from true LOO rankings. KAIROS offers exact computation of MMD-based influence, providing users with transparent and reliable valuations. This transparency is particularly important in applications like data markets where stakeholders need to understand exactly what the valuations represent and how they were computed. We acknowledge that if someone specifically requires Wasserstein distance for their application, LAVA would be the appropriate choice. However, our empirical evaluations show that MMD-based data valuation generally works well across diverse tasks, and KAIROS consistently achieves superior performance compared to existing methods.\n\n**W3 & Q1: If we use validation data to estimate P(y|x), what is the point of data valuation when the goal is often to estimate P(y|x)?**\n\nThank you for this important question. The validation set is generally small (300 samples in our experiments), so P(y|x) estimates are noisy. However, this empirically seems to provide sufficient signal to identify corrupted training points. After removing these points, we train on the cleaned, much larger training set to get better P(y|x) estimates and higher accuracy.\n\nWe demonstrate this on CIFAR-10 feature noise: A model trained only on validation data achieves 72% test accuracy, while training on the full training set after removing the bottom 20% (identified by KAIROS) achieves 88.5% test accuracy. Even selecting just the top 300 training samples (matching validation size) gives 87.3% accuracy, demonstrating that KAIROS improves sample quality beyond what the small validation set can provide. We will add this discussion and results to the final paper.\n\n**W4 & Q2: The equation for influence of E-MCMD (Eq 8) does not seem to depend on training distribution Q. Is the dependence hidden?**\n\nThank you for this insightful question. The dependence on Q appears in two ways:\n\nFirst, in our derivation of this influence (Appendix B.4, Line 634), there is a Q-dependent constant term. We omit this term in practice because it only affects absolute magnitudes, not relative rankings. However, the absolute value of the influence (that determines whether a small perturbation increases or decreases the E-MCMD) does depend on Q directly.\n\nSecond, the expected E-MCMD influence at any point x, $E_{y \\sim Q(y|x)}[IF_{EMCMD}(x,y;P,Q)]$, depends on Q's conditional distribution. This means that when Q is noisier (more corrupted labels), more points will have high negative influence values. This allows KAIROS to detect label corruption as shown in our experiments.\n\n**W5: Description of Theorem 1**\n\nThank you for pointing this out. We will change the wording to: \"Under mild regularity assumptions, the expected train–validation loss gap is bounded above by the sum of the marginal MMD and conditional E-MCMD. Consequently, removing a point which decreases this distance provably tightens the out-of-distribution error bound for any learning algorithm. The influence is a first-order approximation of the effect of removing a point, suggesting that removing points with large influence scores could decrease the out-of-distribution error.”\n\n**W6: Symmetry property (Proposition 2)**\n\nThank you for this question. We will clarify that points making equal contributions to $\\hat{\\text{MMD}}$ receive identical $\\hat{\\text{IF}}$ scores. Since the true MMD is unknown in practice and $\\hat{\\text{MMD}}$ has established convergence bounds to MMD [5], this consistency property is meaningful. For the true MMD and IF, defining \"equal contribution\" would require circular reference to the influence function itself, making such a statement trivial. We will revise our wording to reflect our claim correctly.\n\n**W7: Unclear Notation**\n\nWe will make changes in the final revision to make the notation clear - defining $n$, $N$, the kernel $k$, and clarifying that the rescaling is up to additive and positive multiplicative constants.\n\n**W8: References**\n\nThank you for pointing this out. We will cite specific pages and results in the final version.\n\n1. For Line 105, the O($\\frac{1}{n²}$) error follows from Taylor expansion of d($P$,$Q_ε$) around ε=0. The second-order term is O(ε²). For leave-one-out with n samples, ε = $\\frac{1}{n-1}$, giving O($\\frac{1}{n²}$) error. We will cite Chapter 20, Page 291 of the textbook [6] which shows a similar expansion.\n\n2. For the non-uniqueness of Kantorovich potentials, we will cite Page 256, Remark 10.30 of the book [7]. We will also cite [8] and Remark 2.3 in [12] which discuss this in detail. Further, we will add this concrete example to illustrate non-uniqueness: \n\nConsider P = $\\frac{\\text{Uniform(0,1)}}{2}$ + $\\frac{\\text{Uniform(10,11)}}{2}$ and Q = $\\frac{\\text{Uniform(2,3)}}{2}$ + $\\frac{\\text{Uniform(12,13)}}{2}$, where Uniform(a,b) denotes the uniform distribution on interval [a,b]. Both of these functions are optimal dual solutions for Wasserstein-1:\n  - f₁(x) = x  \n  - f₂(x) = x if x < 5, f₂(x) = 10-x if 5 ≤ x < 7, f₂(x) = x-4 if x ≥ 7\n\n This demonstrates that optimal Kantorovich potentials are not unique.\n\n3. We will make two clarifications in lines 122-126:\n  - Change \"Crucially, this benign decay does not hold for most IPMs\" to \"Crucially, this benign decay need not hold for most IPMs.\"\n   - Add a reference to Page 39-40 [9] which provides an example where the rate $O(f^{\\*} - f^{\\*}_{ε})$ is less than 1. In our case, this corresponds to a uniform distribution P with Q and Q_ε being the pushforward measures under potentials |x| and max(|x|,$ε^{\\frac{1}{d}}$). The analysis shows that $O(f^\\* - f^\\*_ε)$ = $ε^{\\frac{d+2}{2d}}$ < $ε^1$ when d > 2.\n  - For the claim \"In the Wasserstein-1 case, $f^\\*$ corresponds to a Kantorovich potential, which is non-unique\" we will provide the specific citations discussed in point 2 above to establish that Kantorovich potentials need not be unique, which may lead to non-deterministic influence values for LAVA.\n\n[1]Reuters. Time, openai sign multi-year content deal, June 2024. Published 27 Jun 2024.\n\n[2]Sherwood News. Ai deals to make up a third of shutterstock’s revenue by 2027, June 2024\n\n[3]Cottier, Ben, et al. \"The rising costs of training frontier AI models.\" arXiv preprint arXiv:2405.21015 (2024).\n\n[4]Ortigosa, Adrián Palma. \"Data and Data Governance (Article 10).\" The EU regulation on Artificial Intelligence: A commentary. Wolters Kluwers Italia, 2025.\n\n[5]Gretton, Arthur, et al. \"A kernel two-sample test.\" The journal of machine learning research 13.1 (2012).\n\n[6] Van der Vaart, A.W. Asymptotic statistics, volume 3. Cambridge University Press, 2000.\n\n[7]Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2008.5\n\n[8] Staudt, Thomas, Shayan Hundrieser, and Axel Munk. \"On the uniqueness of Kantorovich potentials.\" SIAM Journal on Mathematical Analysis 57.2 (2025).\n\n[9] Letrouit, Cyril. \"Lectures on quantitative stability of optimal transport.\" (2025).\n\n[10] Lee, Katherine, et al. \"Deduplicating training data makes language models better.\" arXiv preprint arXiv:2107.06499 (2021).\n\n[11] Carlini, Nicholas, et al. \"Poisoning web-scale training datasets is practical.\" 2024 IEEE Symposium on Security and Privacy (SP). IEEE, 2024.\n\n[12] Del Barrio, Eustasio, and Jean-Michel Loubes. \"Central limit theorems for empirical transportation cost in general dimension.\" (2019)."}]

Author rebuttal:

Meta-review: The paper proposes a framework for determining how influential individual training examples are to overall model performance.  The work introduces influence function based on Maximum Mean Discrepancy and their approach offers theoretical guarantees such as symmetry and density separation. Experimental results demonstrate competitive performance across various tasks, including noise detection and data pruning.

Strengths:
1) In the online setting, the method achieves significantly higher computational efficiency due to its closed-form structure.
achieves notable runtime improvements in the online setting.
2) The work addresses a major weakness in the existing approach, LAVA.

Weaknesses:
1) The evaluation is fairly basic and could be improved, e.g., by including large scale data sets.
2) In the offline settings, it is comparable to existing methods.

Overall:  The reviewers were very positive about this paper and remained positive post-rebuttal.  The contribution is novel, comes with theoretical guarantees, and has good performance in the online setting.
Decision: Accept (poster)