[Re] Classwise-Shapley values for data valuation
Authors: Markus Semmler
Miguel de Benito Delgado
Venue: Accepted by TMLR ()

Abstract: We evaluate CS-Shapley, a data valuation method introduced in Schoch et al. (2022) for classification problems. We repeat the experiments in the paper, including two additional methods, the Least Core (Yan & Procaccia, 2021) and Data Banzhaf (Wang & Jia, 2023), a comparison not found in the literature. We include more conservative error estimates and additional metrics, like rank stability, and a variance-corrected version of Weighted Accuracy Drop, originally introduced in Schoch et al. (2022). We conclude that while CS-Shapley helps in the scenarios it was originally tested in, in particular for the detection of corrupted labels, it is outperformed by the conceptually simpler Data Banzhaf in the task of detecting highly influential points.

Published in Transactions on Machine Learning Research (06/2024)
[Re] Classwise-Shapley values for data valuation
Markus Semmler appliedAI Initiative GmbH
m.semmmlerr@pppiedaddddd de
Miguel de Benito Delgado
migucl@appliedai.comm de
appliedAl Institute gGmbH
Reviewed on OpenReview: https: // openreview. net/ forum? id=sFFEYJkqD1
Abstract
We evaluate CS-Shapley, a data valuation method introduced in Schoch et al. (2022) for classification problems. We repeat the experiments in the paper, including two additional methods, the Least Core (Yan & Procaccia, 2021) and Data Banzhal (Wang Jia, 2023), comparison not found in the literature. We include more conservative error estimates and additional metrics, like rank stability, and varianco-corrootedd version of Weighted Accuracy Drop, originally introduced in Schoch et al. (2022). We conclude that while CS- Shapley helps in the scenarios it was originally tested in, in particular for the detection of corrupted labels, it is outperformed by the conceptually simpler Data Banzhaf in the task of detecting highly inffuential points, except for highly imbalancod multi-class problems.
1 Definitions and notation
We define data valuation as the task of assigning a scalar value training points which measures their contribution to the estimated performance of a supervised machine learning model. This sets in the framework of model-agnostic data valuation (in the sense that any model can be used, but one is needed), and we specifically focus the class of methods based in maryinal contributions. These define the value of datum 2; in the training set T := {z1- = (I;, v,) : = 1,.... n as a function of its marginal utility, which is the difference in performance when training with and without Zi, measured over a separate valuation set D. A third set Drest for testing is held out and only used for final evaluation.
The simplest example of valuation method based on marginal contributions Leave-One-Out (LOO), which is defined as the marginal utility of z; for the whole T:
Uno (20 := uT) u( (z,)).
where the utility up 2 + R is the performance of the model when trained on any S C T, measured on a held-out valuation set D. The standard choice for u in classification is accuracy, while for regression one can take for example the negative empirical risk. Because for large training sets T the contribution of single points will be vanishingly small, LOO is typically outperformed by methods averaging many marginal contributions to different subsets of T.
Drawing from the literature in cooperative game theory, the seminal paper DATA-SHAPLEY (Ghorbani & Zou, 2019) takes a weighted mean of the marginal utility for every subset S CTV (2,), with weights given by the Shapley coefficients:
Jshap (2:) == T1
1
Σ
n1 (1=1) [u(SU -u()] |S|
(CT)(*.)
The field includes a wider range of definitions of value. For an overview goals and applications data valuation, refer Sim (2022) well to the documentation of PyDVT. (Transferaah, 2022).

---

Published in Transactions on Machine Learning Research (06/2024)
Because of the exponential cost of O12 evaluations of the utility (each of which entails retraining the model of interest), previous work on Shapley value (Castro et al., 2009) had proposed iterating over permutations to increase the accuracy of Monte Carlo estimates: A random permutation of the index set of T' \ {z.} is sampled, and one iterates over the permutation, incrementally computing marginal utilitics. The authors of Ghorbani & Zou (2019) truncate this process after the inclusion of a new point results within a certain threshold of the total utility. Their method Truncated Monte Carlo Shapley (TMCS) can drastically reduce computation time, thus enabling Shapley-based methods for ML applications for the first time."
This spurred a series of works proposing different Monte Carlo estimators (Okhrati & Lipani, 2021), variations of the Shapley value (Kwon et al., 2021), and other game-theoretic solution concepts like the Least Core (Yan & Procaccia, 2021; Bemmerzoug & de Benito Delgado, 2023) (LC in the sequel), which we include in our analysis. Several sampling strategies to reduce the variance of the Monte Carlo estimates exist (Wu et al., 2023; Covert et al., 2024), but generalizations to so-called semi-values are more successful. The idea is to change the weights in Equation (2) with the goal of compensating for the variance of u(S), which typically increases with model capacity at low set sizes |S]. Beta-Shapley (Kwon & Zou, 2022) gives more weight to subsets S in the lower- to mid-size range by using Beta function for the coefficients. A simpler approach, Data Banzhaf (Wang & Jia, 2023) (DB in the soquel), defined as follows, is of particular interest to our analysis:"
Ubat (Zi) = 2
[u(S (zi)) - -z())- SCTV(#))
Σ
(3)
As mentioned, the main motivation behind DB is addressing the stochasticity in S u(S) (ef. Section 3.5 for more randomness). Intuitively, the constant coefficients 2n-1 are the best one can do for general since given any weighting scheme, it is always possible to adversarially construct a utility with high variance at the set sizes with highest weights. The authors do indeed prove certain optimality results with respect to (wrt.) a notion of stability in value rankings. We will see in the experiments that this simple idea yields the best results in many siluations.
Further lines of work tackle the exponential complexity by learning a model for the utility after a few evaluations (Wang et al., 2022), or by replacing the values with a proxy model altogether Jia et al. (2021). Alternative approaches avoid the cost of game solution concepts altogether, like DATA-OOB Kwon & Zou (2023) which uses the out-of-bae, error estimate of a bagging model. Reformulations of Shapley like AME Lin et al. (2022) use off-tho-shelf models like LASSO to look for sparse solutions with many points being assigned zero value, since one is typically interested in the extreme siluations.5
In this context, CS-SHAPLEY Schoch et al. (2022) (CS in the sequel) appears as valuation method based on Shapley values exclusively designed for classification problems, and aware of the in-class and out-of-class impact of training points. The core observation motivating the work (Claim 1) is that mislabeled points can simultancously improve overall accuracy (the utility), while being detrimental for the model's performance on the class they actually belong to. The authors propose that a better definition of value should account for this phenomenon, and introduce a novel utility Equation (4) which considers the positive or negative impact a datum has within its class and outside it.
To define CS, fix 2; (x,, v1) E T and split the valuation data into Dy. the data with the same label as wi, and its complement D- == DADy,- Analogously, T TT, T-y, is a partition into the subsets of all training data with the same and different class than y, respectively. Trained over any S C T the model
'The rationale for the truncation is that as the size & of the subsets S grows towards [D], individual points contribute less and less to the performance of a model when trained on S, as happens with LOO. Some estimates from stability theory show that it is reasonable to expect an upper bound on the utility of (11kk). It is interesting to note that diminishing returns apply also as & decreases to O. This motivates computing (2) for certain range of set sizes [S| C + BBow Bup], proposed in Watson et al. (2023), an idea that is also behind the weighting strategy of Kwon & Zou (2022) described later.
Despite the formulation with sum over the powerset of T') (2,), as with all game-theoretic methods, in practice one does described above, or uses another sampling strategy. To this avail,
not draw subsets S from it, but iterates over permutations Wang & Jia (2023) introduced Maximum Sample Reuse, an efficient strategy with good performance, see below.
*(Li & Yu, 2023) recently extended DB to weighted Banzhaf values, but we were not able to include this method in our experiments.
TTiis is by far not an exhaustive list, but we want to restrict ourselves mostly methods we discuss.
2

---

Published in Transactions on Machine Learning Research (06/2024)
has in-class accuracy: (z(Dy,) == (# correct predictions over pt.)/DD and out-of-class accuracy: as(D = (# correct predictions over D- D... Given an arbitrary set S yyy; S ET-,, ye the conditional utility conditioned on S_y, v, is defined for every Sy. _ Ty, as:
u(Sy, ( = as Ss )eas(D m.)
(4)
where we define ; Sy US- in order to compute the accuracies." Because of this form and as (Dv)),) + as os(D),)) & 1, ag (D())y.) is controlled by as(Dy,) in the sense that when the latter is small, the effect of out-of-class accuracy on u is negligible, cf. (Schoch et al., 2022, Figure 2). In particular, even a small in-class accuracy leads to greater utility than perfect out-of-class accuracy, cf. (Schoch et al., 2022, Property 1, p. 5). With these ingredients, Schoch et al. (2022) define the conditional CS-Shapley value of 2, given S_y.
((z)S-
E
[u(Sy, U (zi)(s- w.) 1 IS 1S-v.)-
(5)
|
Sv, STo (z.))
Finally, the CS-Shapley value is an average over all possible out-of-class environments S-v.
r(z,) = 2
Σ
(2;11S v.)
(6)
S-M. ST-M
In practice, this sum is approximated in Monte Carlo fashion with a few hundred S-v.- For each S-y. sampling of the Sy, S Tu Mz,) is not done from the powerset as suggested by Equation (5), but following the permutation approach first proposed in Castro al. (2009). In the implementation of Schoch et al. (2022) and ours, one permutation per set is used.
2 Scope of the reproduction
In this report, we set to verify the main claims of Schoch et al. (2022), incorporating two additional methods: Least Core (Yan & Procaccia, 2021) and DB (Wang & Jia, 2023), which led to some unexpected results. To the best of our knowledge, this represents the first direct comparison of DB with other methods in the tasks described in Section 3.4. This work also extends the evaluation of LC conducted in Yan & Procaccia (2021) and replicates the findings of Benmerzoug & de Benito Delgado (2023) across many more datasets and scenarios. Finally, we strive to provide more accurate error estimates and include additional metrics.
Claim Schoch al. (2022) Training points con be simaltaneously beneficial for average accuracy, and detrimental for in-class accuracy. A valuation method accounting for this should perform better than one that does not.
This observation is backed by (Schoch et al., 2022, Figure 1), using one point and one set. How often does it happen that the marginal global accuracy of a datum is positive, but the marginal in-class accuracy is negative, and vice versa? And, how do these sition correlate with better performance of CS wrt. other methods? We address these questions in Section 4.1.
Claim 2 Schoch et al. (2022) CS is generally better suited for classification problems.
We find that is in fact DB which outperforms all methods in the detection of highly influential points, except in the case of highly imbalanced multiple classes, although it fails to do so in noise detection. See Scction 3.4 for a description of these tasks, and Scctions 4.2 and 4.4 for the conclusions.
Claim 3 Schoch et al. (2022) Data values can be successfully transferred across classifiers, including to newral models.
We follow the notation of Schoch et al. (2022), but observe that the sub-index in ag variable. more obvious notation would be a(S, ). Thhere problem with the original notation the paper: p. it. stated that 8-v, is the complement, in S of Sy:: which what is intended. Instead S-g, an arbitrary subset T-w:-
3

---

Published in Transactions on Machine Learning Research (06/2024)
We partially verify the claim for scenario in Section 4.3, but observe that the signal is rather small, if at all present in many cases. Using a second larget classifier, observe almost complete failure to transfer values. This does not correlate with any obvious characteristic of the data, like class imbalance or the prevalence of the property described in Claim 1, leaving open for practitioner the fundamental question of what source model use for value transfer for particular dataset.
Additionally, we observe the following (these claims are our own):
Claim Under full rondommess of data sampling, subset sampling, and training method, CS tends exhibit higher variance than most other methods in the point removal task.
When we resample the datasets for each experiment run, note that the behaviour of CS is more strongly affected by the training / valuation / test split than other methods. This is of relevance since in practice, cross-validation of values is computationally prohibitive. Nevertheless, we observed that the general trends were respected across splits.
Claim 5 Under similar computational budgels, DB is preferable to all other melhods for the idenlification of highly inftuential points, while CS is for the detection of corrupted ones.
We substantiate this in Section 4.2 and Section 4.4.
Claim modified version of the metric WAD (ef. Section 3.4) is better suited for quantitative comperison of valuation methods.
We address this in Section 4.5.
3 Methodology
We run all experiments on the same datasets and models as Schoch et al. (2022), adding two new valuation methods. Details of the datasels, and parameters of the classifiers and valuation methods follow.
3.1 Datasets
Datasets are from OPENML (Vanschoren al., 2013). All but COVERTYPE and MNIST-MULTI are for binary classification. Stratified sampling was used for the splits to maintain label distribution. Image datasets underwent feature extraction using RESNET-18, then dimensionality reduction using 32 PCA components.
Dataset
Type
Features
Instances
%positive
Training
Valuation
Test
Diabetes
tabular
768
65.1
128
128
512
Click
tabular
g*
3000
95.5
500
500
2000
Covertype
tabular
54
3000
(q)
500
500
2000
CPUt
tabular
21
3000
46.7
500
500
2000
Phoneme
labular
3000
70.6
500
500
2000
FMNIST
image
32
3000
50
500
500
2000
CIFAR10 1+
image
32
3000
50
500
500
2000
MNIST-binary* 1
image
32
3000
51.9
500
500
2000
MNIST-multi
image
32
3000
(tt)
500
500
2000
Table 1: Datascts used. (+) 11 fcatures declared online, but effective after fetching using the openml library.
(t) CPU is originally regression dataset, binarized using the threshold 89. (tt) FMNIST is restricted to
the classes "t-shirt and tops" "shirts". tt CIFAR10 is restricted to the classes "automobile" vs "truck".
(t t ti) MNIST-binary is MNIST restricted to the classes "1" vs "7". (t) Covertype is highly imbalanced with 7 classes with frequencies 36.5% / 48.8% / 6.2% / 0.5% 1.6% / 3% 3.5%. (1z) MNIST-multi has 10
classes with almost equal frequencies.

---

Published in Transactions on Machine Learning Research (06/2024)
3.2 Valuation methods tested
Parameters for all methods were taken as suggested in Schoch et al. (2022) or the corresponding papers. Convergence criteria for the methods were kept as consistent as possible by stopping computation after the value for every training point had been updated at least fixed amount of times.
Method
Convergence crit.
Utility evaluations
Parameters
LOO
NA
[T] +
NA
TMCS (Ghorbani & Zou, 2019)
MinUpdates>500
((KTT)
10-4
BetaShap (Kwon & Zou, 2022)
MinUpdates>500
((K]T))
16, 8
CS-Shapley (Schoch et al., 2022)
MinUpdates>>50
(RK(TT//2)1 K
10-1, K =
Data Banzhaf (Wang & Jia, 2023)
MinUpdates>5000
K = 5000 samples
Least Core (Yan & Procaccia, 2021)
NA
K = 5000 constraints
Methods evaluated. Convergence criteria as provided by PYDVL (TransferLab, 2022). See the text for details on each method. (t) "NA" = Not Applicable. (tt) [7|/2 is the expected size of a set S-v, sampled from 2 and hence of its complement. K is the number of samples taken, Le, of permutations for all methods, except for Least Core, where we sample from the powerset. For CS-Shapley, R the number of context samples S v.
Table 2:
Leave-One-Ouu Baseline, no parameters. Values given by Equation (1).
Truncated Monte Carlo Shapley, TMCS (Ghorbani & Zou, 2019): This was the first elficient Shapley-based method and remains one of the most effective approximations. iterates over permutations of the index set of T to reduce variance of the Monte Carlo estimate, and heuristic stopping to reduce computation. For every permutation, marginal utilities in the permutation. The process is stopped early for permutation when the relative change in marginal utility is below a threshold e. Therefore the number of utility evaluations is OKK]T)).
calculated incrementally using the next index
Beta Shapley, BS (Kwon & Zou, 2022): A semi-value approach like TMCS, where the weights for the marginal utilities are defined using a Beta function. For the parameters we use the best values according to the paper, a 16 and B = 1. We use the same permutation sampling scheme as for TMCS.
CS-Shapley, CS (Schoch et al., 2022): The key parameter mentioned in the paper is the number R of context samples S-y,, which we mimic with MinUpdates. For each one of these, one permutation of Sy, is used. Additionally, threshold is used to compare the absolute difference between value estimates and update the values. Finally, there are two minor variants of the algorithm not detailed in the paper. Their implementation does not condition on sets S with |S] < c- 1, where e is the number of classes, which for binary problems means that one never conditions on the empty set. We use the default in PYDVL which does not include this restriction, hence sometimes conditioning on S @ and setting u((] = u(S). Wo lested both without observing major differences.
Data Banzhaf, DB (Wang & Jia, 2023): A semi-value approach with constant weights, with the goal of counteracting the variance in stochastic utility functions. The idea that for any particular choice of weights there will always be utility for which they perform poorly, thus making constant the best choice. In particular wrt. rank stability of the methods, see Figure 9.
The paper also introduces an efficient sampling technique, dubbed Maximum Sumple Reuse (MSR), which for every sample S C 7 updates all indices in the training set. This drastically reduces the amount of utility evalustions required for approximation, by a factor [T' wrt. TMCS.
The idea is that Vbet (2:) Ear. (SUU Elu(S U (=+))] E[n(S)), which can be approximated by sampling sets Sy. T((2,) splitting them into
u(S)), and by lincarity of the expectation ooat
5

---

Published in Transactions on Machine Learning Research (06/2024)
S3i == {Sk: 22 Sk), Szs= == {Sk+ 2, $ S and setting:
(1z)(2;) = SS SES.
((S)
u(S).
S
SESa
In our experiments we chose to stop the mothod after all values had been updated at 5000 times, which means the same amount of utility evaluations. This was to provide a comparison to LC. We also ran DB with permutation sampling as TMCS and Beta Shapley, but performance was comparable or worse at a much higher computational cost.
Least-Core, LC (Yan & Procaccia, 2021): Another game-theoretic approach. It computes values with stability property called Coalitional Rationality, which ensures (in the exact case) that every subset is assigned an aggregate value at least as large as its utility. Given the different nature of the algorithm, which solves linear program and cannot use the same convergence criteria, we choose K = 5000 constraints for stability and run-time considerations."
3.3 Models for value computation
Values are computed using each of the models in Table with the given parameter choices.
Model
Changed parameters
Logistic regression
aolver='liblinen
Gradient Boosting classifier
n_ estimators=40, min_ hts_samples__ee _eaaff66 maxdepth=2
K-Nearest Neighbours
n_ __neghborr=55, reights='unifond kermel='rbf'
SVM
Table 3: Models used to compute values and changes made to the default parameters in SCIKIT-LEARN 1.2.2.
3.4 Tasks for the evaluation of data valuation methods
Data valuation is particularly useful for data selection, pruning and inspection in general. For this reason, the most common benchmarks are data removal and noisy label detection. We describe these and related ones here, and present the results in Section 4.
High-value point removal. (Section 4.2) After computing the values for all data in T = {z; == 1=1, I,...,,,, the set sorted by decreasing value. We denole by Tie the sorted sequence of points (z,, (2,, 2n) for 1 & E n. Now train successively frie and compute its accuracy aTy (Dest) on the held-out test set, then plot all numbers. By using Drest one approximates the expected accuracy drop on unseen data. Because the points removed have high value, expects performance to drop visibly wrt. random baseline.
Low-value point removal. The complementary experiment removes data in increasing order, with lowest valued points first. Here one expects performance to increase relatively to randomly removing points before training. Additionally, every real dataset will include slightly out-of-distribution points, so one should also expect an absolute increase in performance when some of the lowest valued points are removed.
8This principle guarantees that each group is compensated with at least the value it brings in terms of the specified utility. This is deemed particularly relevant when compensating multiple data providers: as purchaser, one would sock a system of credit allocation that motivates the contribution of data. However, we do not believe LC or Shapley-based methods to be relevant in data markets for practical reasons like the scarcity of data, which translates into highly unstable and potential unfair valuations, and the concentrated distribution of values, which make noise major issue (because only those at the extrema are separated enough to be robust against raudonness in the utility, cf. Figure 10).
'A natural choice would be K 125000 constraints, in order to have the same order of magnitude in the number of utility evaluations to Beta Shapley and TMCS (accounting for truncation permutations at around half length). However, the solvers often failed converge and had reduce the number constraints, settling the end for 5000. Given that LC was not the focus of this reproduction, decided postpone investigation of this issue.

---

Published in Transactions on Machine Learning Research (06/2024)
Value transfer. (Section 4.3) This experiment explores the extent to which data values computed with one (cheap) model can be transferred to another (potentially more complex) one. Different classifiers are used as a source to calculate data values. These values are then used in the point removal tasks described above, but using a different (target) model for evaluation of the accuracics ar|a]- A multi-layer perceptron is added for evaluation as well.
Noisy label detection. (Section 4.4) This experiment tests the ability of a method to detect mislabeled instances in the data. A fixed fraction of the training data are picked at random and their labels flipped. Data values are computed, then the a- a-fraction of lowest-valued points are selected, and the overlap with the subsct of flipped points is computed." 10
Rank stability. (Section 4.6) Following Wang & Jia (2023), we look at how stable the top k% of the values is across runs. Rank stability of a method is necessary but not sufficient for good results. Idcally one wants to identify high-value points reliably (good precision and recall) and consistently (good rank stability).
Weighted Accuracy Drop. (Section 4.5) While not a new task, the authors of Schoch et al. (2022) introduce the metric Weighted Accuracy Drop (WAD) as an aggregate way of measuring performance drop in high-value point removal with single scalar. Given a fixed valuation method, with the notation above:
WAD
(an (1 z (log(n) + C)arp
(7)
where we simplified the expression by using the telescopic nature of the inner sum, and the classical approxi- mation of the harmonic series for large n, with C > 0 constant. This weighted mean places more weight on training with the highest valued points, where one hopes to see the highest decrease in accuracy. Note that under the third randomness scenario, i.e, when keeping all but the inner sampling of methods constant, the value aTiu] will be equal for all methods compared, making the first term constant shift, and the sccond term the only effective one.
Even though WAD is convenient for comparison, and it is possible add confidence intervals in Figure 7, we find that it has several drawbacks. First, it is effectively weighted mean of the tail accuracies arli tt]: for which one could argue that stronger decays make more sense (maybe even clipping). More importantly, it does not take into account the heteroscedasticity of the accuracy. In Figure 2 we can see that the 95% Cl tends to grow as data are removed from T, possibly reflecting the fact that after the few initial high values, many are very small, inducing changes in their ranking (see Figure 10 for the distribution of values).
Variance Adjusted Relative Weighted Accuracy Drop. For the aforementioned reason, propose adjustment based on the standard error. We also compare to random values at the same time step, approach we believe to be more informative and easier to interpret. Finally, we keep the hyperbolic decay rate for a closer comparison to WAD, and to avoid an additional degree of freedom choosing decay rates.
Let the valuation methods be indexed with & E fO.....m), where = is random valuation. Let p E (1,..., index runs. Our proposed metric VARWAD for run p and method k > is defined as the average weighted difference to the mean performance achieved with random values:
VarWAD(T. = Σ wx wa(t)(aryo) - arpt)),
(8)
where T.g [t:] == Wrane =p=t (T](c:] is the average performance of the (fixed) model when trained on the subsets of T from the t-th index onwards, as sorted by decreasing value by method in all runs p. The differences are weighted by hyperbolic decay like WAD, and by standard error at each time-step across all
10 synthetic experiment is however hard to put into practical use, since the fraction of course unknown practice. This changes however one retrains with different seeds compute accuracies.
7

---

Published in Transactions on Machine Learning Research (06/2024)
runs: (t) = (1 5 (t))/t, where Sx(t) max, (t) is the normalized standard error sx(t) across runs time-step for method k. The effect of this correction to further reduce contributions where noise is high, and to set them to 0 when it is at its maximum.
3.5 Randomness in value computations
There are three potential sources of randomness that affect the computation of Shapley-like values. The first two stem from the stochasticily of model performance, while the last one is due to the Monte Carlo approximations of the value:
1. For a fixed S C T. the utility u(S) is affected by the randomness in the training algorithm mapping S fs. E.g. for SGD, different initial weights, and different sequences of minibatches (B, (B,) will produce different models fs.
For a fixed, trained model fs, there will be noise in the computation of the score. E.g. the accuracy (S) empirical risk estimate for some loss I (e-g. 0-1 loss), and the distance to the true gencralization error Ex,y (X(s(X), Y)] will depend on the choice of valuation set D.
The set of samples {S) S T = MM used to approximate the sums Equation (5) and Equation (6).
If one is interested in evaluating the convergence properties of a certain valuation method (e.g. how good the Monte Carlo approximation is), one must focus on the third item. One freezes the datasct split, the choice of subsets {S, S T l,..., M} and utility computations u(S,) (which is equivalent to retraining with the same sequence of batches / initialization random seed), and computes the different values with these fixed. A new run of the experiment will keep the split, but sample the subsets Sy anew and compute any new utilities u(S,), reusing the results for all valuation methods. This is the approach we follow whenever possible.
However, the above procedure can produce results which are misleading from a practitioner's point of view. If interested in the performance "in the wild", one wants to at least consider the sensitivity of the method the variance in the estimation of generalization error, mainly in order to understand the impact that the size of D has. In addition, it is crucial consider the stability of the valuation procedure wrt. noise in u(S), e.g. for unstable stochastic training procedures. As discussed above, methods like Beta-Shapley and DB try to tackle this with better choices for the weights of marginal contributions.
3.6 Implementation details
We ran all experiments with the method implementations available in the open source library PYDVI v0.9.1 (TransferLab, 2022), on several high-cpu VMs a cloud vendor. We initially used DVC to organize the experiment pipelines, but lock-file writing times after each combination of parameters where too long (in the several minutes each time). This motivated a switch to MLFLOW (Wilson et al., 2023). Code for all our experiments is available in (Semmler, 2024), including both setups and instructions running them. Tn order to best compare methods and maximally reuse computation, we employed PyDVL's caching strategies for the utility function.
We consistently observed some discrepancies between the results in Schoch et al. (2022) and ours, namely dif- ferent baseline performance of the classifiers across several experiments. We presune different pre- pre-processing or sampling strategies to be the cause, but found the issue of no consequence for the purposes of this report.
I2 experiments this however not the for most modcls Noto that not the magnitude of this in itself, but the variation in value ranking function of the choice of D that is of interest. Note also that we are assuming i.I.d. samples but of course might contain outliers, and mislabeled or corrupt data, thus distorting value computations.
8

---

Published in Transactions on Machine Learning Research (06/2024)
4 Results
We repeat all experiments 20 times, keeping the data splits constant and reusing utility computations whenever possible. All plots use 95% centered percentile bootstrap confidence intervals with 10000 samples, unless otherwise stated. This approach is better suited than Normal CIs for random variables that are not normally distributed (like accuracy numbers) and for which only few data are available. One consequence is that our confidence intervals tend to be larger than those seen in Schoch et al. (2022) and other litcrature.
Box plots display 1tt and 3rd bootstrap quartiles, also using 10000 samples.
4.1 Dataset characteristics
We start with Claim 1, namely the existence of a certain type of points in datasets and the suitability of CS to address it. We look at aggregate statistics over subsets of T: we compute in-class accuracy as (D,,) == (# correct predictions over Dn.) /ID], and global accuracy changes for each training point, averaged over subsets S S T. By using constant coefficients in the averages, they coincide with Banzhaf values Vin Vglob for the respective within-class, and global accuracy utilities. We look at the fraction of points in each dataset that have either Via > and l'glob > E, Vin < and liglab > E, for various values of > 0. Such points are globally useful to the model but either beneficial (denoted *), or detrimental (denoted <>) to their own class, respectively. CS is designed to give more weight to the former than to the latter and is expected to excel for datasets in which the latter abound.
In Figure we see that CPU, DIABETES, FMNIST-BINARE and MNIST-BINARY contain non-negligible amounts of points of type <>, sometimes as much as roughly 40% as many as of type >. However, despite suggestive trends, there is no clear correlation between the frequency (and magnitude) of the phenomenon, and the gain in performance of CS wrt. other methods, and DIABETES, CS seems to leverage the presence of <> points, but this is no longer the case for FMNIST- BINARY, and also not exclusive of these datasets. This rather inconclusive result leaves open the question of how much of an effect the utility Equation (4) has, within the many sources of uncertainly involved.
can be seen e.g. in Figures 2 and & for CPU
cifar10
click
covertype
cpu
diabetes
0.6
0.5
0.5
0.4
0.4
0.4
0.3
0.2
0.2
0.2
0.1
0.1
0.0
0.0
0.0
0.0
0.0
0.02
0.04
0.06 0.08
0.00
0.01
0.01
0.01
0.02
0.01
0.03
0.04
0.01
0.01
0.02
0.02
x10-2
x10"3
x10
x10
x10
fmnist binary
mnist binary
mnist: multi
phoneme
0.4
0.6
0.4
0.4
0.3
0.3
0.3
0.2
0.2
0.2
0.2
0.1
0.1
0.0
0.0
0.0
0.0
0.01
0.03
0.04
0.00
0.10
0.20
0.30
0.40
0.05
0.10
0.15
0.05
0.20
x10-2
x10-3
x10-2
Figure 1: Prevalence of data which positively negatively affects in-class accuracy while increasing global accuracy. The abscissa is change in average marginal accuracy. A point on the <> curve at (I,z) means that a fraction y E [0, 1] of the data simultaneously induce a marginal in-class accuracy decrease and a global accuracy increase of x, averaged over multiple subsets S C T. We note that, surprisingly, the class >< (not plotted) contains anywhere from 10 to 35% of samples for most datasels at E = 0. For the particular case of CLICK, almost 50% of the samples are in <, possibly indicating that the model lacks capacity
4.2 High-value point removal
We continue with Claim 2. In Figure we are able to reproduce the experiment in Schoch et al. (2022), albeit with wider confidence intervals and some spurious differences. The exception is CLICK, where believe
9

---

Published in Transactions on Machine Learning Research (06/2024)
class imbalance to distort results: despite our best efforts, and stratified sampling, all methods except for DB fail at the task for this dataset (observe that more than 40% of the points are removed before a noticeable drop in accuracy). Experimentation with multiple training set splits (not depicted) shows that the general trends are respected under full randomness, indicating that for the remaining datasets, differences to Schoch et al. (2022) are mostly artifacts of the data split. We refer to A.1 for the evaluation with other models.
Qualitatively, CS is outperformed by DB (at much lower computational budget), on 8 out of datasets, seen by the sharp initial decrease in accuracies. We recall that the number of utility evaluations used for DB is 2 orders of magnitude smaller than for the other semi-value methods. LC fails to perform in several cases with the notable exception of PHONEME, where it is unexpectedly on par with DB. For a summary evaluation with WAD and WAD see Section 4.5.
cita/10
click
covertype
cpu
diabetes
0.85
0.95
0.7
0.500
0.75
0.94
0.875
0.00
0.43
0.050
0.75
0.5
0.825
0.800
0.50
200
imnist binary
mnist binary
mnlst. multi
phoneme
0.58
0.96
0:57
0.80
Fects Shoprey
Clas Abe Sheplay
0.55
0.75
Local Cars
0.54
0.5
0.30
Fandom
0.5
vuncatad Marce shaskiy
0.52
0.65
100
200
100
200
100
200
100
200
Figure 2: Accuracy drop of a logistic regression model, for values computed using logistic regression. araxii is number of highest-value samples removed from the training set.
4.3 Value transfer
For Claim 3, we transfer values from each of the 4 models to the remaining ones, plus a neural network for evaluation, resulting in a matrix of 20 experiments. For brevily, we report only on the transfer from logistic regression to a neural network and a gradient boosting classifier, which is the typical situation of interest: cheap value computation, expensive evaluation. For the remaining experiments, we refer to A.2.
With the neural network target (Figure 3), we observe similar behaviour of CS as Schoch et al. (2022), again with the exception of the dataset CLICK. But in our case the addition of DB changes the landscape, since it does better in 8 out of 9 datasets. Interestingly, CS excels at the imbalanced multi-class problem. The trends are similar for all other transfers to the neural network.
For the transfer to a gradicnt boosting classifier (Figure 4), we experience much higher variance than Schoch et al. (2022), leading to the conclusion that all methods are ineffective for all but 4 datasets. In particular, negligible initial performance drop for 4 out of 9 datasets with most methods, invalidating the claim that the most influential samples are assigned the highest values. These, and similar mixed results for other models lead us to question the practicality of value transfer across models, despite occasional good results.
4.4 Noise detection
In Figure 5 we see slightly different. results from those in Schoch et al. (2022): TMCS tends to perform as well as, or better than CS in most cases, while Beta Shapley does much worse. This is best seen in the AUC box plot of Figure 6, where modian AUC for TMCS typically better than for CS. Interestingly, the two cases where CS clearly wins are the imbalanced datasets CLICK and COVERTYPE, whereas loses in the multi-class dataset MUI.T-Muuan.
10

---

Published in Transactions on Machine Learning Research (06/2024)
difar
click
diabeles
0.850
2.55
0.7
0.025
0.875
0.6
0.00
0.93
0.850
0.775
0.92
0.800
0.4
Linatry
binasyy
mmist_multi
phonerme
0.95
0.8
Data Shupky
Ccco es Shacley
Leact Core
Ore
0.2
Tunnaaod Norte Carlo Shep
0.88
100
200
100
202
100
200
100
200
Figure Accuracy drop of a fully connected neural network, for values computed using logistic regression.
covertype
Cpu
0.80
0.90
0.65
0.75
ACC/000
0.75
0.60
0.70
0.92
0.70
0.84
0.68
0.45
0.80
0.55
100
200
100
200
100
200
100
200
20
Imnist binary
mnist binary
mnist multi
phoneme
0.94
0.94
0.60
0.5
Care
0.92
0.92
0.5
0.5
N4R Bundhu Shudley
Lrowe One Out
0.90
Rancem
Trankelkd Shup
0.91
100
200
100
200
Figure 4: Accuracy drop of a gradient boosting classifier, for values computed using logistic regression.
Finally, we observe that DB performs poorly compared with the point removal task, indicating a certain insensitivity to label corruption. It unclear why this happens, but our best guess for this phenomenon is that given a small fraction of corrupted labels, they will be poorly represented among the smaller sample sizes (S), and have negligible effect on the utility for the larger subsets, where they appear more often, while always being weighted with the same coefficients. Given the low rank stability seen in Figure 9, this conjecture must be taken with grain of salt.
4.5 WAD and VarWAD
Following the discussion Section 3.4, we compute WAD (with additional error bars) in Figure for the same high-value removal task of Section 4.2. We then compare this VarWAD Equation (8), which we propose more informative measure of aggregate performance, that better reflects the variability in performance drop.
To see this, consider dataset CLICK in Figure 2: DB exhibits better performance than other methods, albeit with high variability, and yet. WAD reports similar values in Figure 7. VarWAD more accurately depicts the situation in Figure 8, setting DB ahead, while correctly reporting on its high variance. We have a similar situation in MNIST-MULTI, where WAD incorrectly ranks TMCS above DB, but VarWAD yields the intuitively correct ordering. And in CPU, where LC is relocated ahead of Beta Shapley, slightly improving the results with WAD.
11

---

Published in Transactions on Machine Learning Research (06/2024)
cifar10
click
diabetes
0.5
0.8
0.5
0.5
0.5
Prodion
0.6
0.2
0.2
0.2
0.2
02
fmmistt binery
binaty
mn mnnt_multi
phoneme
1.0
1.0
1.0
1.0
0.6
Sela
Cavrrain Staplry
0.6
0.6
Laune
0.4
0.4
0.4
5.6
Thancated Honto Carlo Shaplyy
0.00
0.25
0.50
0.75
1.00
0.00
0.25
3.50
0.75
1.00
0.00
0.23
0.50
0.73
1.00
0.00
0.25
0.50
0.75
1.00
Rocal
Recall
Racal
Roccl
Figure Precision-Recal curve for noisy label detection using logistic regression and 20% of the labels corrupted. Precision is the fraction of noisy samples among the top l values. Recall is the fraction of all noisy samples identifie..
dfar10
click
covertype
diabetes
D
Emnist binary
miiit binary
mnist multi
phoneme
3rt Stapiry
Cuosn ss Stupley
Lram
Leeve
+
0.4
0.6
344
D6
0.
0.6
0.6
0.6
1.0
0.2
0.3
0.4
0.5
0.5
AUC
AUC
AUC
AUC
Figure 6: AUC for Figure 5, noise detection with values computed using logistic regression. Random values removed since they have the expected AUC 0.2.
While we believe VarWAD tends to provide a more useful description of the situation, we note that these are all qualitative observations and somewhat. arbitrary choices. WAD and VarWAD should thus serve as indication that it is important to design better metrics to compare these methods.
4.6 Rank stability
A crucial question in practice how stable is the ranking training points by values across runs. We look at this Figure by plotting the percentage of indices among the top 8% which consistently make these top positions, across all runs. value of 100% for any given means then that the top k% indices by value remain constant in every execution of the method.
The first observation is that both Beta Shapley and TMCS entirely fail to keep any fraction of points among the highest valued across runs. This instability accounts for their inability to induce stark changes in accuracy when removing the first points, as seen in Section 4.2. Next we remark how CS is more stable than DB for COVERTYPE, the highly imbalanced set where clearly outperforms every other method. Besides this case, DB shows generally higher stability than the rest, something it was designed achieve. In particular for CLICK, where the lack of negative cases makes training highly noisy. This is corroborated by the faster drop and wider confidence intervals seen in Figure? 2.
12

---

Published in Transactions on Machine Learning Research (06/2024)
cifar10
click
covertype
disbetes
10
binary
binery
mnisi
phonemm
Heea Shaprey
Checsetc stap
Leant Crre
Truncated Moree Camne Stapley
0.04
0.05
0.08
0.10
0.02
0.04
0.08
2.03
0.10
6.12
0.000 0.225
3-200 0.2/3 0.500 0.525
0.3
0.2
0.3
WAD
WAD
WAD
WAD
Figure 7: WAD for all datasets. Values computed with a logistic regression utility and performance drop of logistic regression model. Because we have fixed the dataset split and reuse utility values, LOO exhibits variance.
cifar10
click
diabetes
3,000 0.002
0.004 0.006 0.008
3.000
6.002
0.004
0.005
9.030
0.015
0,002
0.004
0.005 8 5.008 0.010
0.002 0.004
0.006
0.008
fmnistt
binary
mnist
Clamsate Shapley
Loss core
Leave One Out
Barreat Shapley
Thuncaloed Honto Shaplay
3.000
0.001
0.002
0.200 0.201 0.002
0.003
5.002
0.004
0.006
0.005
0.015
0.035
Figure 8: VarWAD for logistic regression values and point removal using logistic regression for evaluation depicted in Figure 2. Positive numbers indicate relative improvement wrt. to random values. Because we have fixed the dataset split and reuse utility values, LOO exhibits no variance.
Generally speaking however, we observe poor stability across all datasets and methods. Given that the model used deterministic, this means that the given computational budget is insufficient for accurate value estimation. This being a reproduction, we remained faithful to the setup in Schoch al. (2022), but believe that better practical understanding of sample complexity is required for applications. For deterministic methods, Malcki al. (2014) the first to provide Hoeffding-type bounds, which imply up to two orders of magnitude more Monte Carlo samples for an approxination with 0.01 accuracy with probability 0.95. The most influential constant in the bound is factor E = and the choice of this tolerance will depend on the number of training points. For Shapley values, larger datasets imply shorter intervals between values because of the efficiency ariom. 14 But more importantly will also depend on the rank, because most values are concentrated around zero and with very litule distance between each other, as seen in Figure 10. Further bounds in the literature prove even harder apply in practice, e.g. those in Watson et al. (2023). All this means that it is in fact very difficult to choose the constants priori. And the situation in the case of stochastic utilities is clearly even worse, as explained in Wang & Jia (2023).
14This axiom states that the of all the values must equal the total utility, and is fulfiled by Shapley values. For ML applications it has been argued that not essential "distribute" fixed amount of utility among training points Kwon & Zou (2022), and semi-values like RS DB dispense with
13

---

Published in Transactions on Machine Learning Research (06/2024)
cifar10
click
covertype
cpu
diabeles
60
60-
00
60
50-
6^h
20
20
20
20
20
17
53bb
Imnist binaay
maist binary
mnist muiti
phoneme
80
80
80
80-
cambe Shugrey
40
Lovel Core
VS1 Surchal Sruplay
20
20
Travaes North Cute Stapley
33
50
17
33
50
17
50
33
50
Top % considored
Figure Rank stability of all methods. Percentage of indices (y-axis) that remain among the top k% (z-axis) across all 20 runs. Values computed using logistic regression model. LOO is excluded because its rank stability curves essentially flat due to the training set split (which keep constant) being the almost unique source of randomness for LOO.. Similarly we leave out random values which clearly have no stability.
4.7 Value decay
We conclude with an exploration of the value distributions over different datasets. Values tend concentrate around the extrema, following a shape like the tangent function. Because of noise, mid-range values are then typically not informative."" Contrary concentration of values and their rank stability the method's performance.
expectation, observe clear correlation between the
cifar10
click
covertype
cpu
diabeles
1.0
3.0
0.0
-0.
-0.5
-2.
8.10
-1.0
-1.0
125
230
375
500
125
250
375
500
125
250
375
500
125
250
375
500
32
36
128
binary
binary
phoneme
1.0
1.0
3.0
1.0
0.5
0.5
0.5
0.5
Cascutte Shopley
0.0
0.0
0.0
Loadl
Leave 3re Out
VSN Porchal Siaplty
-0.5
Tuncaosd Nonto Carts Stap
230
375
500
125
250
375
500
125
230
375
500
125
250
500
Figure 10: Value decay (y-axis) for all methods. a-axis is value rank. Values computed on a logistic regression model, sorted decrcasingly and normalized for comparison. 99% t-student confidence interval over 20 runs.
5 Discussion
As is often the case with data valuation methods, the multiple sources of randomness make it difficult to draw clear-cut conclusions. We have attempted to isolate the evaluation of convergence properties of the methods, as described in Section 3.5, but the fact remains that stochasticity in u(S) due to training or evaluation is the major hurdle that Shapley-based valuation methods face (besides time complexity). An interesting path
We note in passing that the changes in curvature of the value decay function might be good spots pick for automatic threshald selection in
sample identification.
14

---

Published in Transactions on Machine Learning Research (06/2024)
to explore pointed to us by reviewer would be to use different weighting scheme in Equation (5) and Equation (6), perhaps leading to class-specific Data Banzhal method incorporating the best from both.
It is alas not possible to provide clear advice to a practitioner, other than perhaps to prefer DB for high-value point identification, except for highly-imbalaneed multi-class problems, where CS has proven superior. A very strong argument in favour of the former its much greater sample efficiency when using MSR, but this technique could in principle be applied to the other methods, something that we must leave for future more comprehensive benchmark.
We cannot unequivocally substantiate Claim with our experiments and must conclude that, while definitely present, the successes of CS might be partly due to its unique utility function, partly to sampling properties and other factors. Additionally, we find that DB outperforms CS for high-value point removal with all models, except, interestingly, for COVERTYPE, in favour of the adequacy of CS for multiclass problems. The situation is reversod in noise detection, partially invalidating Claim 2 within the context of the new methods added. Attempts (ransferring values from models to 5 yield mixed results, with DB as winner in many scenarios, but Claim 3 can only be accepted with caution, given the practical difficulties involved. In particular, transferred values might only useful in applications where manual inspection of the selected data is performed (which is what we recommend in any case).
Next, while DB is designed for rank stability, CS performs better in this respect for many datasets, although it must identify less important points than DB, given the worse curves in Section 4.2. This "failure" might well be due to the low sample regime, and inclusion of Maximum Sample Reuse in future benchmarks should DB is proven to be best at compensating for worst-case noisy utility, showing that other choices of valuation method will be preferable in certain situations.
Finally, our proposed metric Equation (8) tends to capture qualitative behaviour of the methods better, but remains heuristic and arbitrary.
We conclude by remarking that the evaluations have been done at a constant computational budget (in terms of utility evaluations) when possible, but that, as explained in Section 4.6, this budget needs to be increased for most methods for reliable value estimation. The exceptions have been LC for computational reasons, and MSR Banzhaf, which we felt was only fair to the original work. Nevertheless, final performance in downstream tasks is what matters in the end and lower budgets might work just fine. All in all, marginal contribution- based data valuation methods of the sort considered in this reproduction, while still far from being fire- and-forget, automatable data selection mechanisms, remain a tool in the belt of a cautious practitioner who appreciates the value of spending time looking at their data.
References
Anes Benmerzoug and Miguel de Benito Delgado. [Re] If you like Shapley, then you'll love the core. ReScience C, #32):#32, July 2023. doi: 11.221/2enoddo.817373.. URL https://zende: org/record//117773.
Javier Castro, Daniel Gómez, and Juan Tejada. Polynomial calculation of the Shapley value based on sampling. Computers & Operations Research, 36(5):1726-1730, May 2009. ISSN 0305-0548. doi: 10.1016/ j.cor.2008.00 20..2008..4.004. URI. https:///wwt http///wwaieneeedrreeeeeee cecceemccedreet..ccc/c/ceeeeeeed wtt.c//oo/cccccceeeeeeeee.eeeeee
lan Covert, Chanwoo Kim, Su-In Lee, James Zou, and Tatsunori Hashimoto. Stochastic Amortization: A Unified Approach to Accelerate Feature and Data Attribution, January 2024. URL httpp//arxxv.org. abs/2401. 15866.
Amirata Ghorbani and James Zou. Data Shapley: Equitable Valuation of Data for Machine Learning. In Proceedings of the 36th International Conference on Machine Learning, PMLR, pp. May 2019. URL
PMLR,
mlr. .aa....ressv/// http:/9/ggghobbaa//a www.gganiiieeeg wwtpaanii9c...haam.
Ruoxi Jia Fan WU, Xuehui Sun, Jiacen Xu, David Dao, Bhavya Kailkhura, Ce Zhang, Bo Li, and Dawn Song. Scalability vs. Utility: Do We Have To Sacrifice One for the Other in Data Importance Quantifi- cation? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation, pp. 8239-828 8249-8247,42 2021. URL https: //opnaccces/ssssss thecvf comm/oontat///////////ttttt vs. vs._ Utility Do_ We Have To Sacrifice One for 2V21_202111____22222222222221112 2021__aaerrr______________a__2e0 html.
15

---

Published in Transactions on Machine Learning Research (06/2024)
Yongchan Kwon and James Zou. Beta Shapley: A Unified and Noise-reduced Data Valuation Framework for Machine Learning. In Proceedings of the 25th International Conference on Arlificial Intelligence and Statistics (AISTATS) 2022, volume 151, Valencia, Spain, Jamuary 2022. PMLR. URL https: //axxv.. org/abs/2114 14049.
Yongehan Kwon and James Zou. Data-OOB: Out-of-bag Estimate as a Simple and Efficient Data Value. In Proceedings of the 40th International Conference Machine Learning, pp- 2023. URL https://proceedding mml. httpreeeddnnn.....rrrrreeeeee/ee
PMLR, July
Sataas/v222//woo2220 httll.
Yongchan Kwon, Manuel A. Rivas, and James Zou. Efficient Computation and Analysis of Distributional Shapley Values. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics, pp. 793-801. PMLR, March 2021. URL https://proceeddina. mml.
ptt.eeeresses.....rrr.r.ss..eeee www.ppresssvii. htt///1110////111 html.
Weida Li and Yaoliang Yu. Robust Data Valuation with Weighted Banzhaf Values. In Proceedings of the Thirty-seventh Conference on Newral Information Processing Systems, New Orleans, Louisiana, USA, November 2023. URL https://openreviet bet/forum?id=uuee
Jinkun Lin, Anqi Zhang, Mathias Lécuyer, Jinyang Li, Aurojit Panda, and Siddhartha Sen. Mcasuring the Effect of Training Data on Deep Learning Predictions via Randomized Experiments. In Proceedings of the 39th International Conference Machine Learning, pp. 13504. PMLR, June 2022. URL attps://proceeding mww.iinnmm.......mmmmm.......... wwtppp/eeee///i//.ppppp www.ii..nnnm......... html.
Sasan Malcki, Long Tran-Thanh, Greg Hines, Talal Rahwan, and Alex Rogers. Bounding the Estimation Error of Sampling-based Shapley Value Approximation. 13066 42636066666.6.0. [es], February 2014. URL https:
//arxiv. org/ abs/aaas/// 1306.///3....00 4265..
Ramin Okhrati and Aldo Lipani. A Multilinear Sampling Algorithm to Estimate Shapley Values. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 7992-7999. IEEE, Jamuary 2021. doi: (AA110///CP11888022202.01199
URL
Stephanic Schoch, Haifong Xu, and Yangfong Ji. CS-Shapley: Class-wise Shapley Values for Data Valuation in Classification. In Proc. of the Thirly-Sirth Conference on Neural Information Processing Systems (NewrIPS), New Orleans, Louisiana, USA, October 2022. URL https://openrrevie:: www.://wwwwe.eetootouuunniii... KTOcrOR5mQ9.
Markus Semmler. [Re] Class-wise Shapley Values for Data Valuation. applicdAI Institute gGmbH, February 2024. URL https: http:///ttaubb.ccccaaaaaaataaaee hataaaaaanattttteeeeeesssseeeeee
Rachael Hwee Ling Sim, Xinyi Xu, and Bryan Kian Hsiang Low. Data Valuation in Machine Learning: "Ingredients', Strategies, and Open Challenges. In Thirly-Fire International Joint. Conference on Artificial Intelligence, volume 6, pp. 5607 5614, July 2022. doi: 10.24963 2t.24493/jjcai../// org/ htgp/porccedins//2222/1771
URL https::/www.ijp
Team TransferLab. DVL The Python Data Valuation Library. appliedAI Institute gGmbH, 2022. URL
Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. OpenML: Networked science in machine learning. STGKDD Eaplorations, 15(2):49-00 60, 2013. doi: 10.1145 000411999.24419999 URL http: //doi. acm. wtrg/10.1145/2855 22211.......111...... 264119.
Jiachen T. Wang and Ruoxi Jia. Data Banzhaf: A Robust Data Valuation Framework for Machine Learning, In Proceedings of The 26th International Conference Artificial Intelligence and Statistics, pp. 6388- 6421. PMLR, April 2023. URL https://proceee http::/ppooeeeiig........... wtt..perrs/vv0///vgggge html.
Tianhao Wang, Yu Yang, and Ruoxi Jia. Improving Cooperative Game Theory-based Data Valuation via Data Utility Learning. In International Conference on Learning Representations (ICLR 2022). Workshop on Socially Responsible Machine Learning. arXiv, April 2022. doi: 10.48550/arXiv www.xxx...117..... URL
http:///arrvv..o/aab////00// https://arxxe wrwg/ba/////////////////// 06336v2..
16

---

Published in Transactions on Machine Learning Research (06/2024)
Lauren Watson, Zeno Kujawa, Rayna Andreeva, Hao-Tsung Yang. Tariq Elahi, and Rik Sarkar. Accelerated Shapley Value Approximation for Data Evaluation, November 2023. URL 05346.
Ben Wilson, Corey Zumar, Daniel Lok, Gabriel Fu, Harutaka Kawamura, Serena Ruan, Weichen Xu, and Yuki Watanabe. MLflow: A Machine Learning Lifecycle Platform. MLflow, December 2023. URL https:
//github. wttp///w/w//////.///////ooo/oo/.
Mengmeng Wu, Ruoxi Jia, Changle Lin, Wei Huang, and Xiangyu Chang. Variance reduced Shapley value estimation for trustworthy data valuation. Compulers & Operulions Reseurch, 159:106305, November 2023. ISSN 0305-0548. doi: 10. 10.1016/j.com 202..0022.202 URL http:: http://wwwiiincciirerrcc.c www.sseeeecdddeett...sososoooeee art.cce/ppL//00666433301187
Tom Yan and Ariel D. Procaccia. If You Like Shapley Then You'll Love the Core. In Proceedings of the 35th AAAI Conference on Artificial Intelligence, 2021, volume 6, pp. 5751-5759, Virtual conference, May 2021. Association for the Advancement of Artificial Intelligence. doi: 10.1609 10.1609/aaai. v3i....35...1172) URL
https://ojs.. aaai org/ prgg/nndx.../ Beeex..pp//AAae php/AAAI/ httpa//aariiceeeeee/11////// //tp://///////////rrr///////////
A Additional experiments
A.1
High-value point removal
citar10
click
covertype
cpu
diabetes
0.951
0.90
Accersoy
0.50
0.952
0.88
0.7
0.550
0.86
0.948
0.84
0.5
100
200
100
200
1.00
binary
Linisty
mnist_multi
phoneme
0.95
buto Sharay
0.94
0.95
0.75
Clematee Stepley
Lead
0.82
0.70
0.50
0.65
Farsom
Thin Nome-Carl Stapley
0.60
100
200
160
200
100
200
Figure 11: Accuracy drop of an SVM, with values computed on an SVM.
difr110
click
covertype
cpu
diabetes
0.70
0.60
0.85
0.65
0.80
0.8
0.50
0.00
0.5
0.75
0.5
0.40
0.4
1,0
200
100
200
1.00
700
130
200
20
40
60
Imnist binary
miist binayy
mnist multi
phoneme
0.0
Stepkey
0.6
0.0
3.8
0.75
Cere
0.7
0.5
5.70
NSR Banduu' Shopray
Lrove One Out
Rantem
0.6
0.6
5.65
Tranceled Murka Carlo Shup
100
200
100
200
Figure 12: Accuracy drop of a KNN classifier, with values computed on a KNN classifier.
17

---

Published in Transactions on Machine Learning Research (06/2024)
cifar10
click
covertype
diabetes
0.80
0,70
0.75
0.65
0.875
0.7
0.60
0.55
0.50
0.125
0.05
0.45
0.5
fmnist binary
binary
mmnis
phoneme
0.94
0.
0.775
Ohvorato Stepley
Dalu Shupley
lead Core
0.15
Maare Shu
0.92
0.40
0.725
0.85
0.700
Tuncated Norte Carto
103
200
0
100
200
100
200
100
200
Figure 13: Accuracy drop of gradient boosting classifier, for values computed using gradient boosting classifier.
A.2 Value transfer
dfar10
click
covertype
dabetes
0.053
0.0
0.775
0.5
0.6
0.750
0.900
0.725
0.5
fnnist hinary
mnist binary
mnist multi
phoneme
0.96
0.08
0.50
bdda Snashey
0.94
0.75
Lead: Core
acne so Shaskay
0.92
One
0.54
0.70
Rantom
Norta Carla Shaprey
0.55
100
200
100
200
100
200
100
200
Figure 14: Accuracy drop of fully connected neural network, for values computed using an SVM.
cifa/10
click
covertypa
diabetes
0.953
0.50
0.75
0.54
0.6
0.70
0.945
0.850
0.50
0.825
0.94
0.800
0.935
200
100
fmnist, binary
binary
phoneme
0.825
0.95
0.800
(byoraie Steprey
Data Shoprey by
0.75
0.775
Local
6.70
Randam
0.58
Tunuaaal Manue Curto
0.68
0.700
100
200
100
200
100
200
100
200
Figure 15: Accuracy drop of fully connected neural network, for values computed using KNN.
18

---

Published in Transactions on Machine Learning Research (06/2024)
1iar100
click
covertype
cpu
disbetes
0.00
0.70
0.65
0.60
2.84
0.60
0.82
100
200
100
200
100
200
100
200
20
fmnist
multi
0.60
0.80
Stapley
0.34
Stapkey
0.75
0.5
Bundhrl Shupray
0.70
Ransom
Trankaind Plrto Carlo Shup
0.50
0.90
0.40
Figure 16: Accuracy drop of gradient boosting classifier, for values computed using SVM.
dfo/10
click
covertype
CPU
diabetes
0.0
0.70
0.50
0.54
0.65
0.88
0.70
0.70
9.53
3.62
0.50
0.60
0.45
0.82
100
200
100
200
100
200
100
200
20
40
60
finnist binery
mnist, binary
multi
phoneme
0.65
0.94
Data Shuskey
0.923
0.02
0.50
0.55
One-
0.875
Viait durchul
Random
0.850
Turcacel Norte Carla Steprey
Figure 17: Accuracy drop of gradient boosting classifier, for values computed using KNN.
19


Reviews:
[{"review_id": "Wcnk14XVE4", "reviewer": "Reviewer_Rorp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Thorough evaluation of CS-Shapley with various experimental settings that are relevant for practitioners.\n- Two additional recent methods are included in the experiments for comparison.\n- The reproduction scopes are constructed by carefully analyzing the claims made in the CS-Shapley.\n- Each experimental setting is described in detail, how the CS-Shapley performance compared with other methods.\n\nWeaknesses:\n - The authors mentioned that there are some discrepancies in the results of their experiments vs the results of Schoch et al. (2022). The authors suspected that it was possibly due to the different pre-processing or sampling strategies. I think this discrepancy needs to be investigated in more detail, on the source of the discrepancy. If it's possible, the authors should consult the CS-Shapley's authors about the experimental setting or hyperparameter tuning, to reduce the discrepancy, so that the comparison will be fairer.\n- As this paper falls into the reproducibility track, a detailed description of the experimental settings needs to be presented in the paper. This is to make sure that all the experiments are conducted in the settings appropriate to the original model and facilitate fair comparison among the models. The detailed description may include the dataset splits, hyperparameter values on each model, the training algorithms and their parameters, the random seed used, as well as other settings that are relevant to the experiments. Therefore, anyone who wants to redo the reproduction can get the same results/conclusions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present an evaluation of CS-Shapley, a classwise data evaluation model by Schoch et al. (2022). The authors reproduce the experiments in the paper with the inclusion of two additional models as comparisons, the Least Core (Yan & Procaccia, 2021) and Data Banzhaf (Wang & Jia, 2023). The authors mentioned that there are some discrepancies in the results of their experiments vs the results of Schoch et al. (2022), possibly due to the different pre-processing or sampling strategies. The authors evaluate the methods in several experimental settings, based on the claims mentioned in the CS-Shapley paper. After the full evaluation, the authors conclude that while CS-Shapley helps in some tasks like the detection of corrupted labels, it is outperformed by  Data Banzhaf in the task of detecting highly influential points.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the concern I have in the weakness section above.", "broader_impact_concerns": "I do not have any broader impact concerns for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "httKuKCV4t", "reviewer": "Reviewer_Wbud", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. I think that reproducibility work (and more general retrospective work) is inherently valuable because the ML community still has a great need for it, despite the existence of venues that now accept this kind of work.\n1. I think that this particular study is thoughtfully done.\n1. The additions of Data Banzhaf, Least Core, and the two metrics are a useful update.\n1. Section 1 also has value in providing a survey of \"marginal contribution-based methods\" for data valuation.\n\n### Weaknesses\n1. As a flip side to the last point under Strengths, it is not clear what is the motivation for focusing the study on CS-Shapley, as one method among many.\n1. I found many phrases, etc. to be unclear:\n    1. Footnote 2, \"diminishing returns apply also as it decreases to 0\": What is \"it\" here? Still the subset size $k$?\n    1. Top of page 3, \"changes in $a_S(D_{-y_i})$ are controlled by changes in $a_S(D_{y_i})$, and the greater in-class accuracy is, the smaller the effect of the other is on $u$\": I do not understand what is meant by one \"controlling\" the other. I also wonder whether \"smaller\" should be \"larger\" since the multiplier $a_S(D_{y_i})$ (the in-class accuracy) is larger.\n    1. Table 2: What are $K$ and $S$? I think that $K$ is the number of permutations, except for Least Core where it is the number of constraints/subsets?\n    1. Footnote 8, \"scarcity of data, and the concentrated distribution of values which make noise a major issue\": Could the authors please elaborate? Is the second point related to Figure 10?\n    1. Below eq. (8), \"corrected by standard error\": What correction?\n    1. Section 3.5 below the list: Does \"choice of subsets\" refer to the $S_j$ in item 3? Later, does \"reusing them for all valuation methods\" mean reusing utilities $u(S_j)$ computed for previous subsets?\n    1. Section 3.6: What is GCP?\n    1. Beginning of Section 4, \"bootstrap confidence intervals with 10000 samples\": Are these 10000 resamples of the 20 runs?\n    1. Section 4.1: It would be good to show equations or refer to previous equations to define in-class and global accuracy and the aggregate statistics of these.\n    1. Middle of page 9\": What is OP?\n    1. \"a factor $\\epsilon^{-2}$, and this choice ... the efficiency axiom\": What is \"this choice,\" the choice of tolerance $\\epsilon$? What is the efficiency axiom?\n1. Possible corrections:\n    1. Bottom of page 2, \"$T = T_{y_i} \\uplus T_{-y_i}$ is a partition into the subsets of all data\": Should \"all data\" be \"training data\"?\n    1. Eq. (5): Should $n - 1$ be $|T_{y_i}| - 1$ instead? Later on in \"powerset as suggested by Equation (6),\" I think the reference should be to (5).\n    1. Eq. (7): I do not see where the $n (n+1) / 2$ factor comes from. Should it be $\\sum_{j=1}^n 1/j$ instead?\n    1. Section 3.6: The link to code is missing.\n    1. Figure 1 caption, \"$y$% of the data\": Is $y$ really a percentage, or a fraction in $[0, 1]$? At $\\epsilon = 0$, should the four fractions $<>$, $>>$, $><$, $<<$ (latter two not shown) add up to 1?\n\n### Minor comments\n- Bottom of page 3, \"Additionally, we observe the following\": I think these are the authors' claims and not Schoch et al.'s. This could be clarified.\n- Section 3.5, item 2 in the list: Would it be possible to stick to the $a_S(D)$ notation instead of introducting $\\hat{\\mathrm{Err}}$?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission is a reproducibility study of CS-Shapley, a data valuation method proposed by Schoch et al. (2022). The experiments of Schoch et al. (2022) are repeated with two additional methods compared, Least Core and Data Banzhaf, and two additional metrics, rank stability and variance-adjusted weighted accuracy drop. The experimental findings of Schoch et al. (2022) are updated in light of the additional comparisons. In particular, while CS-Shapley remains preferred for the task of corrupted label detection, it is outperformed by Data Banzhaf in identifying highly influential points. Positive results on transferring data values from one model to another are also tempered.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In addition to addressing the weaknesses above, which I think is necessary and straightforward, I think an answer to the following would strengthen the work:\n- Since Least Core is a different approach with a different measure of complexity, I wonder about the sensitivity to the somewhat arbitrary choice of $K = 5000$ constraints.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3ouu3Brz9i", "reviewer": "Reviewer_vPWC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThere are a growing number of data valuation methods and a lack of systematic comparisons between them. Furthermore, the various papers typically report only a subset of popular metrics, which creates a risk of reporting only those where the proposed method performs best. This therefore seems like a helpful reproduction study focused on CS, but which also helps us understand the comparative strengths of certain other methods (TMCS and DB). It seems to play a role a bit like the OpenDataVal library/benchmarks (which could possibly be cited), but with some methods that are not included there.\n\nAs mentioned in the summary above, I appreciated that the paper was very clear in its setup and the claims it evaluates. I found the literature review clear and helpful as well.\n\nAmong the experiments, I thought section 4.1 was particularly interesting and not sufficiently explored in the original CS work.\n\n## Weaknesses\n\nMy feelings are mostly positive about the study, and the weaknesses I'm pointing out mostly aren't that important. But they're relatively easy changes that I believe could improve the paper.\n\n- A nit, but in the introduction maybe \"margin-based methods\" should be \"marginal contribution-based methods\" ? Margin-based seems to connote something else.\n- In footnote 2, I'm confused why Watson et al is cited for truncating the computation for large subsets, didn't Data Shapley introduce this?\n- Also in the introduction, Okhrati & Lipani don't propose a variation on the Shapley value - they propose a different Monte Carlo estimator.\n- On the topic of variations on the Shapley value, another recent work is \"Robust Data Valuation with Weighted Banzhaf Values\" (2023).\n- On the topic of reducing the variance of Monte Carlo estimates, there's a recent paper that proposes an amortization trick that would work for many of the data valuation methods here: \"Stochastic Amortization: A Unified Approach to Accelerate Feature and Data Attribution\" (2024).\n- In the middle of page 2, weigthing -> weighting \n- When introducing the CS method at the bottom of page 2, can the authors explain why the original CS paper uses denominator $|D|$ in the class-specific accuracy calculations rather than the respective class counts? The only rationale I can see for this is the property $a_S(D_{y_i}) + a_S(D_{-y_i}) \\leq 1$, but this doesn't seem very important (more on this next).\n- Regarding the above-mentioned property, the authors write that this guarantees that for a fixed first value, an increase in the second value leads to a smaller conditional utility. The second term is put through an exponential in eq. 4, which is an increasing function, so that sounds backwards? \n- It seems like the class-specific aspects of CS are orthogonal to their choice to use Shapley values, and we could just as easily define a class-specific version of Data Banzhaf (say \"CDB\"). Comparing CS to DB is a bit unsatisfactory, because we don't know if the performance differences are due to the class specificity or the Shapley/Banzhaf choice, and this would be best resolved by including the new CDB method. After seeing the results, I think the need for such a comparison is somewhat diminished by the inclusion of TMCS because it helps isolate the impact of class specificity in CS, but it would be nice if the authors discussed this more.\n- I imagine this question could be resolved by reading Yan & Procaccia more closely, but perhaps the authors can comment on the fact that the \"core\" concept is not uniquely defined, or even guaranteed to exist for all cooperative games?\n- I was a bit confused by the emphasis on high variance in the test set accuracy estimates. In many cases, especially those with subsampling to reduce the dataset size (e.g., MNIST), it would be easy to increase the test set sizes from Table 1 and reduce the variance.\n- The VarWAD metric didn't seem very important, and the authors write that it \"barely improves the ranking of WAD.\" What's the motivation to include it here?\n- When discussing the different sources of randomness, it might be worth clarifying that most of the models here have no training stochasticity.\n- In figure 2, it seems surprising that for $\\epsilon = 0$ there aren't more points in the $>>$ category: wouldn't we expect the vast majority of data points to be helpful both globally and for their class? The portion of points in this category never exceeds 0.6 across all the dataset. Perhaps the authors can comment on this.\n- In figure 3, the claim that all methods except DB completely fail seems like an exaggeration. Almost all the methods outperform the random baseline, so that seems like some form of success?\n- None of the figures include x-axis labels. These are sometimes but not always explained in in the figure captions, and it would be much easier for readers to include them in the plots.\n- The rank stability metrics in figure 10 are concerning, they seem to imply that none of the methods use nearly enough compute, including CS and DB. If this paper were proposing a new method and using these experiment settings, I could see this being grounds to reject the paper because none of the valuation scores are accurately computed; but since it's a replication paper settings from the previous works, maybe there should instead just be a stronger statement about this shortcoming in prior works? It seems to underline the need for much more efficient estimation methods.\n- As an example of where the missing x-axis labels is a problem, I can't understand the x-axis in figure 11 and the relevance of 50% of the training data. It might also be helpful to clarify whether the normalization preserves negative values or forces all scores to be positive.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is a replication study of Classwise-Shapley (CS) from Schoch et al. (2022), which examines the main proposal as well as some existing and new baselines, plus evaluation via some existing and new metrics. The paper is very clear about its setup (e.g., the models used, the amount of compute and hyperparameters used for each data valuation method). It's also clear about the main claims it evaluates, which are taken from the original work.\n\nOverall, the paper supports certain claims from the original work and contradicts others, and this is only possible by replicating the experiments in full and adding context from missing baselines. It seems like a valuable study for helping practitioners understand the strengths and weaknesses of the CS method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Several requested changes are described above.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is well-written, with clear objectives, methods, and take-away messages. All of the reviewers voted in favor of accepting this paper, and I agree with them. Since it is a high-quality reproducibility study that goes well beyond simply re-running the code of existing methods, I recommend this paper for a "Reproducibility Certification."
Decision: Accept (poster)