CS-Shapley: Class-wise Shapley Values for Data Valuation in Classification
Authors: Stephanie Schoch
Haifeng Xu
Yangfeng Ji
Venue: NeurIPS 2022 Accept (2022)
Keywords: data valuation
Shapley values
classification
value function

Abstract: Data valuation, or the valuation of individual datum contributions, has seen growing interest in machine learning due to its demonstrable efficacy for tasks such as noisy label detection. In particular, due to the desirable axiomatic properties, several Shapley value approximations have been proposed. In these methods, the value function is usually defined as the predictive accuracy over the entire development set. However, this limits the ability to differentiate between training instances that are helpful or harmful to their own classes. Intuitively, instances that harm their own classes may be noisy or mislabeled and should receive a lower valuation than helpful instances. In this work, we propose CS-Shapley, a Shapley value with a new value function that discriminates between training instances’ in-class and out-of-class contributions. Our theoretical analysis shows the proposed value function is (essentially) the unique function that satisfies two desirable properties for evaluating data values in classification. Further, our experiments on two benchmark evaluation tasks (data removal and noisy label detection) and four classifiers demonstrate the effectiveness of CS-Shapley over existing methods. Lastly, we evaluate the “transferability” of data values estimated from one classifier to others, and our results suggest Shapley-based data valuation is transferable for application across different models.

CS-SHAPLEY: Class-wise Shapley Values for Data Valuation in Classification
Stephanie Schoch" Haifeng Xu Yangfeng Ji
"Department of Computer Science, University of Virginia, Charlottesville, VA 22904 Department of Computer Science, University of Chicago, Chicago, IL 60637
Abstract
Data valuation, or the valuation of individual datum contributions, has seen growing interest machine learning due its demonstrable efficacy for tasks such as noisy label detection. In particular, due to the desirable axiomatic properties, several Shapley value approximation methods have been proposed. In these methods, the value function is typically defined as the predictive accuracy over the entire development set. However, this limits the ability to differentiate between training instances that are helpful harmful to their own classes. Intuitively, instances that harm their own classes may be noisy or mislabeled and should receive a lower valuation than helpful instances. In this work, we propose CS-SHAPLEY, Shapley value with a new value function that discriminates between training instances' in- class and out-of-class contributions. Our theoretical analysis shows the proposed value function (essentially) the unique function that satisfies two desirable properties for evaluating data values in classification. Further, experiments two benchmark evaluation tasks (data removal and noisy label detection) and four classifiers demonstrate the effectiveness of CS-SHAPLEY over existing methods. Lastly, we evaluate the "transferability" of data values estimated from one classifier others, and our results suggest Shapley-based data valuation is transferable for application across different models.
Introduction
Data valuation methods aim to quantify the contribution of each datum the predictive performance of learning model. Among these, Shapley values have been proposed as a means to identify helpful harmful data [3, 10]. A number of approximations and extensions for Shapley-based data valuation have been developed, with demonstrable efficacy for tasks such as mislabeled or noisy example detection and data selection [3, 10, 14, 4, 11]. The performance gains of Shapley- approaches over alternative data valuation methods have typically been attributed to the axiomatic basis of Shapley values that satisfies fairness guarantees from cooperative game theory. Importantly, Shapley values rest an underlying assumption that a game is well-represented by its value function [21].
The value function of prior Shapley- -base data valuation methods has typically been defined as the predictive accuracy over the entire development set. However, in the context of valuing data for learning models on classification tasks, this may have limited ability differentiate helpful or harmful training instances. Consider the case where we want to evaluate the value of data points i and for binary classification task, where both points belong to class 1. As shown in the real world example provided in Figure 1, if the predictive accuracy on the development set is the same when adding each point individually, then the contribution of these two data points considered to be equivalent. However, how and j contribute to the classifier differs. To be specific, the contribution of data point to class positive (helpful), while the contribution of class is negative (harmful). Similar distinction between training instances that are "helpful" and "harmful" to their own class
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Figure 1: Development accuracy by class when adding two different points, i and j. to the training set of a binarized version of CIFARIO, using logistic regression (the experiment setup is provided in sec- tion 5). Both points belong to class and produce the same overall development accuracy change. However, i increases the in-class accuracy, and j decreases the in-class accuracy. If mcasuring contribution using the overall predictive accuracy, and j will have equiva- lent contributions. In contrast, by differentiating be- tween in-class and out-of-class accuracy changes, the proposed value function considers i to have a larger contribution than j.
6,3/0
0.369
11.bb/
0.365
areumey
0,360
0.355
0.350
0.345
0.340
0.3385
0.338
0,335
has previously been used for post-hoc analysis in prior data contribution literature, such as influence functions [12, 20].
In this work, we propose a class-wise value function that differentiates between the contribution of a data point to its own class and to other classes. Consider the running example in Figure I, increases in-class accuracy, while j decreases in-class accuracy. Intuitively, should receive a higher value than as j could be mislabeled, adversarial, or otherwise noisy instance. Our proposed class value function Ply (S {i]) measures the contribution of data point i based on its class label y = 1, where the accuracy of class is measure of contribution of and the accuracy of class 2 is weighting factor. The definition of this new value function is detailed in section 3. For the example in Figure I, this new class-wise value function measures the contribution as (S U (+)) > vi (S U (i)). A key conceptual message of this paper is to demonstrate that such distinction of in-class and out-of-class accuracy not only leads to desirable theoretical properties for measuring data values in classification (section 4) but also exhibits high efficacy in extensive empirical evaluations (section 5).
we propose new value function that differentiates between in-class and out-of- class contribution for computing Shapley values on classification datasets; 2) we theoretically show that this value function is essentially the unique choice up to some freedom to change a constant that satisfies two desirable properties for data valuation in classification; 3) we perform a systematic evaluation on two benchmark tasks using four classifiers, nine datasets, and three baseline methods. Our results demonstrate that our method outperforms existing methods across almost all experimental conditions; 4) last but not least, we also propose new evaluation task to measure the transferability of data values estimated from different classifiers; using the proposed transfcrability task, we show that Shapley- -based data value estimates can be transferred across classifiers, including transfer to neural models.'
Contributions. 1)
Related work
Data valuation methods. Shapley values are foundational concept in cooperative game theory that ensures fair division of rewards in cooperative games [21]. In machine learning setting, Shapley values have been applied to data valuation, i.e. quantifying the contribution of individual datum [3, 10]. Exact computation of Shapley- data values, however, requires exhaustively retraining and evaluating marginal contributions of every datum using every possible data subset. To circumvent this, Shapley-based data values have been approximated with methods such truncated Monte-Carlo Sampling [3], influence-based based approximations of parameters changes [10]. and federated learning [24]. To our knowledge, our work is the first to consider Shapley values induced by value function that discriminates between in-class and out-of-class accuracy. In section 4, we theoretically analyze the desirable properties of class-wise Shapley values within the context of classification.
Other work that builds upon Shapley-based data values includes using the context of the underlying data distribution to increase valuation stability [4, 15), relaxing the Shapley efficiency axiom to reduce noise [14], and using k-nearest neighbor classifiers over pretrained feature embeddings as surrogates for larger models [11]. Notably, there are alternative methods measure data contribution such as
'Code is available at https:

---

the leave-one-ond method [2], influence functions [12], and reinforcement learning [26], however, these methods have not been proven to share the fairness guarantees of Shapley values.
Applications of Shapley-based data values. Prior work has demonstrated the benefits of using Shapley-basee -bsedd data values in many applications, such mislabeled example detection [23, 3, 14], data selection for transfer learning [18] and active learning [5], and data sharing [22, 7J. The core idea behind these applications is that the Shapley value of training instance indicates its contribution to trained predictive model. By designing a new value function, our method aims to provide more effective estimates of data values and has the potential apply to all of these applications. For real-world applications, we recognize the computational challenge of estimating Shapley values directly from classifiers used in practice (e.g., neural network models). Therefore, we also propose to systematically study the transferability of chapley-based data values across different classifiers, in addition to evaluating on two benchmark evaluation tasks.
Proposed method: CS-SHAPLEY
3.1 Preliminaries
Consider a training dataset T = {(I, [:)] that contains n training instances. Let A denote a classification algorithm and v(S) 2 - IR be a value function that evaluates the value of any subset of data S _ T. For classification tasks, vr-) often considered to be the classification accuracy on a development set D [3, 14, 10, 11]. and v(S) represents the development accuracy (D(D) when the classifier is trained Shapley value ox.T, .A, v) is defined as the average marginal contribution of to every possible subset S (Ta):
S and evaluated on D. For each data point i in the training set, the
Definition 1 (Data Shapley value [21, 3]). Given a value function v(.), the Shapley value di(T, A, v) for any data point defined as
E (() - v(S)
2 (T,
SCTVO
(is
When the dataset T, classification model A, and value function are clear from the context, we simply use Φ; to denote the Shapley value. Shapley values satisfy the following axioms [21]:
Symmetry: if for all S C TV (i,j], v(S u (i)) v(SU {j}), then Φ Øj-
Linearity: dz(U+ + w) = (v( + (w) for value functions e and w.
Null player: if for all S C TTU) v(S) uSSU (i)). then Φ; = 0.
Efficiency: v(T) = ΣiEr 6,.
Prior work usually considers v(-) to be the predictive accuracy on the development set. Recalling the example in Figure 1, this may not be an ideal setting to discriminate between harmful (or noisy) and helpful instances. Notably, this limitation cannot be addressed simply by switching to another development set level metric such as F1, precision, or recall; we will further illustrate this with an example in Appendix B. This key drawback motivates the development of new value function, described in the following section, which has been designed to better diffcrentiate between harmful and helpful instances.
3.2 Class-wise data Shapley
Along the previous lines of discussion, we suggest data for classification may contain implicit, pre-existing coalitions based on class membership. which should be accounted for when evaluating contributions. Motivated by this intuition, we propose a new value function that differentiates between the contribution of adding one instance to its own class vs. to other classes. The key idea behind our design is to use in-class accuracy as the measurement of contribution and out-of-class accuracy as a discounting factor. In this way, we gain the benefits of evaluating value the class level, yet assure we do not assign high value to instances that may be detrimental to the oft-of-class performance.
Class wise value function. Consider the problem of estimating the contribution of data point i, (ai, ve). given subset of training instances S C (i).,. and a development set D. To define a class-wit wise value function, need to partition D into two subsets Dy, and D_ p- Dy, contains the
3

---

development instances with the class label yi and D-g. contains the development instances with the other labels. For multi-class classification, D has all the instances with labels other than yi- Similarly, we have Sy, and S- y, with S = Sy. U S - To measure the contribution of data point i to its own class y; and the other classes -/i we define two separate accuracy numbers, in-class accuracy ag( (Dya) and out-of-class accuracy 0.5 (=
the following
correct predictions Dy, ag
of correct predictions in D=
as( Dy()
(2)
D
[
Note that since ag (Dy.) and (D y. share the same denominator, we have as Dy. +as as( (D), which is the accuracy on the whole development set. With as (D.) and as (D -----... our class-wise value function is defined as
Uy. (Sy, 1 a. (D.)) e v.)
Figure 2 visualizes the contour plot of ug (8) based on different as(Dy,) y. and as (D(D-g.). Between the two variables used in the value function, the significant factor is the in-class accuracy as y..) The effect of the out-of-class accuracy as( D is controlled by the value of a (D,,)) Particularly, when as (D,,) is small, the effect of a.s D y. can be ignored. To better under- stand how this value function works, assume as ,., 0.1, which indicates class y; is difficult to learn. Under this condi- tion, the value of adding an instance in this class is primarily 1.0 from the prediction performance improvement of its own class, rather than that of other classes. This a desirable property of the class-wise value function, which will be formally defined in section 4.
1.0
as(D
0.4
0.2
0
0
0.2
0.4
0.6
as(Dy))
Figure 2: Contour plot of og (S).
Class-wise Shapley values. With the new value function, the Class-wiSe Shapley (CS- (SS-SHAPLRY) value of instance conditioned on any out-of-class "environment" S is defined as
(Sy U (1))S_))) Uy (S IS-w)
S S
(4)
8y, ST WO
SS
To compute the marginal CS- CS-SUIAPLEY value of instance i, we then simply average over all possible environmental data S_ C TT- with equal weight, which leads to our following definition of the Canonical CS-SHAPLEY
4 2
Σ 1-y11- 8-
(5)
We remark that the word "canonical" here refers our simple choice of equal weight -ा for each sampled out-of-class environment S_y.i C T- More generally, one could possibly consider non-canonical and more sophisticated weights, e.g. weights depending the size of S_g.- However, it turns out that the canonical choice in Equation (5) already performs very well in our experiments. Following the principle of Occam's Razor, we thus will stick with this canonical form for the remainder of this paper.
Algorithm. Exactly computing in Equation (5) requires averaging over exponentially many S ver which is computationally prohibitive. Thus we use a relatively small number of subsets S-y. C T for approximating o;
Φi
Σ
( ((A) m.
(6)
K
5(h) CT (EE
In our implementation, we use K 500. Such approximation via samples widely used in previous works [14], and has been proved to give good approximations under structural assumptions about the value function [17, 1]. Although the description above only talks about single instance, the actual implementation of the algorithm is much more efficient, if we compute the values per class. The detailed implementation of our algorithm can found the pseudo-code deferred to Appendix A. At a high level, for any given class label y, the algorithm first samples subset S-y from T-y- Then,

---

for all the examples in class we adopt the truncated Monte Carlo algorithm [3] to estimate the conditional class-wise Shapley values defined in Equation (4). By repeating this procedure K times, the CSS-SHAPLEY value estimation is done by Equation (6). Before switching to another class, we normalize the estimated Shapley values by the in-class accuracy when using the whole training set to satisfy the efficiency axiom.
4 Theoretical justifications of the value function choice
In this section, we carry out theoretical analysis to provide insight and justifications about our approach. We will formally prove that, to fulfil some desirable properties of a class-wise value function, the form that we adopt in Equation (3) is essentially the unique choice, up to the choice of the basis of the exponential function.
To distinguish the accuracy from the in-class and out-of-class development set, we start by assuming that the value function is separable and has the following generic form for any subset of data S S T:
byy (S) = l(as((
g(as(D-z))
where f.g 9 are naturally assumed to be continuous and monotone increasing functions. For normal- ization reasons, without loss of generality, we further assume f(0)g(0) = O. Next, we describe two additional desirable properties of the value function on any development set D:
Property 1: Priority of In-class Accuracy (i.e., as(D,,)). Specifically, for any as (D,,) > 0. we have (( ())())) > f10)g(1).
Property 2: In-class Value Additivity and Out-of-class Weight Discounting, Specifically, for any partitions of in-class development set Dy. Dyn.1 Dy.. and out-of-class development D D v1,1 U D 2. we have
[(as(Daa) glas(D-wa)
(as((Py.)) glas(D-w..l)))) (laas(--aiaa (f(.a(Dyy.2)) ((as(D-ag))) -D.) )) (Dls(DDvv..) (8)
The first property above tries to formalize the intuition that in-class accuracy should be prioritized. Concretely, the value function for getting positive in-class accuracy as as(Dy.) and 0 out-of-class accuracy is no less than getting even perfect out-of-class accuracy but in-class accuracy. The following theorem shows that this property is the underlying reason of the observed contour line in Figure 2. This also justifies the adoption of Property 1.
Theorem 1. Suppose the value function defined Equation (7) satisfies the property of Priority of In-class Accuracy, then no contour lines will intersect the axis of as (D D=g.)) except the special line for flas( (y.))) )) g(as (D))) 0.2
Proof of Theorem 1. By the property of Priority of In-class Accuracy, we know (( > (10)g(1) for any (D, > 0. By taking the limit of letting as Dy.) + 0 in the above inequality. have (Pm, J [as( ())() = ((0)s(0) 0 (0))
Next, we prove the theorem by contradiction. Suppose, for the purpose of contradiction, that there exists a e > such that its contour line intersects the axis of as (Ds(D_g)) at some point (0, for some y < 1. Then we have (()) c by the definition of contour line. This however yields the following contradicting inequalities:
0 0 c<c= ((0) - f(0)g(1) 0
(9)
where the last inequality is proved at the beginning of this proof. Therefore, it must be the case that the only countour line that can intersect the as (D(D...) is the f(a((D,))) g(as( (D.,)) line. This concludes our proof.
The intuition behind the second property is based the role of f and g in the definition. As value measurement on the target class, f [ass( (D,)) )) is expected to be the sum of the value of any two weighting function g, the effect of as (D should
non-overlapped splits of Dy.- In addition, be equivalent to applying the weights from as (D 1. and as (D (-,.)) separately.
Figure 2 is example of such contour lines.
5

---

Theorem 2 shows that our previously defined value function (S) ey. (Sy, 15-1) as (D cas(D-w)) is (essentially) the only choice that satisfies the two desirable properties above. This theoretically justifies our choice of the value function.
Theorem 2. the value function satisfies both Property and above, then it must have the form Vy (S) c'as(Dy) casiD=a) -x) for some constant > 1, C 0.3
Remark 1. The re-scaling constant d in the above theorem will not affect the value much. What truly matters in the function format is the parameter c, which affects how fast the weight function g(-) changes. Our value function choice picked as the natural number
Proof of Theorem 2. The non-trivial part of the proof is to first prove f(0) 0 and g(0) 1, which are not clear in hindsight even given the two properties above. With these two "boundary" conditions, will then be able pin down the concrete format of and g.
Letting as (D)) - O, we first have lim, )- (D,)(
f(0)g(0) 0 which is at least (1)) due to the Property By monotonicity, we have for any y E [0,1]
s(0)g(0) s f(0)s(v) S (1))(1)) 0
(10)
This implies that the inequalities above must all be tight, and thus f(0)g(y) for any y. Since g(v) is not always O, this implies f(0) O.
With f()) O as proven above, we now ready to pin down the format of g(-). Then under the special case that Dy = 6, we have ((as(Dy..)) = f(0) = and thus the second property becomes
F()s(Dy))) g(as((- ((ass())))) (D... (las(D-m.2))
(11)
U D y.2. -m..2. Plugging any Dy, such that (ass( (D))) # 0 into the above for any D-g. = D via1
equality, we thus have
glas(D-v.)) = -g(os)))) g(as((Dvy.))
Since as (s(D..))) = (s(D-v..) + (os(D-v.22) this implies log (D())(D))))) is an additive function. That is, there exists c" such that log (((as(D-v)))) = c'as(D (D- or equivalently, g(as(D-p.)) = e ce"
(Ds(D-v;) for e² >
Finally, we prove the format of f(-). The above proof for g(-) implies g(a)(()) (0) c.. Therefore, under the special case that D D_y. = 0, the second property becomes
((as(Dy.) = flas( + f(as (D..2)..2)
(12)
for any Dy, = D U D Dy. That is, f must be an increasing linear function and thus there is a positive c' such that (as((),)) c' x ,... This concludes the proof of the theorem.
5
Experiments
5.1 Experiment setup
To compare with prior Shapley-based data valuation methods, we adopted most of the experiment setup from prior work, detailed along with other implementation details in Appendix A. In this section, we highlight some important details.
Baseline methods. We compare CS-SHAPLEY against three baselines: Data Shapley with the Truncated Monte Carlo approximation (TMC) [3], Beta Shapley [14], and Leave-One-Out (LOO) [2]. For Beta Shapley, we used the best a and 3 values suggested in the original paper, which were also verified by our preliminary hyperparameter search. Note that another popular baseline method, KNN-Shapley [9], is also essentially covered by applying the data Shapley method to KNN classifiers.
Evaluation tasks. We adopted two benchmark evaluation tasks from prior work: high-value data removal and noisy label detection [3, 14]. In addition, we propose a new evaluation task to quantify
We ignored the trivial situation that é' 0 or 1, which is not interesting.

---

Table Weighted accuracy drop for Logistic Regression and SVM-RBF using CS-SHAPLEY (CS), MMC-Shapley (TMC), Beta Shapley (Beta), and Leave-One-O Onee-uu (LOO).
Logistic Regression
SVM-RBF
Dataset
CS
TMC
Beta
LOO
CS
TMC
Beta
LOO
CIFARIO
0.119
0.108
0.062
0.059
0.114
0.098
0.069
0.089
Click
0.053
0.007
0.017
0.016
0.004
0.004
0.004
0.004
Covertype
0.293
0,250
0.112
0.183
0.193
0.214
0.175
0.193
CPU
0.036
0.022
0.029
0.040
0.028
0.027
0.021
0.004
Diabetes
0.114
0.059
0.038
0.062
0.106
0.037
0.022
-0.002
FMNIST
0.091
0.082
0,038
0.062
0.077
0,048
0.032
0.028
MNIST-2
0.014
0.007
0.010
0.008
0.007
0.007
0.006
0.007
MNIST-10
0.128
0.117
0.064
0.050
0.203
0.247
0.093
0.100
Phoneme
0.154
0.009
0.061
0.072
0.051
0.035
0.035
0.030
070
Prodettn
8003
0.0
0.00
Number of tmil data rsmorved
Number dath
Nunbber of tratn data remsoned
Nimber data rexoone
Nentter of tntc data rersoved
(a) CIFARIO
(b) Click
(c) Covertype
(d) CPU
(c) Diabetes
0.44
Tradlction
Beta Shapley
0503
048
LOO TMC-Shapley
cm
0.40
000
Narder of data vonarcal
Mazdae Denowed
Nuxber Lotta iutaowed
Nuxber Crut date mamooed
Shapley
(f) FMNIST Figure 3: Performance across datasets when removing high-value instances for logistic regression.
(g) MNIST-2
(h) MNIST-10
(i) Phoneme
the transferability of data value estimates across classifiers, to reveal a potential solution for mitigating the computational cost of estimating Shapley values for neural models.
Diabetes, CPU, Click, Covertype, CIFARIO (binarized), FMNIST (binarized), MNIST (multi-class and binarized versions, denoted using -2 and -10, respectively), and Phoneme. When creating data subsets, we keep the original label distribution, instead of creating balanced subsets as in prior work. In addition, for each dataset and evaluation task, we systematically test the data valuation performance on four classifiers: logistic regression, SVM with the RBF kernel, KNN, and a gradient boosting classifier. We also include multi-layer perceptron (MLP) as a target classifier to test the transferability of data values, since computing Shapley values with this classifier prohibitively expensive.
Datasets and classifiers. We use nine benchmark datasets:
Summary of experiments in appendix: Due to page limits, we report representative results in the main content and all additional results in Appendix C.
5.2 High-value data removal
Following the setup in prior work [3], for each valuation method, we gradually remove training instances from the highest value to the lowest value. At each removal step, we retrain the classifier and evaluate predictive performance estimates should be helpful for model performance, so we measure the performance of each method with the accuracy drop following their removal. We follow prior work and plot the accuracy drop for up to 50% train data removed. To further quantify the performance differences observed in the plots, we also introduce novel metric named weighted accuracy drop.
the held-out test data. Training instances with high value
Weighted Accuracy Drop. An effective metric needs to evaluate two components underlying removal performance: 1) the total accuracy drop resulting from each valuation method, and 2) how quickly the drop in accuracy was achieved. Intuitively, the higher the relative value ranking of data point, the more weight its impact model performance should hold. We can therefore define the
7

---

Table Area Under the Curve (AUC) for Logistic Regression and SVM-RBF using CS-SHAPLEY (CS), TMC-Shapley (TMC), Beta Shapley (Beta), and Leave-One-Our (LOO).
Logistic Regression
SVM-RBF
Dataset
CS
TMC
Beta
LOO
CS
TMC
Beta
LOO
CIFARIO
0.450
0.429
0.424
0.275
0.387
0.317
0.321
0.272
Click
0.816
0.689
0.797
0.149
0.855
0.769
0.789
0.200
Covertype
0.706
0.766
0.653
0.179
0.712
0.618
0.600
0.196
CPU
0.785
0.779
0.654
0,207
0.808
0.671
0,516
0.189
Diabetes
0,441
0.355
0,435
0.194
0.412
0.362
0.400
0.210
FMNIST
0.570
0.554
0,552
0.340
0.512
0.382
0.412
0.239
MNIST-2
0.831
0.815
0.806
0,280
0.837
0.663
0.611
0.300
MNIST-10
0.877
0.933
0.845
0.371
0.674
0.747
0,510
0.254
Phoneme 0.575
0.535
0.416
0.222
0.579
0.555
0.496
0.255
weighted accuracy drop (WAD) as the summation of the cumulative accuracy drop at each removal step, weighed by the reciprocal of the removal step (i.e. reciprocal of the rank). Formally, for a training set {[z4, }:)] sorted from the highest to the lowest value we have:
n (D) aT am- (D)- (D)
WADT
where T (1:)) represents the training set with the first i instances removed based on the data valuation (D) (r
rank. When 1, ar (1--11)
or (D) equals the predictive accuracy with the full training set T. In effect, this enables us to assign high importance to the highest-ranken data points while still capturing the overall performance across removals, as depicted the plots.
We report the weighted accuracy drop using logistic regression and SVM with the RBF kernel across datasets in Table 1 and plot the removal performance of logistic regression in Figure 3. As shown, our method outperforms the baseline methods in most of the settings. Similar results are observed for the other two classifiers, as shown in Appendix C. This demonstrates the efficacy of using a value function that discriminates between in-class and out-of-class accuracy. For the SVM- RBF results on the Click dataset, we observe the identical performance across methods. Whereas prior work has used artificially balanced datasets, we performed stratified sampling to maintain the label distribution. In the case of Click, the dataset is highly imbalanced and SVM usually needs additional tricks to work on highly-imbalanced datasets [6].
Results.
5.3
Noisy label detection
To generate noisy training data, we shuffle the labels of random 20% of the training data. We compute value estimates on the noised training sets using each valuation method and then simulate manual inspection by checking data labels from lowest value to highest value. The expectation is that an effective data valuation method will assign low values to mislabeled instances relative to the correctly labeled instances [3]. In our work,
rank- approach to directly evaluate performance and visualize the retrieval results with a precision-recall (PR) curve. In addition, we also compute the Area Under the Curve (AUC) of the PR curve for quantitative results.
Results. We report AUC for logistic regression and SVM-RBF Table 2. Similar to the removal experiments, our method has the best overall performance. We do note slightly weaker performance on the multi-class datasets compared to the removal experiments (see rows for Covertype and MNIST-10 in Table I and Table 2). This could be attributable to the simple sampling strategy of constructing S S_ This suggests that a multi-class setting, our method may benefit from increasing the minimum number of out-of-class samples.
5.4 Transferability of data values
Even with approximation, Shapley values can be computationally expensive to compute for larger models. For example, the experiments in subsection 5.2 had 1:120 runtime ratio between the quickest (Diabetes) and longest (Covertype) running datasets on logistic regression. This would have
8

---

2003
2-12
106
26
Nomber date
120
30
Narsher
130
Nirther data
24
300
Nonther
(a) CIFARIO
(b) Click
(c) Covertype
(d) CPU
(c) Diabetes
29230
0003
0.03
rust
Beta Shapley
0.6
LO0
00:30
TMC-Shapley
040
Number trais incaaned
Nunber truie respoced
CS- CS-Shapley
(f) FMNIST
(g) MNIST-2
(h) MNIST-10
() Phoneme
Figure 4: Performance across datasets when transferring values from logistic regression to MLP for the high-value removal task.
scaled to nearly 4-months to run MLP on Covertype." is therefore of great interest to understand to what extent Shapley values computed with a simple classifier can be transferred to other models, such neural networks. In prior work, Jia et al. [1I] demonstrated the efficacy of a specific case by using a KNN trained over pre-trained embeddings as a surrogate classifter for several target learning models. We generalize this idea and try to answer the question: to what extent can Shapley-based data values computed with various simple classifiers be transferred and applied to other classifiers?
To answer this question, we use each of the four classifiers in subsection 5.2 as the "source" classifiers and evaluate the computed data values with other "target" classífiers on the data removal task, In addition to the four classifiers, we also include an MLP classifier in this evaluation, for which the computational cost of TMC-Shapley was prohibitively large during our preliminary experiments. Specifically, for data values computed with a source classifier on given dataset, at each removal timestep we remove an instance, retrain the target classifier, and evaluate predictive performance as in the original removal experiments. In this experiment, we would like to answer two questions: (1) is there a similar pattern of removal performance on the target classifiers as on the source classifiers; and (2) which source classifier and data valuation method causes the greatest performance drop on target classifiers, as this would indicate high applicability in a real world setting?
Results. Figure 4 shows transfer of logistic regression to MLP across all datasets, and we refer the reader to Figure 3 for the source removal plots. Our results suggest that in general, Shapley-based data values are transferable across classifiers. Specifically, across methods the overall pattern of performance drop from source to target classifier is closely aligned. While these results demonstrate that saapley-based data value estimates are transferable from simpler models even to neural models, they also suggest that the valuation performance on the source classifier can be used as an indicator of how well the performance would be on target classifier. As an implication of this, hyperparameter tuning to achieve better source performance may lead to even better transferability results. We leave this to future work. Additionally, this has implications for being able to gain the benefits of application (such as training data selection) for large neural networks. Further, this transferability may indicate that Shapley values capture some implicit data features that are generally beneficial or harmful to learning models. We leave it to future work to empirically test this. Finally, as a result of this transferability, we also observe that since our method outperformed other methods on the source classifier, CS-SHAN SS-APLLLEY also outperforms when transferred across classifiers, and overall, logistic regression highly-effective as source classifier.
Conclusion
In this work, we propose CS-SHAPLEY, a Shapley value with a new value function that discriminates between training instances' in-class and out-of-class contributions. Our theoretical analysis shows the proposed value function is (essentially) the unique function that satisfies two desirable properties for evaluating data values in classification. Further, our experiments demonstrate the effectiveness of CS-CCS OS-SHAPLEY over existing methods on high-value data removal, noisy label detection, and data
"See Appendix A for information compute resources.

---

value transferability. Currently, the proposed method only works on classification problems. In future work, we will explore the possibility of extending the similar idea regression.
Acknowledgements
This material is based upon work supported by the National Science Foundation under Grant No. SaTC 2124538 and an Amazon Research Award to Yangfeng Ji. We thank the anonymous reviewers for their helpful feedback and suggestions. We also thank Hanjie Chen and Wanyu Du, well the rest of the UVA Information and Language Processing Lab, for engaging discussions about this work.
References
[1] Javier Castro, Daniel Gómez, and Juan Tejada. Polynomial calculation of the shapley value based on sampling. Computers & Operations Research, 36(5):1726-1730 2009.
[2] R Dennis Cook. Detection of influential observation in linear regression. Technometrics, 19(1):
15-18, 1977.
[3] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, pages 2242-2251. PMLR, 2019,
[4] Amirata Ghorbani, Michael Kim, and James Zou. A distributional framework for data valuation. In International Conference on Machine Learning, pages 3535-3544. PMLR, 2020.
[5] Amirata Ghorbani, James Zou, and Andre Esteva. Data shapley valuation for efficient batch active learning. arXiv preprint arXiv:2104.08312 2021.
[6] Yves Grandvalet, Johnny Mariéthoz, and Samy Bengio. A probabilistic interpretation of svms with application unbalanced classification. Advances in Neural Information Processing Systems, 18, 2005.
[7] Dongge Han, Michael Wooldridge, Alex Rogers, Shruti Tople, Olga Ohrimenko, and Sebastian Tschiatschek. Replication-robie mayoff-allocatee for machine learning data markets. arXiv preprint arXiv:2006. 14583, 2020.
[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[9] Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas Spanos, and Dawn Song. Efficient task-specific data valuation for nearest neighbor algorithms. arXiv preprint arXiv: 1008:9908.0000 08619, 2019.
[10] Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J Spanos. Towards efficient data valuation based on the shapley value. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1167-1176. PMLR, 2019.
[11] Ruoxi Jia, Fan Wu, Xuchui Sun, Jiacen Xu, David Dao, Bhavya Kailkhura, Ce Zhang, Bo Li, and Dawn Song. Scalability vs. utility: Do we have to sacrifice one for the other data importance quantification? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8239-82340 8247, 2021.
[12] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions.
In International conference on machine learning, pages 1885-1894. PMLR, 2017.
[13] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[14] Yongchan Kwon and James Zou. Beta shapley: a unified and noise-reduced data valuation framework for machine learning. Proceedings of the 25th International Conference on Artificial Intelligence and Statistics (AISTATS) 2022, 2022.
10

---

[15] Yongchan Kwon, Manuel A Rivas, and James Zou. Efficient computation and analysis of distributional shapley values. In International Conference on Artificial Intelligence and Statistics, pages 793-801. PMLR, 2021.
[16] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 36(11):277-22222 1998.
[17] David Liben-Nowell, Alexa Sharp, Tom Wexler, and Kevin Woods. Computing shapley value supermodular coalitional games. In International Computing and Combinatorics Conference, pages 568-579, Springer, 2012.
[18] Md Rizwan Parvez and Kai- Rei Chang. Evaluating the values of sources in transfer learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 5084-518 2021.
[19] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[20] Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 19920-19930, 2020.
[21] Lloyd S Shapley. A value for n-person games, contributions to the theory of games, 2, 307-317, 1953.
[22] Rachael Hwee Ling Sim, Yehong Zhang, Mun Choon Chan, and Bryan Kian Hsiang Low. Col- laborative machine learning with incentive-aware model rewards. In International Conference on Machine Learning, pages 8927-8936. PMLR, 2020.
[23] Siyi Tang, Amirata Ghorbani, Rikiya Yamashita, Sameer Rehman, Jared A Dunnmon, James Zou, and Daniel L Rubin. Data valuation for medical imaging using shapley value and applica- tion to large-scale chest x-ray dataset. Scientific reports, 11(1):1-9, 2021.
[24] Tianhao Wang, Johannes Rausch, Ce Zhang, Ruoxi Jia, and Dawn Song. A principled approach to data valuation for federated learning. In Federated Learning, pages 153-167. Springer, 2020.
[25] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint 1 2017.
[26] Jinsung Yoon, Sercan Arik, and Tomas Pfister. Data valuation using reinforcement learning. In International Conference on Machine Learning, pages 10842-10851. PMLR, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] For theoretical results, section 4 and Appendix B. For empirical results, see section 5 and Appendix C.
(b) Did you describe the limitations of your work? [Yes] See section 6.
(c) Did you diseuss any potential negative societal impacts of your work? [N/A] We do not anticipate any potential negative societal impacts of this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See section
(b) Did you include complete proofs of theoretical results? [Yes] See section 4 and Appendix B.

---

assets... 3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes] See URL for code repository.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See section and Appendix A.
(c) Did you report error bars (c.g., with respect to the random seed after running ex- periments multiple times)? [Yes] We report standard deviation for all experimental results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix A.
4. If you using existing assets (e.g., code, data, models) curating/releasing
(a) If your work uses existing assets, did you cite the creators? [Yes] All sources provided section and/or Appendix A.
(b) Did you mention the license of the assets? [Yes] License information can be found in Appendix A.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] See URL for code repository.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] For existing data sources, we provide the sources for all datasets through which any consent information, if provided by the cited dataset creators, can be found; we state this in Appendix A.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] We make mention that the datasets we selected do not contain offensive or personally identifiable information in Appendix A.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "sSh2JFBdS7p", "reviewer": "Reviewer_47yJ", "summary": "The authors studied the data valuation problem, pointing out that many existing methods are not able to differentiate in-class and out-of-class contributions (Figure 1). To address this issue, they proposed a new value function in equation (3) and the Class-wise Shapley (CS-SHAPLEY) in equation (4). They also discussed theoretical results, showing the new value function is a unique function (up to constant multiplication; Theorem 2) satisfying two properties. Experimental results on various evaluation tasks show that CS-SHAPLEY outperforms many existing data valuation methods.", "questions": "- Figure 1 presents the motivation for this work, but it needs implementation details.\n    - how do you select two data points `i` and `j` in this experiment? Do you have a sense of why one sample increases class 1 accuracy while another point decreases? \n    - The selection of S: is it based on one particular `S`? or an average of multiple `S`? \n\n- If the CS-SHAPLEY is not a form of the Shapley value, could it mean that the Shapley value is suboptimal? \n\n- (Minors)\n    - One missing parenthesis in equation (1).\n    - In equation 1, seemingly, the right-hand side is not dependent on $\\mathcal{A}$. ", "limitations": "As for the possible limitations, the authors discussed its inapplicability on regression in section 6. However, this is not a critical issue for this paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- (+) An idea of differentiating in-class and out-of-class is conceptually reasonable and novel.\n- (+) The paper is well-written: the main idea and experiments are clearly explained.\n- (+) Extensive experiments have been conducted, showing the power of CS-SHAPLEY.\n\n- (-) It is not clear how the theoretical results are connected to the usefulness of CS-SHAPLEY. \n    - Why are the two properties desirable? It seems that the uniqueness of the value function is purely based on mathematical assumptions (continuity, monotonicity, the structure assumption in equation (7), and properties 1 and 2) that might not be directly essential in the data valuation problem. For instance, what is the meaning of equation (8) in terms of data valuation?\n    - I believe the uniqueness of the value function (Theorem 2) is scientifically correct, but it is not about the CS-SHAPLEY but a value function. Given that CS-SHAPLEY is a specific function of the new value functions, I am wondering why this uniqueness theorem is important for CS-SHAPLEY. \n- (-) The relationship between CS-SHAPLEY and the Shapley value: According to the definition of CS-SHAPLEY in equation (5), I am less sure if CS-SHAPLEY has a form of equation (1) with the Shapley value in (3).\n    - Does CS-SHAPLEY satisfy the axioms in section 3.1 (e.g., Symmetry, Linearity, Null player, and Efficiency)? or is CS-SHAPLEY a type of the Shapley value for a certain cooperative game? If CS-SHAPLEY is not a form of the Shapley value, the use of `SHAPLEY` in `CS-SHAPLEY` might be confusing.\n- (-) In all the current results in section 5, due to the lack of standard error information, it makes it difficult to how significantly better CS-SHAPLEY is than other methods? \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ra9aZC8He-j", "reviewer": "Reviewer_HSku", "summary": "The paper investigates the problem of data valuation using Shapley value-based approximation methods. The authors propose a new value function that incorporates the notion of in-class and out-of-class accuracy as opposed to the baseline approach of using overall accuracy. This approach is justified theoretically and shows improvement in empirical evaluation. The authors also investigate how well data valuation methods transfer across classifiers.", "questions": "* Is it possible to use other metrics (e.g. AUC) for $a_S(D_{y_i})$ and $a_S(D_{-y_i})$?\n* In Figure 4, it looks like a number of datasets have a sharp increase or decrease around where 100 samples are removed. Do you have any thoughts as to why this occurs?", "limitations": "Limitations are adequately discussed in Section 6.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n* The proposed value function is a novel approach that improves performance over existing methods. Comparisons are made to 3 competing data valuation methods over 8 different datasets.\n* The proposed method is well-motivated and intuitive. The theoretical justifications are sound.\n\n**Weaknesses:**\n* The proposed method is limited to classification tasks, and only the accuracy metric is investigated for the value function.\n* No sensitivity analysis or justification of choice in c, c' parameters. Adding this analysis would improve the paper.\n\n\n**Miscellaneous Minor Issues / Suggestions**\n* Eq. 1 is missing a right parenthesis: $v(S \\cup \\{ i \\})$\"\n* Line 122 \"For multi-class classification, $D_{y_i}$ has all the instances with labels other than $y_i$.\" I think this should be $D_{-y_i}$?\n* Theorem 2 has an extra right parenthesis in $v_{y_i}(S)$\n* It would be helpful if the shaded area in the plots (Figure 3,4) representing standard deviation was defined in the table caption.\n* Error bars / confidence intervals in the table results would be helpful as a number of the results are very close.\n* It looks like setting $c=1$ and $c' = 1$ would recover the value function for the standard Shapley value. It might be interesting to see whether you can interpolate between the results of standard and classwise Shapley values depending on choice of $c, c'$.\n\n**Summary:**  \nIn general, the proposed method is a novel and intuitive approach which is justified both theoretically and empirically. The paper is also well-written and well-motivated. There are a some areas for improvement, such as extending to metrics other than accuracy and additional sensitivity analysis, however the paper in its current form seems sufficient for acceptance.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PG36nGM9OyL", "reviewer": "Reviewer_DWEJ", "summary": "This paper proposes Class-wise Shapley values as a new valuation method for data in classification tasks by designing a value function that _separately_ considers the in-class accuracy and out-of-class accuracies. The value function is shown to uniquely (up to the basis of the exponential function) satisfies two new properties concerning the value of data in classification tasks (w.r.t. in-class and out-of-class accuracies). The paper also provides experimental comparison with some existing data valuation methods on two prior evaluation tasks (high-value data removal and noisy label detection). In addition, the paper includes experimental observations about the transferrability (acorss classifiers) of Shapley-based data values.", "questions": "Clarification questions\n- Is there a reference for using the term _development set_ or is there a reason why validation/hold-out/test set is not suitable? The latter options are more widely used and more receptive.\n- The example give in Figure.1. Specifically, \"Both points belong to class 1 and produce the same overall development accuracy change.\" How to see the _same_? Or what is the overall development accuracy change?\n- In line 117, \"we do not assign high value to instances that may be detrimental to the out-of-class performance\". It seems to imply some other methods do this, is it true?\n- In line 122, \"For multi-class classification, $D_{y_i}$ has all the instances with labels other than $y_i$.\" Should it be $D_{-y_i}$?\n- In line 142, \"CS-Shapley value of instance $i$ _conditioned_ on any out-of-class environment $S_{-y_i}$ ...\" How is this `conditioned' reflected in Equ.(4)? Or how is $S_{-y_i}$ used in Equ.(4)?\n- In line 145, \"environmental data $S_{-y} \\subseteq T_{-y_i}$ ...\" Should it be $S_{-y_i}$?\n- In line 152, \"Exactly computing $\\phi_i$ in Equation (5) requires averaging over exponentially many $S_{-y_i}$,...\" The exponentially many $S_{-y_i}$ is clear from Equ.(5). The question is, does Equ.(4) also involve exponentially many $S_{y_i}$? If so, does it mean that Equ.(5) somehow contains \"nested\" exponential complexities? \n- In lines 236-237, \"When creating data subsets, we keep the original label distribution, instead of creating balanced subsets as in prior work.\" It seems imbalanced data are not an issue, why? In particular, perhaps CS-Shapley can address this, but how about other baselines?\n- About the introduced metric _Weighted Accuracy Drop_ (WAD)\n    - Is there a comparison with the previous metrics to help understand the difference? In particular, it would be helpful to use this comparison to show WAD is a good choice.\n    - The reciprocal of the rank is used as the weights. What about other monotonically decreasing functions?\n- In lines 276-277, \"CS-Shapley may benefit from increasing the minimum number of out-of-class samples?\" What does \"increasing the minimum number of out-of-class samples?\" Does it mean increasing $S$ or $D$ and what are out-of-class samples?\n\n\n\n\n\n\n\n", "limitations": "The authors have mentioned this work is limited to classification and defer the extension to regression to future work.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The writing of the paper is clear and concise and the theoretical derivations seem correct with suitably defined notations. The perspective of explictly considering in-class and out-of-class accuracies for data values in classification is new (to my knowledge) and is also compared with some intuitive/naive choices such as F1, precison or recall in Appendix. However, the motivation for separately considering in-class vs. out-of-class accuracies can be made clearer/stronger, i.e., _the why_. Following this, the motivation for the two additional properties (which essentially fix the value function) can be made clearer, too, i.e., _the how_. After proposing the CS-Shapley, which seems very computationally costly to calculate, it would be better to describe an approximation algorithm with theoretical guarantees so as to provide a sense for how to apply it and what error to expect in practice. For experiments, Sec. 5.4 presents empirical observations about Shapley-based data values in general, and its relevance specifically to CS-Shapley can be made more explicit.\n\n\nStrengths\n- The writing is clear and the theoretical derivations seem sound.\n- The perspective of separately considering in-class and out-of-class accuracies is new.\n- The overall presentaion is good in terms of organization and the authors included examples and plots to help explain their ideas.\n- The experiments included several known baselines and are performed over various datasets and classifiers.\n\nWeaknesses\n- _The why_ The motivation for separately considering in-class vs. out-of-class accuracies is not so clear. In lines 111-113, \"Along the previous lines of discussion, we suggest data for classification may contain implicit pre-existing coalitions based on class membership, which should be accounted for when evaluating contributions.\"\n    - Speicfically, what are such implicit pre-existing coalitions were not so clear. Specific use-cases or practical application scenarios would help.\n    - The motivation/reason for why such coalitions should be accounted for is not so clear. Empirical evidence or references that justify this would be useful.\n- _The how_  The motivation for the specific way of separating in-class vs. out-of-class accuracies, namely Property 1. is not so clear. The authors argue \"in-class accuracy should be prioritized.\" (line 181) In particular, in lines 182-183, \"Concretely, the value function for getting positive in-class accuracy $a_S(D_{y_i})$ and $0$ out-of-class accuracy is no less than getting even perfect out-of-class accuracy but $0$ in-class accuracy.\" The justification for this statement can be made stronger. Should this statement be true _all the time_ for _all possible cases_? \n- The practical applicability of CS-Shapley can be improved if paried with a corresponding approximation algorithm with theoretical guarantees. The cited previous works such as Data Shapley [3], Beta Shapley [14] and KNN Shapley [9] all paired their proposed/considered Shapley value with an approximation algorithm (including theoretical guarantees) which specifically exploited the defined value (e.g., gradient Shapley) and/or machine learning structural assumptions (e.g., truncation, KNN).\n- The relevance for Sec. 5.4 (transferrability of data values) to CS-Shapley or to separately considering in-class vs. out-of-class accuracies is not so clear. It seems the results are general to Shapley-based data values, instead of CS-Shapley.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MsIyF6ujVXM", "reviewer": "Reviewer_kNhs", "summary": "\nClass-wise Shapley Values (CS-SHAPLEY) is a data valuation algorithm for binary and multiclass classification algorithms.\n\nCS-SHAPLEY distinguishes between training instances’ in-class and out-of-class contributions. Then, the algorithm combines the 2 measurements into one value.\n\nCS-SHAPLEY is the unique function that satisfies 2 properties for evaluating data values in classification:\nProperty 1 (Priority of In-class Accuracy) makes train points that improve their in-class accuracy more important than those that improve the out-of-class accuracy.\nProperty 2 (In-class Value Additivity and Out-of-class Weight Discounting) makes the data values and weights additive.\n\nCS-SHAPLEY takes too long to compute. Its approximation is compared against Truncated Monte-Carlo (TMC), Beta Shapley, and Leave-One-Out and did perform better in most of the baseline tasks.\n", "questions": "Figure 3: how do you explain that other methods like LOO are more efficient at the beginning (first 20 train points removed)? Have you thought about an ensemble method?\n\n\nConcerning the sub-plot \"(e) Diabetes,\" why is it limited to less than 75 data points?\n\n\n# Relation To Prior Work\nAuthors did consider popular contributions like Data-Shapley, LOO, and Beta Shapley.\nDid you consider G-Shapley? (see [3])\n\n# Additional Feedback\n\nDid you consider the effect of hyperparameter tuning of the ML algorithm on the data valuation? i.e., the minimum starting score of the considered tasks is 0.68. Would CS-Shapley be the best if the starting accuracy was 0.3?", "limitations": "no negative societal impact", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nCS-SHAPLEY is an original approach to valuate train data points. It is the only one distinguishing between in-class and out-of-class accuracy.\nThe authors propose a method that is analyzed using both axioms/properties and practical evaluation tasks (benchmark). \nCS-SHAPLEY performs well on various tabular data.\n\nAuthors provide a promising metric to objectively evaluate data valuation algorithms on the task of removing high-value instances. Rather than visually comparing curves, the authors propose a one value score that considers the order in which data points were removed. Nevertheless, this metric seems very close to the AUC. A small comparison between both would be interesting.\n\n\n# Weaknesses\n\nHigh-value data removal might not be sufficient to confirm that the proposed method is performing better: As explained by the authors, data removal might cause imbalance and a faster drop in performance. Therefore, it would be better to complement the experiment with the opposite task (adding data points)\n\n\n\nThe authors did not reuse the same datasets and tasks used in prior work. Therefore it is still difficult to confirm that this method is better. Please include, for example, the Skin Cancer dataset used in the TMC-Shapley paper [3].\n\n# Correctness\n\nRandom data removal is a good baseline to check if, eventually, all algorithms are failing a specific task. Please include it to confirm that CS-SHAPLEY is better. \n\n# Clarity\n\nLine 92: the definition of v(S) and a_S(D) are condensed into one sentence, which makes it ambiguous.\n\nFigure 3: adding an x-axis with the percentage of data removed would help better understand the real contribution of these deleted data points. a more straightforward fix is to add the total train set size to the x-axis title \nPlease add to the y-axis an arrow down to express that \"lower is better\".\n\nThe abbreviation TMC was defined but never used. Instead, the authors did use \"Data-Shapley\". Please use TMC-Shapley as proposed in the original paper hoping to make it clearer for the readers.\n\n\n\n# Additional Feedback\n \nFigure 3: you could remove the redundant y-axis's title.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept