Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper investigates the problem of data valuation using Shapley value-based approximation methods. The authors propose a new value function that incorporates the notion of in-class and out-of-class accuracy as opposed to the baseline approach of using overall accuracy. This approach is justified theoretically and shows improvement in empirical evaluation. The authors also investigate how well data valuation methods transfer across classifiers.
strengths_and_weaknesses: **Strengths:**
* The proposed value function is a novel approach that improves performance over existing methods. Comparisons are made to 3 competing data valuation methods over 8 different datasets.
* The proposed method is well-motivated and intuitive. The theoretical justifications are sound.

**Weaknesses:**
* The proposed method is limited to classification tasks, and only the accuracy metric is investigated for the value function.
* No sensitivity analysis or justification of choice in c, c' parameters. Adding this analysis would improve the paper.


**Miscellaneous Minor Issues / Suggestions**
* Eq. 1 is missing a right parenthesis: $v(S \cup \{ i \})$"
* Line 122 "For multi-class classification, $D_{y_i}$ has all the instances with labels other than $y_i$." I think this should be $D_{-y_i}$?
* Theorem 2 has an extra right parenthesis in $v_{y_i}(S)$
* It would be helpful if the shaded area in the plots (Figure 3,4) representing standard deviation was defined in the table caption.
* Error bars / confidence intervals in the table results would be helpful as a number of the results are very close.
* It looks like setting $c=1$ and $c' = 1$ would recover the value function for the standard Shapley value. It might be interesting to see whether you can interpolate between the results of standard and classwise Shapley values depending on choice of $c, c'$.

**Summary:**  
In general, the proposed method is a novel and intuitive approach which is justified both theoretically and empirically. The paper is also well-written and well-motivated. There are a some areas for improvement, such as extending to metrics other than accuracy and additional sensitivity analysis, however the paper in its current form seems sufficient for acceptance.

questions: * Is it possible to use other metrics (e.g. AUC) for $a_S(D_{y_i})$ and $a_S(D_{-y_i})$?
* In Figure 4, it looks like a number of datasets have a sharp increase or decrease around where 100 samples are removed. Do you have any thoughts as to why this occurs?
limitations: Limitations are adequately discussed in Section 6.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: 
Class-wise Shapley Values (CS-SHAPLEY) is a data valuation algorithm for binary and multiclass classification algorithms.

CS-SHAPLEY distinguishes between training instances’ in-class and out-of-class contributions. Then, the algorithm combines the 2 measurements into one value.

CS-SHAPLEY is the unique function that satisfies 2 properties for evaluating data values in classification:
Property 1 (Priority of In-class Accuracy) makes train points that improve their in-class accuracy more important than those that improve the out-of-class accuracy.
Property 2 (In-class Value Additivity and Out-of-class Weight Discounting) makes the data values and weights additive.

CS-SHAPLEY takes too long to compute. Its approximation is compared against Truncated Monte-Carlo (TMC), Beta Shapley, and Leave-One-Out and did perform better in most of the baseline tasks.

strengths_and_weaknesses: # Strengths

CS-SHAPLEY is an original approach to valuate train data points. It is the only one distinguishing between in-class and out-of-class accuracy.
The authors propose a method that is analyzed using both axioms/properties and practical evaluation tasks (benchmark). 
CS-SHAPLEY performs well on various tabular data.

Authors provide a promising metric to objectively evaluate data valuation algorithms on the task of removing high-value instances. Rather than visually comparing curves, the authors propose a one value score that considers the order in which data points were removed. Nevertheless, this metric seems very close to the AUC. A small comparison between both would be interesting.


# Weaknesses

High-value data removal might not be sufficient to confirm that the proposed method is performing better: As explained by the authors, data removal might cause imbalance and a faster drop in performance. Therefore, it would be better to complement the experiment with the opposite task (adding data points)



The authors did not reuse the same datasets and tasks used in prior work. Therefore it is still difficult to confirm that this method is better. Please include, for example, the Skin Cancer dataset used in the TMC-Shapley paper [3].

# Correctness

Random data removal is a good baseline to check if, eventually, all algorithms are failing a specific task. Please include it to confirm that CS-SHAPLEY is better. 

# Clarity

Line 92: the definition of v(S) and a_S(D) are condensed into one sentence, which makes it ambiguous.

Figure 3: adding an x-axis with the percentage of data removed would help better understand the real contribution of these deleted data points. a more straightforward fix is to add the total train set size to the x-axis title 
Please add to the y-axis an arrow down to express that "lower is better".

The abbreviation TMC was defined but never used. Instead, the authors did use "Data-Shapley". Please use TMC-Shapley as proposed in the original paper hoping to make it clearer for the readers.



# Additional Feedback
 
Figure 3: you could remove the redundant y-axis's title.


questions: Figure 3: how do you explain that other methods like LOO are more efficient at the beginning (first 20 train points removed)? Have you thought about an ensemble method?


Concerning the sub-plot "(e) Diabetes," why is it limited to less than 75 data points?


# Relation To Prior Work
Authors did consider popular contributions like Data-Shapley, LOO, and Beta Shapley.
Did you consider G-Shapley? (see [3])

# Additional Feedback

Did you consider the effect of hyperparameter tuning of the ML algorithm on the data valuation? i.e., the minimum starting score of the considered tasks is 0.68. Would CS-Shapley be the best if the starting accuracy was 0.3?
limitations: no negative societal impact
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes Class-wise Shapley values as a new valuation method for data in classification tasks by designing a value function that _separately_ considers the in-class accuracy and out-of-class accuracies. The value function is shown to uniquely (up to the basis of the exponential function) satisfies two new properties concerning the value of data in classification tasks (w.r.t. in-class and out-of-class accuracies). The paper also provides experimental comparison with some existing data valuation methods on two prior evaluation tasks (high-value data removal and noisy label detection). In addition, the paper includes experimental observations about the transferrability (acorss classifiers) of Shapley-based data values.
strengths_and_weaknesses: The writing of the paper is clear and concise and the theoretical derivations seem correct with suitably defined notations. The perspective of explictly considering in-class and out-of-class accuracies for data values in classification is new (to my knowledge) and is also compared with some intuitive/naive choices such as F1, precison or recall in Appendix. However, the motivation for separately considering in-class vs. out-of-class accuracies can be made clearer/stronger, i.e., _the why_. Following this, the motivation for the two additional properties (which essentially fix the value function) can be made clearer, too, i.e., _the how_. After proposing the CS-Shapley, which seems very computationally costly to calculate, it would be better to describe an approximation algorithm with theoretical guarantees so as to provide a sense for how to apply it and what error to expect in practice. For experiments, Sec. 5.4 presents empirical observations about Shapley-based data values in general, and its relevance specifically to CS-Shapley can be made more explicit.


Strengths
- The writing is clear and the theoretical derivations seem sound.
- The perspective of separately considering in-class and out-of-class accuracies is new.
- The overall presentaion is good in terms of organization and the authors included examples and plots to help explain their ideas.
- The experiments included several known baselines and are performed over various datasets and classifiers.

Weaknesses
- _The why_ The motivation for separately considering in-class vs. out-of-class accuracies is not so clear. In lines 111-113, "Along the previous lines of discussion, we suggest data for classification may contain implicit pre-existing coalitions based on class membership, which should be accounted for when evaluating contributions."
    - Speicfically, what are such implicit pre-existing coalitions were not so clear. Specific use-cases or practical application scenarios would help.
    - The motivation/reason for why such coalitions should be accounted for is not so clear. Empirical evidence or references that justify this would be useful.
- _The how_  The motivation for the specific way of separating in-class vs. out-of-class accuracies, namely Property 1. is not so clear. The authors argue "in-class accuracy should be prioritized." (line 181) In particular, in lines 182-183, "Concretely, the value function for getting positive in-class accuracy $a_S(D_{y_i})$ and $0$ out-of-class accuracy is no less than getting even perfect out-of-class accuracy but $0$ in-class accuracy." The justification for this statement can be made stronger. Should this statement be true _all the time_ for _all possible cases_? 
- The practical applicability of CS-Shapley can be improved if paried with a corresponding approximation algorithm with theoretical guarantees. The cited previous works such as Data Shapley [3], Beta Shapley [14] and KNN Shapley [9] all paired their proposed/considered Shapley value with an approximation algorithm (including theoretical guarantees) which specifically exploited the defined value (e.g., gradient Shapley) and/or machine learning structural assumptions (e.g., truncation, KNN).
- The relevance for Sec. 5.4 (transferrability of data values) to CS-Shapley or to separately considering in-class vs. out-of-class accuracies is not so clear. It seems the results are general to Shapley-based data values, instead of CS-Shapley.

questions: Clarification questions
- Is there a reference for using the term _development set_ or is there a reason why validation/hold-out/test set is not suitable? The latter options are more widely used and more receptive.
- The example give in Figure.1. Specifically, "Both points belong to class 1 and produce the same overall development accuracy change." How to see the _same_? Or what is the overall development accuracy change?
- In line 117, "we do not assign high value to instances that may be detrimental to the out-of-class performance". It seems to imply some other methods do this, is it true?
- In line 122, "For multi-class classification, $D_{y_i}$ has all the instances with labels other than $y_i$." Should it be $D_{-y_i}$?
- In line 142, "CS-Shapley value of instance $i$ _conditioned_ on any out-of-class environment $S_{-y_i}$ ..." How is this `conditioned' reflected in Equ.(4)? Or how is $S_{-y_i}$ used in Equ.(4)?
- In line 145, "environmental data $S_{-y} \subseteq T_{-y_i}$ ..." Should it be $S_{-y_i}$?
- In line 152, "Exactly computing $\phi_i$ in Equation (5) requires averaging over exponentially many $S_{-y_i}$,..." The exponentially many $S_{-y_i}$ is clear from Equ.(5). The question is, does Equ.(4) also involve exponentially many $S_{y_i}$? If so, does it mean that Equ.(5) somehow contains "nested" exponential complexities? 
- In lines 236-237, "When creating data subsets, we keep the original label distribution, instead of creating balanced subsets as in prior work." It seems imbalanced data are not an issue, why? In particular, perhaps CS-Shapley can address this, but how about other baselines?
- About the introduced metric _Weighted Accuracy Drop_ (WAD)
    - Is there a comparison with the previous metrics to help understand the difference? In particular, it would be helpful to use this comparison to show WAD is a good choice.
    - The reciprocal of the rank is used as the weights. What about other monotonically decreasing functions?
- In lines 276-277, "CS-Shapley may benefit from increasing the minimum number of out-of-class samples?" What does "increasing the minimum number of out-of-class samples?" Does it mean increasing $S$ or $D$ and what are out-of-class samples?








limitations: The authors have mentioned this work is limited to classification and defer the extension to regression to future work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors studied the data valuation problem, pointing out that many existing methods are not able to differentiate in-class and out-of-class contributions (Figure 1). To address this issue, they proposed a new value function in equation (3) and the Class-wise Shapley (CS-SHAPLEY) in equation (4). They also discussed theoretical results, showing the new value function is a unique function (up to constant multiplication; Theorem 2) satisfying two properties. Experimental results on various evaluation tasks show that CS-SHAPLEY outperforms many existing data valuation methods.
strengths_and_weaknesses: - (+) An idea of differentiating in-class and out-of-class is conceptually reasonable and novel.
- (+) The paper is well-written: the main idea and experiments are clearly explained.
- (+) Extensive experiments have been conducted, showing the power of CS-SHAPLEY.

- (-) It is not clear how the theoretical results are connected to the usefulness of CS-SHAPLEY. 
    - Why are the two properties desirable? It seems that the uniqueness of the value function is purely based on mathematical assumptions (continuity, monotonicity, the structure assumption in equation (7), and properties 1 and 2) that might not be directly essential in the data valuation problem. For instance, what is the meaning of equation (8) in terms of data valuation?
    - I believe the uniqueness of the value function (Theorem 2) is scientifically correct, but it is not about the CS-SHAPLEY but a value function. Given that CS-SHAPLEY is a specific function of the new value functions, I am wondering why this uniqueness theorem is important for CS-SHAPLEY. 
- (-) The relationship between CS-SHAPLEY and the Shapley value: According to the definition of CS-SHAPLEY in equation (5), I am less sure if CS-SHAPLEY has a form of equation (1) with the Shapley value in (3).
    - Does CS-SHAPLEY satisfy the axioms in section 3.1 (e.g., Symmetry, Linearity, Null player, and Efficiency)? or is CS-SHAPLEY a type of the Shapley value for a certain cooperative game? If CS-SHAPLEY is not a form of the Shapley value, the use of `SHAPLEY` in `CS-SHAPLEY` might be confusing.
- (-) In all the current results in section 5, due to the lack of standard error information, it makes it difficult to how significantly better CS-SHAPLEY is than other methods? 


questions: - Figure 1 presents the motivation for this work, but it needs implementation details.
    - how do you select two data points `i` and `j` in this experiment? Do you have a sense of why one sample increases class 1 accuracy while another point decreases? 
    - The selection of S: is it based on one particular `S`? or an average of multiple `S`? 

- If the CS-SHAPLEY is not a form of the Shapley value, could it mean that the Shapley value is suboptimal? 

- (Minors)
    - One missing parenthesis in equation (1).
    - In equation 1, seemingly, the right-hand side is not dependent on $\mathcal{A}$. 
limitations: As for the possible limitations, the authors discussed its inapplicability on regression in section 6. However, this is not a critical issue for this paper.
review_title: null

Meta-review: metareview: The reviewers agree that the paper makes a potentially interesting contribution and clearly holds promise, but also that it needs another round of revision. The authors' rebuttal was very much appreciated and could indeed clarify some misunderstandings (e.g., a single vs. several cooperative games), though it also left some questions open (and even gave rise to a few others). Eventually, the reviewers' reservations regarding the proper formalisation of the problem, the meaning and interpretation of the approach (e.g., meaning of comparing many Shapley values from different cooperative games), as well as the experimental validation couldn't be completely resolved. 
confidence: Certain
Decision: Accept