Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper uses a synthetic two-dimensional dataset to visualize the importance of different data points on machine learning model performance. In particular, they used a multi-layer perceptron as the model, and they used four different schemes by which to measure the importance of individual data points. Not surprisingly, they show that regions of data that are misclassified tend to have relatively high importance in determining model performance. However, there are notable differences among the different schemes in their performances. This paper discusses the nature of those differences.
main_review: Strengths: This paper compares several methods by which to assess the value of data points for classification.

Weaknesses:
1. The results that are surprising or are contradictory with other papers in the area (e.g., adding data points to larger training sets leads to larger changes in the model, only misclassified points close to the boundary are important) are not explained in any detail. Just noting the disagreement is not helpful---an explanation, or at least a hypothesis, of why there is a disagreement is needed to give some useful insight.
2. Since Support Vector Machines are a well-established and high performing model that assesses the value of data points, comparison with SVM should be done. I realize that the goal here is to compare different methods for assessing the value of points for a particular classifier (multi-layer perceptron). However, I still feel that a comparison to SVM is valuable since it is so well established.
summary_of_the_review: This paper compares methods by which to assess the value of data points for classification. However, insufficient analysis of surprising findings and a lack of comparison to a well-established method that assesses the value of data points (SVMs) weaken the contribution of this paper significantly.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents an empirical study of different data valuation techniques based on 2D synthetic data and discusses the findings.
main_review: Pros:

- The topic is of critical importance to the machine learning community. 
- The paper is well-written and easy to follow.

Cons:
- The paper lacks theoretical justification for the findings.
- The experiments are only performed on MLP with a single layer and low-dimensional synthetic data. It is questionable whether the findings from the experiments contribute generalizable knowledge to deeper neural networks, higher-dimensional, real-world data, which is of more interest in practice.
- The paper presents a comprehensive overview of data valuation ideas. For Shapley value, though, the authors only discuss one specific algorithm, DataShapley, to calculate the Shapley value. This algorithm is a heuristic for calculating Shapley and there are no theoretical guarantees for the approximation error. Hence, it remains a question whether the findings presented in the paper are property of Shapley value or property of the DataShapley heuristic. In particular, there exist efficient algorithms for calculating Shapley value with provable error guarantees, e.g., permutation sampling and group testing in [1], for the scale of the experiments conducted in the paper. It'll be great to include these algorithms in the comparison.

[1] https://arxiv.org/abs/1902.10275 
summary_of_the_review: The paper studies a very timely and important topic. Yet, the main concern is that the paper lacks formal justification and the experiments are done in a synthetic setting. It is questionable how generalizable the findings are beyond the specific synthetic data and the specific model considered in the paper.

--
[post-rebuttal] Thanks to the authors for the response. This is an interesting paper but my concerns about the lack of theoretical justification and experiments of modern DNNs as well as missing baselines still remain. Hence I will maintain my score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a new method for identifying important points in a dataset given the task of classification. The paper introduces a new valuation function. The paper takes into consideration other methods that deal with the same problem and reevaluated them using the new scoring function. The scoring function focuses on the effect points have on the decision boundary. The results show that the most valuable points are not the ones close to the boundary but the misclassified ones. The result is interesting but not surprising. 
main_review: The paper is pretty straightforward. It modifies four different well-known techniques for evaluating data. The biggest weakness of the paper is that it is based on a single small size 2-dimensional synthetic dataset. The result and the conclusions are based on empirical visualizations. The results are very preliminary and limited to make any secure conclusions. The method is not practical at all for real data, since it is difficult to visualize. 

The intuition and the methodology that the authors are using are in the right direction, but it is half-baked. The authors need to find a way to work it out in higher dimensions and in more real-world scenarios. One direction that the authors could explore is to measure the value of the points in terms of dollars (or any other coin) and not in terms of accuracy.

The appendix is not very useful. There are many more experiments mentioned there, but it makes it very hard to follow the paper. There are way too many plots. It would make more sense to create some measures over the plots and present them on a table.
summary_of_the_review: The approach of the paper is interesting, but the paper needs a lot of work to be a full conference paper. It makes more sense to present it at a workshop and get more feedback. As mentioned in the main review the authors present some preliminary evidence for their findings, but more experiments are needed in order to have a strong claim.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduce a novel framework to assess data points importance as the change of the decision boundary for a model with respect to added samples. They demonstrate their claim by training a MLP model on a synthetic 2D data set in several modalities and show the criticality of misclassified points.

main_review: The paper provides a relevant contribution on a fairly recent ML area concerning the importance of data points.
The authors show, on a simple and controlled 2D synthetic dataset, different situations and different impact of data points on the considered predictive model.
They also provide a comprehensive definition of importance for a data points, separated into two aspects that they show to be quite similar for noise-free datasets.
Throughout a suite of experiments by MLP in different setups they found that most methods agree on miss-classified points to be important.
A number of interesting figures are included showing the different impact of the added data points to the boundary shapes: however, a few tables reporting the actual numbers to quantitatively support the authors' claims would be recommended.
As the authors acknowledge, the limitation of tested models (MLPs) and datasets make the contribution just a proof of concept to be used as a basis for a larger study to lay down the foundation of a grounded theory of data importance.

summary_of_the_review: Paper is well written, readable by a wide audience and tackling a quite relevant problem.
Experiments are convincing and supporting the authors' claims, although limited to a single model type and simple 2d synthetic dataset.
Overall, the authors contribute novel insights on a interesting ML problem,.
review_title: null

Meta-review: The reviews are of adequate quality. The responses by the authors are commendable, but ICLR is selective and reviewers continue to believe that more experiments and more rigorous analysis are needed.
Decision: Reject