OpenDataVal: a Unified Benchmark for Data Valuation
Authors: Kevin Fu Jiang
Weixin Liang
James Zou
Yongchan Kwon
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Data valuation
Benchmark
Data-centric AI
TL;DR: We propose OpenDataVal, an easy-to-use and unified benchmark framework for data valuation.

Abstract: Assessing the quality and impact of individual data points is critical for improving model performance and mitigating undesirable biases within the training dataset. Several data valuation algorithms have been proposed to quantify data quality, however, there lacks a systemic and standardized benchmarking system for data valuation. In this paper, we introduce *OpenDataVal*, an easy-to-use and unified benchmark framework that empowers researchers and practitioners to apply and compare various data valuation algorithms. *OpenDataVal* provides an integrated environment that includes (i) a diverse collection of image, natural language, and tabular datasets, (ii) implementations of eleven different state-of-the-art data valuation algorithms, and (iii) a prediction model API that can import any models in scikit-learn. Furthermore, we propose four downstream machine learning tasks for evaluating the quality of data values. We perform benchmarking analysis using *OpenDataVal*, quantifying and comparing the efficacy of state-of-the-art data valuation approaches. We find that no single algorithm performs uniformly best across all tasks, and an appropriate algorithm should be employed for a user's downstream task. *OpenDataVal* is publicly available at https://opendataval.github.io with comprehensive documentation. Furthermore, we provide a leaderboard where researchers can evaluate the effectiveness of their own data valuation algorithms.

OpenData Val: a Unified Benchmark for Data Valuation
Kevin Fu Jiang" Columbia University
Weixin Liang" Stanford University
James Zou Stanford University
Yongchan Kwon Columbia University
Abstract
Assessing the quality and impact of individual data points critical for improving model performance and mitigating undesirable biases within the training dataset. Several data valuation algorithms have been proposed to quantify data quality, however, there lacks systemic and standardized benchmarking system for data valuation. this paper, we introduce OpenDataVal, an easy-to-use and unified benchmark framework that empowers researchers and practitioners to apply and compare various data valuation algorithms. OpenDataVal provides an integrated environment that includes (i) diverse collection of image, natural language, and tabular datasets, (ii) implementations of eleven different state-of-ther -hee--rr- data valu- ation algorithms, and (iii) prediction model API that import any models in scikit-learn. Furthermore, we propose four downstream machine learning tasks for evaluating the quality of data values. We perform benchmarking analysis using OpenDataVal, quantifying and comparing the efficacy of statc-of-the-arr data valu- ation approaches. We find that no single algorithm performs uniformly best across all tasks, and appropriate algorithm should be employed for a user's 's downstream task. OpenDataVal publicly available at /tttps::/opendatavaal github. io with comprehensive documentation. Furthermore, we provide a leaderboard where researchers can evaluate the effectiveness their own data valuation algorithms.
Introduction
Real-world data often exhibit heterogeneity and noise they collected from wide range of sources and are susceptible to measurement and annotation errors [9] 21). When such data are incorporated into model development, they introduce undesirable biases and potentially hinder model performance. Managing and addressing these challenges crucial to ensure the reliability and accuracy of the insights derived from real-world data. As such, it becoming increasingly important to understand and evaluate the intrinsic properties of data, such as data quality, data bias, and their influences on the model training process [14 20). From all these motivations, data valuation has emerged as a principled evaluation framework that aims to quantify the impact of individual data points on model predictions or model performance.
There have been substantial recent efforts developing different data valuation methods, and they have demonstrated promising outcomes in various real-world applications. One widely used data valuation method is DataShapley [7], which is based on the Shapley value from game theory [31]. This method has been applied detect low-quality chest X-ray image data [35] and design data
"Co-first author
CCoresponding author (e-mail: yk301 columbaaa..
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Image
Label
Value
Datasets
Dog
0.99
Cat
0.97
Data Valuation
opendataval
Algorithms
Pradiction
Cat 0.02
Models
Dog 0.01
Figure 1: OpenDataVal is an open-source, easy-to-use and unified benchmark framework for data valuation. It contains a diverse collection of datasets, implementations of eleven state-of-the-ar data valuation algorithms, and variety of modern prediction models. In addition, OpenDataVal provides several downstream tasks for evaluating the quality of data valuation algorithms.
marketplace [[]. BetaShapley generalizes DataShapley by relaxing the efficiency axiom in the Shapley value and has demonstrated its efficacy in identifying mislabeled images the CIFAR- 100 test dataset [17]. Recently, Ilyas al. [11] propose datamodels and provide qualitative analyses of the train-test leakage that could hinder fair evaluations. Apart from these methods, many data
valuation algorithms have been developed capture different aspects of data values 42 18
improve computational efficiency [40]. Nevertheless, there has been little attention to establishing a user-friendly and standardized benchmarking system, and this is the main goal of this work.
Our contributions We introduce OpenDataVal, easy-to-use and unified benchmark framework that allows researchers and practitioners to apply and compare different data valuation algorithms with few lines of Python codes. OpenDataVal provides an integrated environment that includes (i) a diverse collection of image, natural language, and tabular datasets that are publicly available at open platforms OpenML (37), scikit-learn [27]. and Py PyTorch (26), (īi) implementations of eleven different sate-offthhear data valuation algorithms in Table M and (iii) a prediction model API that enables to import any machine learning models in seikit-learn and PyTorch-baat based neural network models. In addition, OpenDataVal provides four downstream machine learning tasks for evaluating the quality of data values, noisy label data detection, noisy feature data detection, point removal experiment, and point addition experiment. We also demonstrate how OpenDataVal can be employed through benchmarking analysis in Section We find single data valuation algorithm performs uniformly superior across all the benchmark tasks. That is, different algorithms have their own advantages and capture different aspects of data values. We encourage their analysis.
to choose appropriate algorithm for
OpenDataVal is designed advance the transparency and reproducibility of data valuation works and to foster user engagement data valuation. To support this endeavor, we have made OpenDataVal open-source at fictps://gitan
enabling anyone to con- tribute to our GitHub repository. Additionally, we have developed leaderboard where users participate in noisy data detection competitions. OpenDataVal will be regularly updated and maintained by the authors and participants.
com/ ponodddtaval/peddaaaava
Related works Sim al. conducted a technical survey of data valuation algorithms, focusing on the necessary assumptions and their desiderata. While their analysis provides mathematical insights, lacks comprehensive empirical comparison. Our framework primarily focuses on this problem, creating easy-to-use benchmarking system that facilitates empirical comparisons. ValdaF] is a Python package with set of data valuation algorithms including LOO, DataShapley (7). and BetaShapley [17]. OpenDataVal provides most of the data valuation algorithms available in Valda, plus additional methods, as well as an casy-to-use environment which practitioners
httpp:://uvanlp.orr/raaa

---

Quality of Data Values
Algorithms
Underlying Method
Noisy Label
Noisy Feature Detection
Point Removal
Point Addition
Detection
1.00
Marginal contribution
DataShapley
7
Marginal contribution
KNNShapley
12
Marginal contribution
++
Volume-basee Shapley
41
Marginal contribution
BetaShapley
Marginal contribution
++
++
DataBanzhaf
39
Marginal contribution
+
AME 22
Marginal contribution
InfluenceFunction
5
Gradient
LAVA [3
Gradient
DVRL
42
Importance weight
Data-OOB
18
Out-of-bag estimate
Table 1: A taxonomy of data valuation algorithms available in OpenDataVal. LOO, KNN Shapley, Volume-based Shapley, DataShapley, BetaShapley, DataBanzhaf, and AME can be expressed as functions of marginal contributions, There have been alternative approaches to defining data values: the gradient, importance weights, and out-of-bas estimate. KNNShapley only works for KNN predictors; the other data valuation methods are compatible with any prediction models. The 'Quality of Data Values' section summarizes the benchmark results in Section 4 The symbols -- /- ++' indicate that corresponding data valuation method achieves 'similar or worse / better much better' performance than a random baseline, respectively. There is no single method that uniformly outperforms others in every task. Therefore, it is critical for users to selectively consider data valuation algorithms for their own downstream tasks.
and data analysts compute and evaluate various state-of-the-art data valuation algorithms in diverse downstream tasks.
DataPerf 24 has provided six realistic benchmark tasks: training set creation, test set creation, selection algorithm, debugging algorithm, slicing algorithm, and valuation algorithm. These tasks cover important data-centric problems that can arise in the model development process, but we emphasize that our main objective differs from that of DataPerf work. While they mainly focus creating benchmark tasks and competitions, our main objective is to provide a unified environment for quantifying and comparing various data valuation algorithms.
2 A taxonomy of data valuation methods
In this section, we introduce taxonomy of data valuation methods. For d E N, we denote input space and an output space by S IRd and y C I,, respectively. We denote training dataset by an input and its label for the i-th datum. Then, all
D = {xx, p/) where x; X' and y: data valuation methods can be seen as a mapping that assigns a scalar score to each sample in D, and each quantifies the impact of the point on the performance of model trained on D.
For instance, the leave-one-out (LOO) defined as (1000 y:) == U(D) ( where U represents preset utility function that takes input a subset of the training dataset. In classification settings, a common choice for U is the test accuracy of a model trained on the input. In this case, the i-th LOO value (roo((;; M:) measures the change in the test accuracy when the i-th data point (I,, y/)) is removed from the training procedure.
This concept of LOO has been generalized to the marginal contribution, which captures the average change in utility when certain datum is removed from a set with a given cardinality. To be more specific. for given cardinality 1 < j < n, the marginal contribution of (2,,8) E D with respect U(S U ((x,, 1)))) - U(S) where to j samples is defined as A (Z,, y1) = (3-1 EscD)(e..))
D. (Fr.a))) := {S : S C (x,, }. S = - 1}. Many data valuation methods are expressed as function of marginal contributions, for instance, DataShapley 7 considers the simple average
of marginal contributions Φshap (I,, y:) : _ Σn 4, yi) and BetaShapley [17] considers
weighted average of marginal contributions @beta (It, v.) = Σラ= Wbeta, A (2, y:) for some

---

weight vector (Wbeta.1:- Wbets.n). LOO, DataBanzhaf [39]. DataModels [ and the average marginal effect (AME) [22] are also included in this category, and they differ in how to combine marginal contributions into a single score.
There are alternative approaches to describing the concept of data values: gradient, importance weight, and out-of-bag estimate. The gradient methods are to quantify the rate at which a utility value changes when particular data point is more weighted. The influence function (5] and LAVA [13] are included in this category. They differ depending on what utility value is being measured: the influence function uses the validation accuracy as a utility while LAVA uses the optimal transport cost.
The importance weight method learns weight function X' x y + R that associated model Jn achieves a good accuracy on the holdout validation dataset. Here, the model fa : : - IR is often obtained by minimizing weighted risk R[w((X [ for some loss function 8: y x y - IR and a preset weight function w. Data valuation using reinforcement learning (DVRL) 42 is in this category, and they proposed a reinforcement learning technique obtain a weight function.
An out-of-bag (OOB) estimation approach measures the contribution of each data point to out-of-bag accuracy when a bagging model is employed. Data-OOB [18] is in this category, and it captures how much nominal label differs from predictions generated by weak learners. We provide a rigorous definition of each data valuation method along with their properties in Appendix. A taxonomy of data valuation algorithms with brief summary of empirical results is presented in Table [
3 Overview of OpenDataVal
OpenDataVal provides a unified API called of data values and systemic comparative experiments. With ExperimentMediatonn users can specify various datasets, data valuation algorithms,, and hyperparameters (e.g., prediction model, model performance metric, and synthetic noise). The following code snippet illustrates how to compute data values using the embedding of the CIFARIO dataset. Here, the embedding is obtained from the penultimate layer of the ResNet50 model [10] pretrained on the ImageNet dataset [4].
which enables the quantification
from
import ExperimentMediattor
exper_ __ed______d__ = ExperimentMediator laade_ffcttry_coo (sttup( dataset_ catassttnameeeceee cataeeeeiccaaar--emmeeeea satriicnaae="aaccuree
model_names model_rane="CCassiiioree "ClassffieMMP"" autputadir""uuppurr
add_noise= "mix _abelss" noise_kwargss (""oarrsss"niis_raaae 0.23
)
aata_evaluatoors [DataShapley(] O), DVR
exper_med = exper_mes wwwpute.mmeeee.ccoomomoo data_ values uee (esta__vallatoos)
This code snippet shows how to compute DataShapley 77] and DVRL 42 values for the CIFAR-10 dataset using multilayer perceptron classifier as prediction model, with classification accuracy the evaluation metric. With 'add noise" and 'noise kwargs' arguments, a synthetic noise has been applied: 20% of the training dataset is randomly chosen and their original labels have been changed to different labels.
Code snippet 1:
In addition, OpenDataVal provides a collection of evaluation tasks demonstrated in Section 4 Using the computed data values in the previous step, users can conduct noisy label data detection tasks to systematically evaluate the performance of data valuation algorithms.
OpenDataVal provides a convenient way to compute and compare various state-of-the-ae data valuation algorithms with few lines of Python codes. It is also designed to be highly extensible and casily adaptable to custom experiment settings. We provide details on customization in Appendix.
Leaderboards OpenDataVal introduces the public leaderboards to promote transparency and healthy competition in the field data valuation. the initial release, we provide a noisy label data detection task for a selection of datasets. The challenge datasets, accessible within the OpenDataVal platform, contain label noise that is unknown to the user. The goal to develop their own data valuation algorithms and find which data points have had noise added. Users can submit their

---

from opendataval ppendttavlleexpermmnntsss import noisy_detection
perform noisy label data detection and save results exper_ __trr__ed____ evaluate dentctdde,,,,,on, save_output=True) kmeans_f1
>>>
>>> DataShapley ()
0.5625
>>> DVRL ()
0. 6825
Code snippet 2: This code snippet showcases the task of noise label data detection and evaluates the F1-scores of the DataShapley and DVRL algorithms.
Input Dimension
Number of Classes
Minor Class Proportion
Dataset
Sample Size
Data Type
Source
electricity
38474
6
0.5
Tabular
fried
40768
10
0.498
Tabular
OpenML-901
2dplanes
40768
10
2
0.499
Tabular
OpenML.-727
pol
15000
48
2
0,336
Tabular
OpenMI.-722
MiniBooNE
72998
50
2
0.5
Tabular
29
nomao
34465
89
2
0,285
Tabular
3
bhc-embedkling
2225
768
0.17
Text
g
IMDB- HeDB-eebbediigg
50000
768
0.5
Text
P6]
CIFARIO-embeduen
50000
2048
10
0.1
Image
16
Table 2:
summary of classification datasets used the benchmarking analysis. For natural language and image datasets, the pretrained DistilBERT (30) and ResNet50 [10] models are employed to extract an embedding. For 'fried', '2dplanes', , and 'pol' datasets, the number indicates their OpenML-Datae ID. All datasets readily available in OpenDataVal.
algorithm outputs in the form of a CSV file, and our interactive ranking system will periodically update the leaderboard, showcasing the performance of different submissions. Submissions and competition details are available at htttp::/openddatavall github. wwwphai.iio/eeeebbbbrrrreeee html
Benchmarking Analysis
We present the practical applications of OpenDataVal through benchmarking analysis. We conduct the four machine learning tasks: noisy label data detection, noisy feature data detection, point addition experiment, and point removal experiment. These experiments have been widely used in the literature 7
We provide details on evaluation metries in Appendix. All the datasets, data valuation algorithms, prediction models, and downstream tasks employed in the benchmarking analysis are readily accessible within OpenDataVal, ensuring the transparency and reproducibility of all benchmark results.
42 B
Experimental settings We use the nine datasets in Table 2 These datasets are standard and widely used within the literature [8]. Due space limitations in the main text, primarily focus on the analysis of the six tabular datasets, but our main findings are consistently observed across different modalities and datasets. Results for the text and image datasets Appendix. We compare the eleven different data valuation algorithms in Table T Additionally, we include bascline algorithm, which assigns random numbers data values. The InfluenceFunction Volume-basse Shapley, DataBanzhaf, AME, and Data-OOB require training models multiple times, and the number of trained models has a substantial impact on both performance and runtime. To ensure fair comparison, we have set this parameter to a fixed number of 1000. As for the other algorithms that do not require training models multiple times, we maintain fair comparisons by utilizing hyperparameters similar to those specified in the original papers. As for the base prediction model, we use a logistic regression model. However, is important to note that the logistic regression model is not compatible with KNNShapley as it is implicitly based KNN predictors and has closed-form expression. The sample size for the training dataset is set a 1000. Implementation details provided in Appendix.
provided in
5

---

Dataset: 2dplanes
Dataset: electricity
Dataset: MiniBooNE
SetaShopley
F1-score
0.5-
F1-score
0.4
KNNS hapley
DataShap
Volumeshapiey
0.31
DataB BataBanzhaf
influencefunction
0.2
AME
0.1
0.0
LeaveOneOut
Rar dom
Noise proportion (%)
Noise proportion (%)
Noise proportion (%6
Dataset: pol
Dataset: fried
Dataset: nomao
0.6
0.5
1-scoree
0.5
0.4
9.4
0.3
0.2
0.2
0.2
0.1
0.0
15
Noise proportion (%)
Noise proportion (%)
Noise proportion (%)
Figure 2: Noisy label data detection. Fi-score of different data valuation algorithms on the four noise proportion settings. The higher the Fi-score is, the better the data valuation algorithm is. The error bar indicates 95% confidence interval based 50 independent experiments. Data-OOB demonstrates significantly superior performance in detecting mislabeled data points in various situations.
4.1 Noisy data detection
We compare the detection capabilities of different data valuation algorithms using synthetically generated noisy datasets. We consider two types of synthetic noise. The first type label noise where we flip the original label to its opposite label. The second type is feature noise where we add standard Gaussian random errors to the original features. After selecting one of two noises, then randomly choose Puoise % of the training dataset to perturb. Here, the four different levels of noise proportion Puoise E {5, 10, 15, 20) considered.
Data values are computed for this noisy dataset, and we investigate which data valuation algorithm
the most effective identify these noisy data points. Based the literature (39 17 18
follow the approach of applying the k-means clustering algorithm to the data values 2 This allows dichotomize the training dataset into two groups: beneficial and detrimental. We assign the
detrimental label to the cluster with a lower average of data values. Given our expectation that mislabeled data points are likely to have low values, we predict data points within the detrimental group as noisy samples. We evaluate the Fl-score between this prediction and the ground-truth annotation. Note that do not use ground-truth annotations for noisy data points when computing data values. We only use them when we evaluate the quality of data valuation algorithms.
Noisy label data detection Figure shows the Fl-score of data valuation algorithms for the noisy label data detection tasks. The results be grouped into three clusters based on their performance. The first cluster consists of a single algorithm, Data-OOB. It achieves significantly better results than other data valuation algorithms the 'MiniBooNE: dataset with Pnoise = 20%, Data-OOB shows 0.65 Fi-score when the second best algorithm KNNShapley shows 0.53 and the random baseline shows 0.29. The second cluster consists of DVRL, KNNShapley. DataShapley, and BetaShapley. They show reasonably sound detection ability compared the random baseline. Volume-based Shapley, LAVA, LOO, InfluenceFunction, AME, and DataBanzhaf belong to the third cluster and show similar performance to the random baseline.
most of the datasets and noise proportions. For instance,
Noisy feature data detection Figure 3 shows the F1-score of different data valuation algorithms for the noisy feature data detection tasks. contrast to the noisy label data detection, LAVA generally outperforms other methods with substantial performance gap. This difference is potentially due to the transport cost function, one of the key hyperparameters LAVA [3]. The default choice is designed to be sensitive to noises in features, leading to good performance in noisy feature

---

Dataset: 2dplanes
Dataset: electricity
Dataset: MiniBooNE
Data-008
BelaStupley
0.6
KNNSnopley
F1-score
0.5
F1-score
Dat aShop lyy
0.4
0.3
ValumeShapley
DataBarahaf
0.3
Infuenceeunction
02
LAVA
0.3
0.1
LeavcOneOut
10
20
10
15
Random
Noise praportion
Noise proportion (%)
Naise proportion (%)
Dataset: fried
Dataset: pol
Dataset: nomao
0.7
F1-scofe
F1-score
0.6
0.4
0.4
0.3-
10
20
10
20
Noise proportion (%6
Noise proportion (%)
Noise proportion (%)
Figure 3: Noisy feature data detection. Fi-score of different data valuation algorithms on the four noise proportion settings. The higher the Fl-score is, the better the data valuation algorithm is. The error bar indicates a 95% confidence interval based on 50 independent experiments. LAVA generally outperforms other data valuation algorithms with a big performance gap.
data detection. However, we observe that LAVA's performance varies a lot across different datasets, suggesting that this choice of the transport cost function is important. Following LAVA, Data-OOB and Shapley-based methods demonstrate a better performance than the random baseline, showing their potential efficacy in noisy feature data detection. Volume- based Shapley, DVRL, Influence AME, and LOO show only comparable performance to the random baseline in this task.
4.2 Point removal and addition experiments
Data values can be used to find influential samples. To evaluate this quantitatively, we perform the point addition and removal experiments used in [8]. The point removal experiment is performed with the following steps: For each data valuation algorithm, we remove data points from the entire training dataset in descending order of the data values. Each time the datum is removed, we fit logistic regression model with the remaining dataset and evaluate its test accuracy on the holdout dataset. As remove the data points in descending order, in the ideal case remove the most helpful data points first, and thus model accuracy is expected to decrease. For the point addition experiment, perform similar procedure but add data points in ascending order. Similar to the point removal experiment, the model accuracy is expected to be low as we add detrimental data points first. Throughout the experiments, to Pnoise 20%. The sample size of the holdout test dataset is set to 3000. All the procedures here can be easily performed in OpenDataVal.
used perturbed dataset with label noise. The noise ratio is set
Point removal experiment Figure shows test accuracy curves for the point removal experiment. Overall, all data valuation algorithms except Data-OOB perform better than the random baseline. Data-OOB often shows worse performance than the random baseline, indicating its effectiveness finding low-quality data points does not imply finding helpful data points. AME, DataShapley, BetaShapley, and KNNShapley generally show superior performance. In particular, on the 'fried' dataset, these methods achieve at most 75% test accuracy after removing 20% of the most influential data points. DataBanzhaf and InfluenceFunction also show solid performance, while DVRL shows large performance differences across different datasets. LOO shows a slightly better performance than the random baseline.
Point addition experiment Figure 5 shows test accuracy curves for the point addition experiment. All data valuation algorithms generally outperform the random baseline, showing their efficacy in identifying low-quality data points. Data-OOB uniformly shows the lowest test accuracy compared
7

---

Dataset: 2dplanes
Dataset: electricity
Dataset: MiniBooNE
0.92
0.72
0.82
Date-Onn
Test accuracy
0.80
Bod BalaShappey
DVAL
Shaplyy
6.78
0.550
DataShoplcy
0.76
Vokumeble
0.74
0.74
DataBannzef
inllencouundtio
0.75
1.0
16.5 Dataset:
11.0
15.0
15. Proportion (%) data removed
15.0 10.0 Proportion (86) of data removed
5.0
7.0 11.0
Proportion (%) of data removed
Random
0.875
0.89
0.850
0.80
0.88-
accuracg
0.825
0,800
0.78
0.775
0.750
0.84
0.725
0.72
0.83
0.70
3.0
7.0
13.0 15.0
19.0
7.0
3.0
Proportion removed
Proportion
Figure 4: Point removal experiment. Test accuracy curves as a function of the most valuable data points removed. Lower curve indicates better data valuation algorithm. The error bar indicates 95% confidence interval based on 50 independent experiments. AME and Shapley-base based methods exhibit superior performance.
Dataset: 2dplanes
Dataset: electricity
Dataset: MiniBooNE
0.8
0.7
0.7
BetaShapley
Test accuract
Text accuracy
KNNShapley
DataShapley
0.5
DataBanshaf
VolumeShapiin
0.4
0.0
Infuercceuunction
0.3
AME
0.2
LeaveoOneOut
3.0
Rancom
Proportion (%) data added
Proportion (%) of data added
Proportion (%) data added
Dataset: pol
Dataset: fried
Dataset: nomao
0.8
Test accuraty
Test acturayy
0.7
0.6
0.6
0.6
0.5
0.5
0.4
0.4
0.3
0.2
0.2
0.2
0.1
3.0
7.0
11.0
15.0
13.0
3.0
7.0
11.0
15.0
19.0
3.0
7.0
11.0
15.0 19.0
Proportion (96) data added
Propartion (86) of data adieed
Proportion (5) data added
Figure 5: Point addition experiment. Test accuracy curves function of the least valuable data points added. Lower curve indicates better data valuation algorithm. The error bar indicates a 95% confidence interval based 50 independent experiments. Data-OOB, DVRL, and Shapley-based methods exhibit superior performance.
to other algorithms, demonstrating the best performance in finding low-quality samples. Following Data-OOB, DVRL., DataShapley, and BetaShapley show promising performance in this task. Volume- based Shapley. LAVA. LOO, and InfluenceFunction show similar test accuracy, but InfluenceFunction exhibits a slightly better performance than other methods across all datasets. Volume-based Shapley is generally worse only better than the random baseline, which has also been observed in the Shapley is independent of the label information,
literature 13. This is because the and thus it incompletely reflects the quality of data points.
4.3 Runtime comparison
We also conduct a comparison of the runtime required compute 1000 data values for the noisy label data detection task. In this task, use the three tabular datasets (2dplanes, electricity, and

---

Dataset: 2dplanes
Dataset: electricity
Dataset: MiniBooNE
Time (in seconds)
Time (in seconds)
10t
Time seconds)
10-
DVAL
BetsShapley
KNNtlhappey
101
10
DataShapley
Shaple
10P
DataBanchaf
rilluercauunctinn
LAVA
10-1
AME
67
07
6²
LeaveOneOut
F1-Score
F1-Score
F1-Score
Dataset: IMDB
Dataset: BBC
Dataset: CIFAR10
Time (in seconds)
Time (in seconds)
104
Time (in seconds)
10f
201
104
101
to
103
0.00
0.23
0.00
0.50
0.55
0.30
0.35
0.50
F1-Score
F1-Score
F1-Score
Figure 6: Runtime comparison. The runtime to compute 1000 data values is measured for the noisy label data detection task. The z-axis represents the Fi-score used in Section 4.1 and the y-axis represents the runtime in the log scale. KNNShapley shows the best performance terms of speed, while Data-OOB achieves better Fl-score in reasonable time frame. Both DataShapley and BetaShapley are computationally expensive because they are based on marginal contribution estimation algorithms. They perform similarly in BBC, IMDB, and CIFAR10 datasets, and their points are visually overlapped.
MiniBooNE), two text datasets (BBC and IMDB), and one image dataset (CIFARIO) in order to show that findings overall consistent across different modalities. We find similar pattern with other tabular datasets. We utilize noise dataset containing label noise with a noise ratio of Proise = 20%. but we highlight that the runtime is independent of the specific type of noise its intensity. Figure illustrates the runtime of the data valuation algorithms, along with the corresponding F1-scores for the noisy label data detection. Across most datasets, LAVA and KNNShapley demonstrate the shortest computation time because LAVA does not require model training and KNNShapley has closed-form expression. In contrast, DataShapley and BetaShapley are shown to be computationally costly because they need to estimate the marginal contribution, which requires large number of model training. Aside from these three algorithms, most of the other data valuation algorithms exhibit similar computational times as their key hyperparameter, the number of models to train, is set to the same value of 1000. Among them, Data-OOB demonstrates the best Fi-score in noisy label data detection.
5 Discussion
In this paper, we propose OpenDataVal, easy-to-use benchmark environment, and provide convenient way to compute and compare data valuation algorithms with a few lines of Python codes. To ensure transparency and reproducibility, we have made our framework publicly available.
Our benchmarking analysis demonstrates no single method uniformly superior across all evaluation metrics. Data-OOB achieves powerful performance in the noisy label data detection tasks, but performs less well in the point removal experiment. That is, it is very effective in identifying low-quality data points, but it fails to find the most positively influential data points. In contrast, AME shows the opposite results. By leveraging the LASSO model, it shows competitive performance in identifying the most beneficial data points in the point removal experiment, while it shows weak performance finding low-quality data points the noisy data detection task. We also find that Shapley-based methods (DataShapley, BetaShapley, and KNNShapley) have comparable performances. KNNShapley faster but is limited to using KNN predictors, while DataShapley and BetaShapley can flexibly work with other predictors. Many theoretically rigorous and empirically

---

comprehensive analyses are critical, and we believe OpenDataVal opens the first step and facilitates fair and reproducible analyses in various experimental settings.
There are several intriguing future directions. In many real-world applications data can be duplicated and fraction of them can be hard-encoded with certain value (9999 -9999 for missing data) (14). Furthermore, data can contain incomplete information in survival analysis, semi-supervised learning, and weakly-supervised learning settings, raising questions about how evaluate the influence of such data points. Developing data valuation for sequentially observed data is also an important direction. For instance, time-series forecasting multi armed bandit problems, previously observed data can substantially affect the subsequent data points, which makes it difficult to deploy techniques in typical i.i.d. learning settings. Developing notion of data values these scenarios will be an interesting and potentially influential problem.
In broader context, the practical implementation of data valuation can give rise several economic and societal questions. In data marketplaces, for instance, data providers might duplicate, transform, or even maliciously modify data maximize their business profits. Most of the existing data valuation algorithms are not designed to handle these types of attacks, and as result, erroneously evaluated data values can negatively affect the establishment of reliable and transparent data marketplaces. Developing incentive-compatible data valuation is an interesting new direction [32 28 34], Another potential concern is data security in distributed and federated learning scenarios. Data owners may hesitate to share their data with the main server due to privacy issues, especially if data owners are competing with each other [34]. Evaluating data points based on aggregated summaries (e.g.. gradients) will be an interesting topic as existing data valuation algorithms require direct data access.
Acknowledgements
The authors would like thank all anonymous reviewers for their comments. We would like thank Eunsil Seok and Ian Covert for their constructive feedback. We also acknowledge computing resources from Columbia University's Shared Research Computing Facility project, which is supported by NIH Research Facility Improvement Grant 1G20RR030893-01, and associated funds from the New York State Empire State Development, Division of Science Technology and Innovation (NYSTAR) Contract C090171, both awarded April 15, 2010.
References
[1] Anish Agarwal, Munther Dahleh, and Tuhin Sarkar. A marketplace for data: An algorithmic solution. In Proceedings of the 2019 ACM Conference on Economics and Computation, pages 701-726, 2019.
[2] David Arthur and Sergei Vassilvitskii. k-means++ the advantages of careful seeding. In Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, pages 1027-1035, 2007.
[3] Laurent Candillier and Vincent Lemaire. Design and analysis of the nomao challenge ac- tive learning in the real-world. In Proceedings of the ALRA: active learning in real-world applications, workshop ECML-PKDD, pages 1-15. Citeseer, 2012.
[4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image datahase. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[5] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33: 2881 2891- 2020.
[6] Joao Gama, Pedro Medas, Gladys Castillo, and Pedro Rodrigues. Learning with drift detection. In Advances in Artificial Intelligence-SBIA 2004: 17th Brazilian Symposium on Artificial Intelligence, Sao Luis, Maranhao, Brazil, September 29-Ocotber 2004. Proceedings 17, pages 286-295. Springer, 2004.
[7] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference Machine Learning, pages 2242-2251, 2019.
10

---

[8] Derek Greene and Pádraig Cunningham. nance in kernel document clustering. In Proceedings of the 23rd international conference on Machine learning, pages 377-384, 2006.
Practical solutions to the problem of diagonal domi-
[9] Frank R Hampel, Elvezio M Ronchetti, Peter Rousseeuw, and Werner A Stahel. Robust statistics: the approach based on influence functions. Wilee-Interscienee; New York, 1986.
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[11] Andrew lyass Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Datamodels: Understanding predictions with data and data with predictions. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pages PMLR, 17-23 Jul 2022. URL https::/proceeedingggge .ml. www.srressaeee (rrress/v622i1yya2222
html
[12] Ruoxi Jia, David Dao, Boxin Wang. Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas Spanos, and Dawn Song. Efficient task-specific data valuation for nearest neighbor algorithms. Proceedings of the VLDB Endowment, 12(1 1):16-1-16,,, 2019.
[13] Hoang Anh Just, Feiyang Kang, Tianhao Wang, Yi Zeng, Myeongseob Ko, Ming Jin, and Ruoxi Jía. LAVA: Data valuation without pre-specified learning algorithms. In The Eleventh International Conference on Learning Representations, 2023. URL cattps://openee
net:/torum?2d--JJP6666BB3.
[14] Alan F Karr, Ashish P Sanil, and David L. Banks. Data quality: A statistical perspective. Statistical Methodology, 3(2):137-173 2006.
[15] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions.
In International Conference on Machine Learning, pages 1885-1894, PMLR, 2017.
[16] Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
[17] Yongchan Kwon and James Zou. Beta shapley: unified and noise-reduced data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics, pages 8780-8802, PMLR, 2022.
[18] Yongchan Kwon and James Zou. Data-oob: Out-of-bag estimate as simple and efficient data value. 2023.
[19] Weixin Liang and James Zou. Metashift: A dataset of datasets for evaluating contextual distri- bution shifts and training conflicts. In International Conference on Learning Representutions, 2022. URL httpttss:///ppeeeee //tpprrve///peevieeeee http:pp//.pppppereee.eeee/////ee
[20] Weixin Liang, Girmaw Abebe Tadesse, Daniel Ho, L Fei-Fei, Matei Zaharia, Ce Zhang, and James Zou. Advances, challenges and opportunities in creating data for trustworthy ai. Nature Machine Intelligence, 4(8):669-61) 677, 2022.
[21] Weixin Liang, Yining Mao, Yongchan Kwon, Xinyu Yang, and James Zou. Accuracy the curve: On the nonlinear correlation of ml performance between data subpopulations. In /CML 2023 Poster, 2023. URL httpps://ppenrevier http:://pppareeiieeeaa//oooddddd
[22] Jinkun Lin, Anqi Zhang, Mathias Lécuyer, Jinyang Li, Aurojit Panda, and Siddhartha Sen. Measuring the effect of training data on deep learning predictions via randomized experiments. In International Conference on Machine Learning, pages -3504 PMLR, 2022.
[23] Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the asxociation for computational linguistics: Human language technologies, pages 142-150,
2011.
[24] Mark Mazumder, Colby Banbury, Xiaozhe Yao, Bojan Karlaš, William Gaviria Rojas, Sudnya Diamos, Greg Diamos, Lynn He, Douwe Kiela, David Jurado, et al. Dataperf: Benchmarks for 2022.
data-centric ai development. arXiv preprint

---

[25] Curtis G Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label emrors in test sets destabilize machine learning benchmarks. In Thirty-fifth Conference on Neural Informa- lion Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URI. https hwpperawwwee..eee//oommmtttccoed
[26] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- rerforerformecc deep learning library. Advances in neural information processing systems, 32, 2019.
[27] F. Pedregosa,
G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-leamn Machine learning in Python. Journal of Machine Learning Research, 12:2825-2833 2011.
[28] Rui Qiao, Xinyi Xu, and Bryan Kian Hsiang Low. Collaborative causal inference with fair incentives. 2023.
[29] P Roe Byron, Yang Hai-Jun, Zhu Ji, Liu Yong, Stancu Ion, and McGregor Gordon. Boosted decision trees, an alternative artificial neural networks. Nucl. Instrum. Meth. A, 543:577-584, 2005.
[30] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.011 108, 2019.
[31] Lloyd S Shapley. A value for n-person games. Contributions the Theory of Games, 2(28): 307-317, 1953.
[32] Rachael Hwee Ling Sim, Yehong Zhang, Mun Choon Chan, and Bryan Kian Hsiang Low. Col- laborative machine learning with incentive-aware model rewards. In International Conference on Machine Learning, pages 8927-8936. PMLR, 2020.
[33] Rachael Hwee Ling Sim, Xinyi Xu, and Bryan Kian Hsiang Low. Data valuation in machine "ingrciintn,, strategies, and open challenges. In Proc. IJCAI, 2022.
[34] Mukund Sundararajan and Walid Krichene. Inflow, outflow, and reciprocity in machine learning. 2023.
[35] Siyi Tang, Amirata Ghorbani, Rikiya Yamashita, Sameer Rehman, Jared A Dunnmon, James Zou, and Daniel L. Rubin. Data valuation for medical imaging using shapley value and applica- tion to large-scale chest x-ray dataset. Scientific reports, 1 1(1):119, 2021.
[36] Zhihua Tian, Jian Liu, Jingyu Li, Xinle Cao, Ruoxi Jia, and Kui Ren. Private data valuation and fair payment in data marketplaces. arXiv preprint arXiv:2210.0877 2022.
[37] Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo Openml: networked science in machine learning. SIGKDD Explorations, 15(2):99-60, 2013. doi: 10.1145/2641190 2641 198. URL http:://op...... acm. org/10. 11w.11111///1990/ 264119
[38] Dootika Vats and Christina Knudson. Revisiting the gelman-rubin diagnostic. Statistical Science, 36(4):518-52) 2021.
[39] Jiachen T. Wang and Ruoxi Jia. Data banzhaf: A robust data valuation framework for machine learning. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, 206:6388-644 2023.
[40] Mengmeng Wu, Ruoxi Jia, Wei Huang, Xiangyu Chang, et al. Robust data valuation via variance reduced data shapley. arXiv preprint arXiv:2210. 16.85, 2022.
[41] Xinyi Xu, Zhaoxuan Wu, Chuan Sheng Foo, and Bryan Kian Hsiang Low. Validation free and replication robust volume-based data valuation. Advances in Neural Information Processing Systems, 34:10837-10848, 2021.
[42] Jinsung Yoon, Sercan Arik, and Tomas Pfister. Data valuation using reinforcement learning. In International Conference on Machine Learning, pages 10842-10851. PMLR, 2020.
12

---

Appendix
The appendix provides technical details on data valuation algorithms (Section and the four downstream tasks (Section B available in OpenDataVal. Additional benchmarking analysis results are also presented (Section C and we observe consistent findings on different datasets. Lastly, we provide detailed descriptions on OpenDataVal APIs in (Section[D)
A Data valuation algorithms
This section elaborates on different data valuation algorithms available in OpenDataVal. While some notations have already been defined in Section 2, present comprehensive set of notations here. For d E N, we denote an input space and an output space by X' S F and y S R, respectively. We denote training dataset by D {(11..))) )= where æi X' and yı E y are input and its label for the i-th datum. For set S, we denote its power set by 2 and its cardinality by (S]. We set [i] : {l.....3 for j E N, and (A) is an indicator whose value is one if statement A is true and zero otherwise.
A.1 Marginal Contribution- methods
The marginal contribution-based methods are formally introduced, which include LOO,, DataShapley 77. KNNShapley [12), Volume-based Shapely [1], BetaShapley [17] DataBanzhaf 39 and AME
122..
We first define the marginal contribution below.
Definition A.1 (Marginal contribution). For set function U 2n + and j [n]. the marginal contribution of (z,, 81) E DD with respect to samples is defined as follows.
( yri U)
Σ
U(SU (1., USS),
where D) () = SS
S -|
The marginal contribution (x; ves U') measures the average change in U when a particular data point (Ii; 31) is removed. Here, a common choice for set function U the performance of a model trained on a subset of the training dataset D. We provide two examples below.
Example A.1 (Classification). A default choice for a set function U in OpenDataVal is the validation classification accuracy of an empirical risk minimizer trained subset of DD. To be more specifie, we denote the validation dataset by {ã Ui)
Poual Then,
1 Rual
U(S)
Σ
1 f
Tlral
where IS == argginn JEF Ejes (y, + (xx,)) for some class of classification models F {S X - >}. When = 0 the empty set, US) is set to be the performance of the best constant model by convention.
Example A.2 (Regression). A default choice for set function U in OpenDataVal is the validation negative mean squared error of an empirical risk minimizer trained on a subset of D. Using the same notation defined in Example A.l
U(S) = Thaal
Σ [ fs(
where is argmingez " y}.
for some class of regression models F :
It noteworthy that a set function U may depend on the choice of learning algorithms (e.g., empirical risk minimization) and a model class F. For notational simplicity, we suppress a learning algorithm and model class.
13

---

Estimation of the marginal contributions. Following the existing literature 77 OpenDataVal uses a truncated Monte Carlo algorithm. Specifically, let π (T......,,, E N be random permutation of In]. Then, for j E [nl, we compute the utility U(((TTL Di, ( )=))) To gradually increase the cardinality, iteratively increment j from to We stop increasing the cardinality if a relative change is converged. Specifically, for [n], we define the relative change as follows.
IU (f(IT.+ Uoa k= U (Ime UF141 }) U (((ITT Vra )11=1)
Vi
U (f( Urr )kk))
We stop increasing the cardinality at j = argmin E [n] S V 10-*H| 2 10), Le., it is the smallest cardinality j that there are 10 cardinalities less than j with small relative change Vi.
While repeatedly generating random permutations, we use the utility gap to estimate the marginal contribution. In other words, the multiple estimates of (( "j, Tz-13) UU nz)
are used to estimate Aj+1 (Imzy Vr..a). We stop generating random permutations when a simple average of marginal contributions is converged. We used the Geeman-Rubll statistics [38] by constructing ten independent Monte Carlo chains for each training data point. The maximum of these statisties across samples is considered, and stop drawing random permutations if the maximum value of Sellman-Rubin statistics is less than the threshold value of 1.05 or the number of random permutations exceeds 10 In OpenDataVal, to obtain reliable estimates, we the default value for the minimum number random permutations as 300.
LOO. The LOO measures the change in U when one data point of interest is removed from the entire dataset, and is given as follows.
Pioo (mt, Uı) == (n, (T). W.)
Note that (T, y4) U(D) U(D (z,,y:))) 9:))). .) In OpenDataVal, the computation of LOO is done by naively computing (D) and (D x))) for all i [m].
DataShapley. DataShapley takes a simple average of all the marginal contributions as follows.
Oshap (I y1) := Σ (z y:). j=1
In OpenDataVal, as suggested Ghorbani and Zou 77 we estimate the marginal contributions using the truncated Monte Carlo algorithm and compute its simple average.
KNNShapley. KNNShapley, like DataShaplcy, is based on the Shapley value but is characterized by the use of a specific utility function. In classification settings, for instance, KNNShapley uses the following utility function.
Real
U(S)
Σ
1 mj),
Te
(EN,
where & E [n]] is a preset hyperparameter and represents l nearest neighbors in KNN predictors and \(E,, S) C D is a set of min(k, 6, (S]) nearest neighbors of ž,. In regression settings,
Rval
U(S) Ti
Σ
(ū. 3
(R..)))N With these utility functions, 12 showed that KNNShapley yields a closed-form expression, and they are implemented in OpenDataVal.
Volume-based Shapley The idea of the volume- Shapley by Xu et al. 41 is to the same Shapley value function as DataShapley, but it is characterized by using the volume of input data for a utility function.
U(S) = Vol(Xs) = V
Σ r.afll i
where G is determinant a matrix G E Rddd It is noteworthy that Equation I does not require a validation dataset, leading validation-fres data valuation method. However, it does not use label information, and thus it is not appropriate to describe the quality of labels. The previous work [3] observed that it is often the main reason for poor performance in many downstream tasks.
14

---

BetaShapley.
BetaShapley has a form of a weighted mean of the marginal contributions.
Obet (1t, yc)
Wbeta. W (24,
Beea()+---1n--ta)) where Wbeta.j j-11
is the Beta function and r the Gamma function. In OpenDataVal, (a. (4, 1) set to the default parameter and similar to DataShapley, BetaShapley values are obtained after estimating the marginal contributions.
- - +
Beta(o,
DataBanzhaf. DataBanzhaf's is based on the Banzhaf value, which is defined as follows.
bannz (Zis (eis = Σ Uy
where Wbanz.j = =2-"("1) In OpenDataVal, instead of using the marginal contribution estimates, we follow the original algorithms proposed by Wang and Jia [39]
AME. Lin et al. [22] proposed the average marginal effect, which is also known as AME.
AaME( 34) == (s {T;; v.)1) - U(S)..
where the expectation Es is taken over a random set S with a user-defined distribution defined on the discrete space (((r.))) Lin et al. [22] showed that AME can be efficiently estimated by predicting model's prediction. Specifically, for a regularization parameter À > 0, a predefined transformation function g 1) -> IR"", and set S (S S D), AME can be estimated solution of a LASSO regression model.
(U(()) -U(S))9((s)))) o((
argmin
mll
S
SES
where 1 E {0, 13" is n-dimensional vector whose element is one if its index is an element of S, zero otherwise. In OpenDataVal, following Lin et al. [22], the same transformation function and the S that follows the same uniform distribution are used. We choose the optimal regularization parameter with the cross-validation using 'LassoCV' in 'scikit-learn' with its default parameter values.
A.2 Gradient-based methods
Influence function. The influence function originally introduced robust statistics measure the effect of data point on a statistical estimator by its Gateaux derivative [9. In machine leamiin, has been often approximated by LOO, or more generally, the difference between two average model performances: one containing data point of interest in the training procedure and the other not [15 To be more specific, for pre-defined hyperparameter M E [m]. we denote a set of subsets with the same cardinality by S(M = {S s(M)
S D. = M). For i [mi]. we set Lin == (S S(M) (Ii,V:) SS} and s/M)) == {S E S(M) (æi, S). Note that
((M)
n s.S.)) () for all E [n]. Then, the influence function used in Feldman and Zhang [5] is defined as follows.
Lin USS! S(M))) = S and ((M)
Σ U(S)
Σ U(S).
(M)
Siin
(
SestAry S,on
In OpenDataVal, suggested by Feldman and Zhang (5] M is chosen as [0.7 n] where [x] is the biggest integer that is less than or equal to σ.
"The authors in 17 show that (a. = (16, 1) yields the best performance, but we empirically find (a, 8) (4, 1) also yields competitive results in various datasets. In our settings, (a, B) (4, often shows better results, and thus it chosen as the default value.
'The authors in [15] proposed an efficient way to compute the influence function using Hessian-vector products, but it has been approximated with LOO.
15

---

LAVA. LAVA I3
proposed to measure how fast the optimal transport cost between training dataset and validation dataset changes when a training data point of interest is more weighted. This idea is formalized via the gradient of the optimal transport cost with respect to the probability mass, and in their paper, they showed that this gradient is expressed follows.
OLAVA(FA =
Σ K5.
71
where (hf...., (i)) is part of the optimal solution of the dual problem: for validation dataset {T D:) 8e)
h" h","" == argmax 1 (o))))) + (g, nval
where (x y) is the set of all continuous function defined on X' V, and ((z,)) is the Delta measure at (z. 3/ X
A.3
Importance weight- based methods
Data valuation using reinforcement learning (DVRL) proposed by Yoon et al. involves the utilization of reinforcement learning algorithms to compute data values. DVRL trains model X' x [0, 1] that outputs its importance weight by solving the following objective function.
min E[(Y fx(x)) s.t. fn argmin fEF /BP (X)))(())))) bEH
where H (h computed as the importance weight h2x y). In OpenDataVal, we follow the hyperparameters used Yoon al. 42
[0, 1}} and : (f X' y}. Then, the data value of (z, y) is
A.4 Out-of-bag setimate-basee methods
Data-OOB by 18] is a distinctive data valuation algorithm, which uses the out-of-bag estimate to describe the quality of data. Suppose we have a bagging model that consists of B weak learner models: for E [B], we denote the b-th weak learner by fo X + D, which trained on the b-th bootstrap dataset. It can be expressed as minimizer of a weighted risk as follows.
fo = argmin JEF Whyllyi f(x;))
where 1: IR is loss function and Whj Z is the number of times the j-th datum (æj, Vj) selected the b-th bootstrap dataset. With these notations, for E [n], we propose to use the following quantity data values.
EB 1(wb ()T fo(xz))
(2)
Σ 11 (ww = 0)
where T :
3/xx x - R. score function that represents the goodness weak learner fo at the i-th
datum (t). For instance, we can use correctness function T(m, fo(a:)) 1 = f((x))) in classification settings and the negative Euclidean distance T f((x,)) = -(y - f²(x:))2 in
regression settings.
B Tasks and their evaluation metrics
We formally introduce the four machine learning tasks that assess the effectiveness of data evaluation algorithms: (i) noisy label data detection, (ii) noisy feature data detection, (iii) point removal experiment, and (iv) point addition experiment.
16

---

B.1 Noisy label data detection
The main goal of noisy label data detection is to identify data points that have been mislabeled. We consider a synthetically generated mislabeled dataset: we randomly choose Pnoise % of the training dataset and replace the original label with a different label. Let S ((mislabeled) C [ri] be a set of indices of mislabeled data points. This annotation is not available any data valuation algorithms and is only available when we assess the data valuation algorithms.
Let Φ x E be data valuation algorithm and denote set of computed data values by 2,, = (0(21,
p(xn. Bn))) We divide a set of n points dn into two clusters using the k- means clustering algorithm [2] and denote these two clusters of indices by s(low) and S Sthigh) That is, s(low) U. ال [n]] and s(low) ( (). Without loss of generality, assume S(low) has
lower mean than 5/20 (:high)) i.e. [s((or)) D' (II, y1) S (S
-esssshia Φ(2;, 31).
Since it is desirable for mislabeled data points to have lower values, we regard sollow as the prediction for the set of mislabeled data points. Then, we compute the Fl-score between the prediction set s(low) and the ground-trun truth annotation ((mislabeled) as follows
(low) n S(mislabeled))
F1(5 (low)) S g(mishabeled) = [
(mislabbel)))
It is noteworthy that F1 (S
(( low) S (mialabeledd) is the harmonic mean of the precision and recall the
(aa))) ns (mistabbeled
[ ns (gmiislabeled) S ((mistabalec)
precision is expressed as
and the recall expressed as
B.2 Noisy feature data detection
The main goal of noisy feature data detection is identify data points with perturbed features. Similar to Section B.I we consider a synthetically generated perturbed dataset: we randomly choose
Phoise % of the training dataset and add some Gaussian random errors to every original feature. Let S (pertur)ed))
C [n] be set of indices of perturbed data points. For each data valuation algorithm ø. we compute s s(low) and then we evaluate F1(s( ((pertarbed) as in Section B.1
B.3 Point removal experiment
Let Φb x be data valuation algorithm and denote set of computed data values by 2 (0(zI,11).... o(In, )n)). Suppose
(I),..., TT) N" represents indices of data values descending order, i.e., Φ(TT, Nn, 2 ... 2 ((I,,, )a,.). In the point removal experiment, we remove data points one by one in descending order from the entire training dataset. That is, we remove (In, Vr,, first and (IT.. Vr.)) ) last. Each time the datum is removed, we fit logistic model with the remaining dataset. For instance, after & data points removed, we train logistic model with D) {Irz Va, l We denote the performance after i data points are removed by perfg. Then, for preset K N, the point removal experiment evaluates the average performance after the removal of points:
K
perf k-
K
k=1
Throughout this paper, consider K the performance perf,.. As we removed the most valuable data points first, lower value indicates a better performance. The point removal curves in Figures 4 and 2 are (perf,,.... perf x and thus the evaluation metric represents the normalized area under the curve. In experiments, a coarser grid used that evaluates perf every five-point removal, not one data point.
|0.2 x nj. and the classification accuracy is used compute
B.4
Point addition experiment
The point addition experiment similar the point removal experiment, but the only difference that we add data points from the empty set in ascending order. That is, we first add first (Ix., yx.)
17

---

and (Ixi: Mr.
last. The evaluation metric for the point addition experiment is
perf
We consider K = 10.2xn]. and the classification accuracy is used for the performance perf A lower value indicates better performance. The curves in Figures and 10 See Section B.3 for more details.
are perf k)
C Additional Benchmarking Analysis
C.1 Missing details experimental settings
Datasets For tabular datasets, a standard normalization procedure is employed, wherein each feature is centered and normalized have a mean of zero and a standard deviation of one. However, for non-tabular datasets such as CIFAR I0, BBC, and IMDB, no specific normalization procedures applied except for the feature extraction. We the pretrained ResNet50 model for CIFARIO
[10] and the pretrained DistilBERT model for texts [30] Once the preprocessing steps are completed, following the literature [18] we split it into three datasets, namely, training dataset, a validation dataset, and a test dataset. The data valuation process focuses on assessing the value of data within the training dataset, while the validation dataset is utilized for evaluating the utility function 6 The sample sizes for the training and validation datasets are set to 1000 and 100, respectively. The size of the test dataset is fixed 3000 for all datasets, except for the text datasets (BBC and IMDB), where it is set to 500. The test dataset is used for point removal and addition experiments only when evaluating the test accuracy. As for the noisy feature data detection problem, we generate a random error from a Gaussian distribution with mean of zero and a standard deviation of two. Although our benchmark analysis fair and captures the gist of different data valuation algorithms, we encourage users to establish various experimental settings with OpenDataVal.
Data valuation algorithms We compare the eleven different data valuation algorithms listed in Table Regarding the hyperparameters, we use the default settings provided by OpenDataVal, with most of the default values derived from the original research papers. There are some hyperparameters we choose for a fair and efficient comparison: for the K KNNShapley, we set the number of k-nearest neighbors 10% of training data points. In the case of DVRL,, the model is trained for total of 2000 epochs. Data-OOB, InfluenceFunction, AME, and DataBanzhaf have a shared key parameter 'the number of models to train', and set to the same number of 1000, throughout all experiments.
C.2 Additional Experimental Results
We show additional benchmarking analysis results using one image dataset (CIFAR 10) and two text datasets (BBC and IMDB). Figures 7 and show the noisy label and feature data detection results the three non-tabular datasets, respectively. Figures 1h[sww the point removal and addition experiment results, respectively.
Overall, the findings presented in Section 4 are consistently observed in the non-tabular datasets. That is, Data-OOB shows promising empirical performance in detecting low-quality data points but turns out to be less effective in identifying beneficial data points. Shapley- similar performance each other, which is better than a random baseline. AME performs poorly in identifying low- www-qulltt data points, which is likely due to its sparse data values, but shows competitive performance in the point removal and addition experiments. InfluenceFunction and LOO outperform a random baseline in point removal/addition experiments, but they often perform worse than other data valuation algorithms.
methods exhibit
C.3 Robustness of findings against different prediction models
We conduct additional numerical experiments to demonstrate our empirical findings in Section 4 are robust against different prediction models. We compare the result with a logistic regression model
'Nte that Data- MOBB does not require the validation dataset and it does not use additional validation data points.
18

---

Dataset: CIFAR10
Dataset:
Dataset: IMDB
Dala-00
0.8
DVAL
Detathaphyy
KhNSShaple
0.4
6.6
DanaShapley
0.3
6.
DamBanahaa
Influenoeuucbbrn
0.2
0.11
LeareDraGuu
10 15
10
15
26
Random
Noise proportion (%)
Noise proportion (%)
Noise proportion (%)
Figure 7: Noisy label data detection on non-tabular datasets. Fi-score of different data valuation algorithms on the four noise proportion settings. The higher the FI-score is, the better the data valuation algorithm is. The error bar indicates a 95% confidence interval based on 50 independent experiments. Data-OOB significantly outperforms other methods in detecting mislabeled data points in various situations.
Dataset: CIFAR10
Dataset: BBC
Dataset: IMDB
Data-008
1.3
1.0
1.0
BotaShoplcy
3.8
KNVSSrapley
F1-score
DataShaplyy
0.6-
VoumeShapkyy
Delddannhal
0.4
0.4
0.4
62
02
AME
LeaweOneOut
20
Random
Noise proportion (%))
Noise proportion (%6)
Noise proportion
Figure 8: Noisy feature data detection on non-tabular datasets. Fl-score of different data valuation algorithms on the four noise proportion settings. The higher the Fi-score is, the better the data valuation algorithm is. The eтror bar indicates a 95% confidence interval based on 50 independent experiments. As in tabular datasets, LAVA shows remarkable performance compared to other data valuation algorithms.
with that of two prediction models: a decision tree model, denoted by TREE, and support vector machine, denoted by SVM. As for the datasets, we used 2dplanes, pol, fried, and IMDB. We consider every data valuation algorithm used in Section 4 but we excluded DataShapley and BetaShapley for IMDB due to their expensive computational costs.
Figures I1 and show that the Fl-score comparison on the four datasets noisy label data detection and noisy feature data detection, respectively. Note that the performances of both KNNShapley and LAVA do vary across different prediction models as their algorithms do rely on prediction model. As for other data valuation algorithms, while there are some variations, we observe an overall consistent trend across different prediction models: Data-OOB (resp. LAVA) outperforms other methods with large margin in the noisy label (resp. feature) data detection task. Another finding is that the performance of DVRL highly variable compared to other methods. This could be because of its dependence on complicated models and the instability of the reinforcement learning algorithm.
Every data valuation algorithm has various hyperparameters that could affect the performance by large margin. In our experiments, we followed the literature, and the default or suggested hyperparameters were employed. However, we acknowledge that it is possible that carefully tuned hyperparameters can yield better or different results. Comprehensive and extensive comparisons can be done with OpenDataVal Val, and it will be an important future research topic.
D Available APIs in OpenDataVal
D.1 Datasets
We separate the preprocessing of raw dataset (acquisition and parsing) and the preparation for numerical experiment (splitting, noisifying) using Register and DataFetcher object, respectively. A Register object downloads, caches, and preprocesses the raw dataset. The DataFetcher, which
19

---

Dataset: CIFAR10
Dataset: BBC
Dataset: IMDB
0.30
Date-Onn
accuracy.
8.60
0.62
0.86
0.84
0.74
0.72
Bid SolaSrapp
DataShoplcy
DVAL
Shaplyy
Testt
0.82
0.70
Vokumeb
PataBanzhef
0.56
inllenncouudtion
0.78
1.0
11.0 15.0
14.0
3.0
7.0
11.0 15.0 19.0
5.0
7.0
11.0 15.0 10:0
One
Proportion (%) of data removed
Proportion (36) of data removed
Proportion (86) of data removed
Random
Figure 9: Point removal experiment on non-tabular datasets. Test accuracy curves as a function of the most valuable data points removed. Lower curve indicates better data valuation algorithm. The error bar indicates 95% confidence interval based on 50 independent experiments. AME and Shapley-based methods, especially KNNShapley, exhibit superior performance.
Dataset: CIFAR10
Dataset: BBC
Dataset: IMDB
0.5
0.8
0.7
BetaSh-pley
Tast accuraty
0.4
Test accuracy
0.6
accurac
0.6
DataShapicy
KNNShappey
0.5
Shapme
0.4-
DataBanzhaf
0.2
InfuenceFunction
0.1
0.3-
AME
0.0
0.21
3.0 7.0
11.0
15.0
13.0
3.0
7.0
11.0
15.0
19.0
7.0
11.0 15.0
19.0
Proportion (96) data added
Propartion (16 of data ad ded
Propartian (56) data added
Figure 10: Point addition experiment non-tabular datasets. Test accuracy curves function of the least valuable data points added. Lower curve indicates better data valuation algorithm. The error bar indicates 95% confidence interval based 50 independent experiments. Data-OOB exhibits superior performance.
named to prevent collision with the PyTorch DataLoader, prepares dataset into the training, testing, and validation datasets for experiments.
Register The Register object allows assign unique string identifier to each dataset. For instance, OpenDataVal provides various image datasets ('mnist10", 'cifar10', 'cifar100', 'cifar10 embeddings') ) text datasets ('bbe-ebbeddings', 'imdb-mbeddding'', and tabular datasets ('iris', 'digits'. 'breast_cancer '2dplanes', 'electricity', 'MiniBooNE', 'pol', 'fried', 'nomao", and 'cred- itcard', just name few). We will keep updating list of available registered datasets. With the registered object, we can easily load the raw dataset, which allows us to have reproducible prepro- cessing. Also, as Code snippet demonstrates, users can register and preprocess their own datasets. All the registered datasets are in the "opendataval/dattlladeee datasets directory.
We highlight that two options available to access the image and text datasets. The first option the raw data using Py PyTchc data loader [26), which allows direct handling and processing of the data. Alternatively, pretrained DistilBERT embeddings for text datasets [30]. These embeddings provide compact
also offer pretrained ResNet50 embeddings for image datasets [1O] and
representation of the data, enabling efficient analysis and experimentation.
DataFetcher Once dataset is registered, becomes accessible through the DataFetcher. The Code demmostttdessssttttates how to retrieve registered dataset using a string identifier and split it into training, validation, and test datasets.
Additionally, we offer an alternative approach to retrieve datasets without the need for registration, as depicted in Code Snippet[5] For short-lived tasks, DataFetcher.from ffromddtaaaaaaaaaa allows you to add data from numpy arrays.
To add synthetic noise, we can use one of the functions from the "noisify" file in Code snippet 6 With this API, we can specify how to prepare datasets where the preprocessing needs to consistent across experiments.
20

---

from opendataval dataloader import Register
Register dataset
oRegister (name= custom "uustom__ataset"", one one_hot hea_hot-Tree True, Tacheable-Trus
def read_custon___ mustom_datasett.
# read a raw dataset
data pd.read_ccvv data_path.csv
preprocess coveriates and labels.
covariates, labels = data.loc(:: :1], data. 1ott:,oc[[,,1] return covariates, labels
Code snippet The Register feature simplifies the process of reading and preprocessing raw datasets. The following code snippet demonstrates how to read a custom CSV dataset from my working directory and preprocess it to have covariates and labels. The function component can be easily modified. Additionally, users have the flexibility to download datasets from the internet and apply various normalization techniques as needed.
from opendataval dataloader .ataallooddd import DataFetcher
Access registered dataset fetcher DataFetcher
Splitting and loading the data fetcher Setther.split_deee bellitdattaee_by__e x_train, y y_train, x_ x_valid, y y_vaid,, x test, y_ y_test fetcher. fetcher.datapoina datapoints
(300, 100, 100)
Code snippet 4: With DataFetcher, users can conveniently access registered datasets and randomly split them into training, validation, and test sets, thereby facilitating smooth and reliable execution of subsequent experiments.
D.2 Prediction Models
Data values are often computed in the context of how well they improve the prediction model performance. As such, many data valuation algorithms require class of prediction models. In OpenDataVal, one can take a prediction model from a comprehensive collection of PyTorch classi- fiers and regressors, empowering users with wide range of machine learning models for their data valuation tasks. Additionally, the framework provides convenient wrappers for scikit-learn models, enabling seamless integration of scikit-learn models into the OpenDataVal ecosystem. These wrap- pers ensure that seikit-learn models can be readily utilized and leveraged within the framework's 's data valuation pipeline.
To case use, OpenDataVal provides a ModelFactory that will load a prediction model from string identifier of its name. The Code snippet shows how to select and initialize model.
D.3 Data valuation algorithms
OpenDataVal provides eleven state-of-the of-the-ar data valuation algorithms in Table with unified API called DataEvaluator. All the data valuation algorithms are importable in the 'opendataval/ddataall directory. One can develop a new data valuation algorithm by simply inheriting from the provided abstract DataEvaluator base class. DataEvaluator often requires a performance metric and prediction model to be specified order to determine data values. However, KNNShapley only produces data values for KNN classifier. The injected prediction model will not be used compute data values but will retained for evaluation tasks.
21

---

from apendataval.datal.. dataloader import DataFetcher
Quick random regression task covariates, labels np random rand (500, ,),, np random. rand (500,2)
fetcher DataFetcher PataFetchher.froo.aataaCCaC wwt..o.aaaaaccovartatcco.oo labels, one_hot=Fale
Splitting and loading the data
fetcher split_dataset, by_iniicess ([i for i in range(300)),
for i range 400)], [i for i in range (400,500)]) x_train, y_train, x_valid, y y_valid, x_test, y_test fetcher datapoints
Code snippet 5: Even if a dataset is not registered, can readily loaded. This code snippet demonstrates how create DataFetcher object using 'numpy' arrays for covariates and labels allowing seamless access to the dataset. Also, this code snippet shows how to split the dataset by preset indicies.
from daaadataaal.dttllladee.... import [add_aass_noise, mix labels
Split data
fetcher wettcee.split_eeeee.. split_dattaeet_by_coutt (300, 100, 100)
Adds noise
fetcher poisify(mix__abee
noise_rate=0
fetcher fetcher.noisifiieeeee noisify _noss,, Paase_rate=-..0 mu=0, sigma=1)
Code snippet 6: In our benchmarking analysis, we utilized synthetically generated noisy dataset to assess the quality of data values. By employing the 'noisify' function, one can introduce label noise or feature noise.
from opendataval. modeel import ModelFactory
mooeel
Examlle 1: Loading Pyserch-based Logistic regression model
logistic_model DooeelPactorry"logssttttrr ("Iggssticreeggessiirr fetcher, device="cuda"
Ezample le 2: Loading scikit- learn model
linear_moder ModelFactory
("stttorry"""xiieeee"" ("sklinreg", fetcher, fit_inntercept-True)
Code snippet 7: Two examples of loading prediction models from PyTorch and scikit-learn us- ing a string identifier. Available string identifiers include 'logisticregression,, 'classifiermlp' 're- gressionmlp'. 'lenet', and 'sklogreg'. to name few. A full list of identifiers is defined in the "opendataval/modde/ init init__pppppp _py file.
22

---

Dataset: 2dplanes, Noise type: label noise
Data- 00B
BetaShapley
KNNShaplcy
DataShapley
VolumeShapley
0.4
DataBanzhhf
0.3
InfluenceRunction
0.2
AME
0.1
LeaveOneOut
Random
0.0
LOGISTIC
SVM
TREE
Prediction Model
Dataset: pol, Noise type: label noise
0.8
Dala-O0B
DVRL
0.7
BetaShapley
KNNShapley
DalaShapley
VolumeShapley
0.4-
Datalanzhhf
0.3
InfluenceFunctton
AME
LeaveOneOut
Rondom
LOGISTIC
SVM
TREF
Prediction Model
Dataset: fried, Noise type: label noise
DVRL
BetaShapley
KNNSShapee
Fisccoee
DataShapley
0.4
VolumeShapley
DataBanzhaf
InfluenceFunction
LNW
0.1
LoaveOneOut
Random
Prediction Model
Dataset: imdb-embeddings, Noise type: label_noise
0.6
Data-008
DVRL
KNNShapley
VolumeShapley
F1-score
DataBanzhaf
InfluenceFunce
LAVA
LeaveOneOut
Random
0.1
0.0
LOGISTIE
SVM
TREE
Prediction Model
Figure 11: Noisy label data detection across different prediction models. F1-score of different data valuation algorithms on the four datasets. The higher the Fl-score is, the better the data valuation algorithm is. Across different prediction models, Data-OOB generally outperforms other data valuation algorithms.
23

---

Dataset: 2dplanes, Noise type: feature noise
0.7
Data-O08
DVRL
0.6-
BetaShapley
KNNShapley
F1-score
DataShapley
0.4
ValumeShapley
DataBanzhaf
0.3
InfluenceFunction
0.2
AME
Leavedneout
Random
LOGISTIC
SVM
TREE
Prediction Model
Dataset: pol, Noise type: feature, noise
1.0
Dala-O0B
BetaShapley
KNNShapley
11-score
0.6
DalaShapley
VolumeShapley
DataBanzhaf
0.4
InfluenceFunction
AME
LeaveOneOut
Random
0.0
LOGISTIC
SVM
TREE
Prediction Model
Dataset: fried, Noise type: feature noise
Data-O0B
DVRL
0.6
BotaShapley
KNNShapley
0.5
DataShapley
VolumeShapley
DataBanzhaf
InfluenceFunctiio
LAWA
0.2
AME
0.1
LeaveOneOut
Random
LOGISTIC
SVM
Prediction Model
Dataset: imdb-embeddings, Noise type: ceature_noise
1.0
DVRL
0.8
KNNShapley
VolumeShaploy
F1-score
DataBanzhaf
0.6
InfluenceFunction
LAVA
0.4
LeaveOnrOut
Prediction Model
Figure 12: Noisy feature data detection across different prediction models. F1-score of different data valuation algorithms the four datasets. The higher the F1-score is, the better the data valuation algorithm is. LAVA significantly outperforms other data valuation algorithms.
24


Reviews:
[{"review_id": "c6QxNyumrO", "reviewer": "Reviewer_hFxZ", "summary": "", "questions": "", "limitations": "The authors have partially addressed the limitations.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The significance of the work lies in developing a benchmark for a trendy problem of data valuation, facilitating the comparison of different data valuation algorithms.\n\nIn addition, the paper is easy to follow and well-structured.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "fuNjJJYYEC", "reviewer": "Reviewer_CYt9", "summary": "", "questions": "", "limitations": "Data valuation algorithms may demonstrate different performance when the model used to compute the utility varies. This paper only uses the logistic regression as the underlying model. The paper adds some discussion or experiments about how different underlying model may impact the performance of different data valuation function.", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. [significance of the contribution & relevance to the broader research community] This paper provides a solid benchmark for evaluating data valuation functions because of the diverse dataset inclusion, and evaluation metrics (performance on downstream tasks). It provides the performance insights of state-of-arts algorithms as well as an easy-to-use API to evaluate newly developed algorithms. I believe this benchmark framework and the insights obtained can provide convenience to the broader research community for data diagnosing and data quality improvement.  \n2. [quality of research] This paper employs solid methodologies in benchmark establishment, experiments. Most design choices such as dataset selection, start-of-arts algorithms selections, downstream tasks are well justified. \n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "6voV7pz9Fp", "reviewer": "Reviewer_8KNo", "summary": "", "questions": "", "limitations": "- Authors address some current limitations and future directions in the conclusion section. \n- Currently, there is no discussion on the potential negative societal impacts of their work. One negativity could be the fact that for data valuation techniques described in this paper, data owners should share their dataset, which would have security/privcacy implications. ", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "This paper's stregths are as follows:\n\n- The paper is well written and easy to follow. \n- It provides the readers with a unified tool to implement, use and compare different data valuation metrics. \n- Most of the well-known model-based metrics are included. This paper could be a good starting point for a student researcher to learn about these different metrics.\n- Although the final insgihts (the fact that there is no \"one fits all\" type solution when it comes to data valuation metrics) are predictable and as observed in other works too, the downstream machine learning tasks used to draw these conclusions are sufficiently comprehensive for comparing the performance of different metrics.  ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Q57cNq2Efy", "reviewer": "Reviewer_2UEH", "summary": "", "questions": "", "limitations": "I didn't identify any major societal concerns of methods limitations that were missing here. As noted above, the discussion of societal impact is on the shorter side, but this is fair given the nature of the contribution.", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "In my view, this is overall a very strong submission and is a very good fit for the dataset and benchmarks track. The focus on open and reproducible benchmarking is likely to greatly benefit members of the ML research community interested in data valuation (for a variety of applications) and may attract additional interest in the topic area (i.e. by making it easier to get started with existing implemenations and by making it easier to evaluate new approaches).\n\nSignificance of contribution: high. While no new algorithms or techniques are proposed, this is an area where benchmarking, aggregating implementations, and standardizing practice can be extremely useful. I believe this paper/resource could become quite central in providing some collective scaffolding for data valuation research going forward.\n\nRelevance to broader research community: Data valuation is likely to be relevant to a good portion of the ML community. There is a great interest in interdisciplinary work using these approaches as well in economic (e.g. data markets) and sociology (e.g. data-related collective action). While there's not much discussion of applied uses here, I think the collective scaffolding benefits described above will flow downstream to \"applied data valuation\".\n\nQuality: implementation and writing are all high quality (more comments on this below).\n\nEthical and social implications: While the paper does not have much room allocated for explicitly discussing ethical/societal implications, I didn't see any major risks or concerns with the work. \n\nIt could be worth a brief line of discussion on how making data valuation more accessible might impact the data economy (e.g. could data values be used to exclude people from markets or other systems), but ultimately I think these questions will be part of a long line of future work. I think overall, making benchmarking better falls firmly into the camp of benefits (transparency, openness, replicability, etc.) outweighing speculative harms.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The main weaknesses highlighted by reviewers have been addressed by authors in the updated version of this work, notably, (1) model-based vs model-free data valuation metrics and (2) enabling hyper-parameter tuning in the codebase. One small comment is that Reviewer 2UEH notes the arbitrary selection of the datasets and the authors do not address this comment in their responses -- authors should consider adding more details as to the justification of the inclusion of specific datasets in an appendix. 

Thank you to all reviewers and authors for their timely and thoughtful engagement. Overall, this paper is clearly well above the acceptance threshold.
Decision: Accept (Poster)