SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kevin Fu Jiang*
Weixin Liang*
Yongchan Kwon†
Assessing the quality and impact of individual data points is critical for improving model performance and mitigating undesirable biases within the training dataset. Several data valuation algorithms have been proposed to quantify data quality, however, there lacks a systemic and standardized benchmarking system for data valuation. In this paper, we introduce OpenDataVal, an easy-to-use and unified benchmark framework that empowers researchers and practitioners to apply and compare various data valuation algorithms. OpenDataVal provides an integrated environment that includes (i) a diverse collection of image, natural language, and tabular datasets, (ii) implementations of eleven different state-of-the-art data valuation algorithms, and (iii) a prediction model API that can import any models in scikit-learn. Furthermore, we propose four downstream machine learning tasks for evaluating the quality of data values. We perform benchmarking analysis using OpenDataVal, quantifying and comparing the efficacy of state-of-the-art data valuation approaches. We find that no single algorithm performs uniformly best across all tasks, and an appropriate algorithm should be employed for a user's downstream task. OpenDataVal is publicly available at https://opendataval.github.io with comprehensive documentation. Furthermore, we provide a leaderboard where researchers can evaluate the effectiveness of their own data valuation algorithms.
Real-world data often exhibit heterogeneity and noise as they are collected from a wide range of sources and are susceptible to measurement and annotation errors [25, 19, 21]. When such data are incorporated into model development, they can introduce undesirable biases and potentially hinder model performance. Managing and addressing these challenges is crucial to ensure the reliability and accuracy of the insights derived from real-world data. As such, it is becoming increasingly important to understand and evaluate the intrinsic properties of data, such as data quality, data bias, and their influences on the model training process [14, 20]. From all these motivations, data valuation has emerged as a principled evaluation framework that aims to quantify the impact of individual data points on model predictions or model performance.
There have been substantial recent efforts in developing different data valuation methods, and they have demonstrated promising outcomes in various real-world applications. One widely used data valuation method is DataShapley [7], which is based on the Shapley value from game theory [31]. This method has been applied to detect low-quality chest X-ray image data [35] and design a data
\*Co-first author
† Corresponding author (e-mail: yk3012@columbia.edu)
Figure 1: OpenDataVal is an open-source, easy-to-use and unified benchmark framework for data valuation. It contains a diverse collection of datasets, implementations of eleven state-of-the-art data valuation algorithms, and a variety of modern prediction models. In addition, OpenDataVal provides several downstream tasks for evaluating the quality of data valuation algorithms.
marketplace [1, 36]. BetaShapley generalizes DataShapley by relaxing the efficiency axiom in the Shapley value and has demonstrated its efficacy in identifying mislabeled images in the CIFAR-100 test dataset [17]. Recently, Ilyas et al. [11] propose datamodels and provide qualitative analyses of the train-test leakage that could hinder fair evaluations. Apart from these methods, many data valuation algorithms have been developed to capture different aspects of data values [42, 18] or improve computational efficiency [40]. Nevertheless, there has been little attention to establishing a user-friendly and standardized benchmarking system, and this is the main goal of this work.
Our contributions We introduce OpenDataVal, an easy-to-use and unified benchmark framework that allows researchers and practitioners to apply and compare different data valuation algorithms with a few lines of Python codes. OpenDataVal provides an integrated environment that includes (i) a diverse collection of image, natural language, and tabular datasets that are publicly available at open platforms OpenML [37], scikit-learn [27], and PyTorch [26], (ii) implementations of eleven different state-of-the-art data valuation algorithms in Table 1, and (iii) a prediction model API that enables to import any machine learning models in scikit-learn and PyTorch-based neural network models. In addition, OpenDataVal provides four downstream machine learning tasks for evaluating the quality of data values, noisy label data detection, noisy feature data detection, point removal experiment, and point addition experiment. We also demonstrate how OpenDataVal can be employed through benchmarking analysis in Section 4. We find no single data valuation algorithm performs uniformly superior across all the benchmark tasks. That is, different algorithms have their own advantages and capture different aspects of data values. We encourage users to choose an appropriate algorithm for their analysis.
OpenDataVal is designed to advance the transparency and reproducibility of data valuation works and to foster user engagement in data valuation. To support this endeavor, we have made OpenDataVal open-source at https://github.com/opendataval/opendataval, enabling anyone to contribute to our GitHub repository. Additionally, we have developed a leaderboard where users can participate in noisy data detection competitions. OpenDataVal will be regularly updated and maintained by the authors and participants.
Related works Sim et al. [33] conducted a technical survey of data valuation algorithms, focusing on the necessary assumptions and their desiderata. While their analysis provides mathematical insights, it lacks a comprehensive empirical comparison. Our framework primarily focuses on this problem, creating an easy-to-use benchmarking system that facilitates empirical comparisons. Valda<sup>3</sup> is a Python package with a set of data valuation algorithms including LOO, DataShapley [7], and BetaShapley [17]. OpenDataVal provides most of the data valuation algorithms available in Valda, plus additional methods, as well as an easy-to-use environment in which practitioners
<sup>3</sup>https://uvanlp.org/valda/
Algorithms	Underlying Method	Quality of Data Values
Noisy Label Detection	Noisy Feature Detection	Point Removal	Point Addition
LOO	Marginal contribution	-	-	+	+
DataShapley [7]	Marginal contribution	+	+	++	++
KNNShapley [12]	Marginal contribution	+	+	++	++
Volume-based Shapley [41]	Marginal contribution	-	-	-	-
BetaShapley [17]	Marginal contribution	+	+	++	++
DataBanzhaf [39]	Marginal contribution	-	-	+	+
AME [22]	Marginal contribution	-	-	++	+
InfluenceFunction [5]	Gradient	-	-	+	+
LAVA [13]	Gradient	-	++	+	+
DVRL [42]	Importance weight	+	-	+	++
Data-OOB [18]	Out-of-bag estimate	++	+	-	++
Table 1: A taxonomy of data valuation algorithms available in OpenDataVal. LOO, KNN Shapley, Volume-based Shapley, DataShapley, BetaShapley, DataBanzhaf, and AME can be expressed as functions of marginal contributions. There have been alternative approaches to defining data values: the gradient, importance weights, and out-of-bag estimate. KNNShapley only works for KNN predictors; the other data valuation methods are compatible with any prediction models. The 'Quality of Data Values' section summarizes the benchmark results in Section 4. The symbols  $\frac{ - / + }{} + +$  ’ indicate that a corresponding data valuation method achieves a 'similar or worse / better / much better' performance than a random baseline, respectively. There is no single method that uniformly outperforms others in every task. Therefore, it is critical for users to selectively consider data valuation algorithms for their own downstream tasks.
and data analysts compute and evaluate various state-of-the-art data valuation algorithms in diverse downstream tasks.
DataPerf [24] has provided six realistic benchmark tasks: training set creation, test set creation, selection algorithm, debugging algorithm, slicing algorithm, and valuation algorithm. These tasks cover important data-centric problems that can arise in the model development process, but we emphasize that our main objective differs from that of DataPerf work. While they mainly focus on creating benchmark tasks and competitions, our main objective is to provide a unified environment for quantifying and comparing various data valuation algorithms.
2 A taxonomy of data valuation methods
In this section, we introduce a taxonomy of data valuation methods. For  $d \in \mathbb{N}$ , we denote an input space and an output space by  $\mathcal{X} \subseteq \mathbb{R}^d$  and  $\mathcal{Y} \subseteq \mathbb{R}$ , respectively. We denote a training dataset by  $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^n$  where  $x_i \in \mathcal{X}$  and  $y_i \in \mathcal{Y}$  are an input and its label for the  $i$ -th datum. Then, all data valuation methods can be seen as a mapping that assigns a scalar score to each sample in  $\mathcal{D}$ , and each quantifies the impact of the point on the performance of a model trained on  $\mathcal{D}$ .
For instance, the leave-one-out (LOO) is defined as  $\phi_{\mathrm{LOO}}(x_i,y_i)\coloneqq U(\mathcal{D}) - U(\mathcal{D}\backslash \{(x_i,y_i)\})$  where  $U$  represents a preset utility function that takes as input a subset of the training dataset. In classification settings, a common choice for  $U$  is the test accuracy of a model trained on the input. In this case, the  $i$ -th LOO value  $\phi_{\mathrm{LOO}}(x_i,y_i)$  measures the change in the test accuracy when the  $i$ -th data point  $(x_{i},y_{i})$  is removed from the training procedure.
This concept of LOO has been generalized to the marginal contribution, which captures the average change in utility when a certain datum is removed from a set with a given cardinality. To be more specific, for a given cardinality  $1 \leq j \leq n$ , the marginal contribution of  $(x_i, y_i) \in \mathcal{D}$  with respect to  $j$  samples is defined as  $\Delta_j(x_i, y_i) := \binom{n-1}{j-1}^{-1} \sum_{S \subseteq \mathcal{D}_j^{(x_i, y_i)}} U(S \cup \{(x_i, y_i)\}) - U(S)$  where  $\mathcal{D}_j^{(x_i, y_i)} := \{S : S \subseteq \mathcal{D} \backslash \{(x_i, y_i)\}, |S| = j-1\}$ . Many data valuation methods are expressed as a function of marginal contributions, for instance, DataShapley [7] considers the simple average of marginal contributions  $\phi_{\mathrm{shap}}(x_i, y_i) := \frac{1}{n} \sum_{j=1}^n \Delta_j(x_i, y_i)$  and BetaShapley [17] considers a weighted average of marginal contributions  $\phi_{\mathrm{beta}}(x_i, y_i) := \sum_{j=1}^n w_{\mathrm{beta},j} \Delta_j(x_i, y_i)$  for some
weight vector  $(w_{\mathrm{beta},1},\dots ,w_{\mathrm{beta},n})$  . LOO, DataBanzhaf [39], DataModels [11], and the average marginal effect (AME) [22] are also included in this category, and they differ in how to combine marginal contributions into a single score.
There are alternative approaches to describing the concept of data values: gradient, importance weight, and out-of-bag estimate. The gradient methods are to quantify the rate at which a utility value changes when a particular data point is more weighted. The influence function [5] and LAVA [13] are included in this category. They differ depending on what utility value is being measured: the influence function uses the validation accuracy as a utility while LAVA uses the optimal transport cost.
The importance weight method learns a weight function  $w: \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$  so that an associated model  $f_w$  achieves a good accuracy on the holdout validation dataset. Here, the model  $f_w: \mathcal{X} \to \mathbb{R}$  is often obtained by minimizing a weighted risk  $\mathbb{E}[w(X,Y)\ell(Y,f(X))]$  for some loss function  $\ell: \mathcal{Y} \times \mathcal{Y} \to \mathbb{R}$  and a preset weight function  $w$ . Data valuation using reinforcement learning (DVRL) [42] is in this category, and they proposed a reinforcement learning technique to obtain a weight function.
An out-of-bag (OOB) estimation approach measures the contribution of each data point to out-of-bag accuracy when a bagging model is employed. Data-OOB [18] is in this category, and it captures how much a nominal label differs from predictions generated by weak learners. We provide a rigorous definition of each data valuation method along with their properties in Appendix. A taxonomy of data valuation algorithms with a brief summary of empirical results is presented in Table 1.
3 Overview of OpenDataVal
OpenDataVal provides a unified API called ExperimentMediator, which enables the quantification of data values and systemic comparative experiments. With ExperimentMediator, users can specify various datasets, data valuation algorithms, and hyperparameters (e.g., a prediction model, model performance metric, and synthetic noise). The following code snippet illustrates how to compute data values using the embedding of the CIFAR10 dataset. Here, the embedding is obtained from the penultimate layer of the ResNet50 model [10] pretrained on the ImageNet dataset [4].
from opendataval.experiment import ExperimentMediator  
exper_med = ExperimentMediator.modelFACTORY_setup( dataset_name="cifar10-embedding", model_name="ClassifierMLP", metric_name="accuracy", output_dir="outputs/", add_noise="mix_labels", noise_kwarges={'noise_rate':0.2} )  
datavaluators = [DataShapley(), DVRL()]  
exper_med = exper_med.compute_data_values(datavaluators)
Code snippet 1: This code snippet shows how to compute DataShapley [7] and DVRL [42] values for the CIFAR-10 dataset using a multilayer perceptron classifier as a prediction model, with classification accuracy as the evaluation metric. With 'add_noise' and 'noise_kwarges' arguments, a synthetic noise has been applied:  $20\%$  of the training dataset is randomly chosen and their original labels have been changed to different labels.
In addition, OpenDataVal provides a collection of evaluation tasks as demonstrated in Section 4. Using the computed data values in the previous step, users can conduct noisy label data detection tasks to systematically evaluate the performance of data valuation algorithms.
OpenDataVal provides a convenient way to compute and compare various state-of-the-art data valuation algorithms with a few lines of Python codes. It is also designed to be highly extensible and easily adaptable to custom experiment settings. We provide details on customization in Appendix.
Leaderboards OpenDataVal introduces the public leaderboards to promote transparency and healthy competition in the field of data valuation. In the initial release, we provide a noisy label data detection task for a selection of datasets. The challenge datasets, accessible within the OpenDataVal platform, contain label noise that is unknown to the user. The goal is to develop their own data valuation algorithms and find which data points have had noise added. Users can submit their
from opendataval.experiments import noisy_prediction  
# perform noisy label data detection and save results  
exper_med.evaluate(noisy_prediction, save_output=True)  
>>> kmeans_f1  
>>> DataShapley() 0.5625  
>>> DVRL() 0.6825
Code snippet 2: This code snippet showcases the task of noise label data detection and evaluates the F1-scores of the DataShapley and DVRL algorithms.
Dataset	Sample Size	Input Dimension	Number of Classes	Minor Class Proportion	Data Type	Source
electricity	38474	6	2	0.5	Tabular	[6]
fried	40768	10	2	0.498	Tabular	OpenGL-901
2dplanes	40768	10	2	0.499	Tabular	OpenGL-727
pol	15000	48	2	0.336	Tabular	OpenGL-722
MiniBooNE	72998	50	2	0.5	Tabular	[29]
nomao	34465	89	2	0.285	Tabular	[3]
bbc-embedding	2225	768	5	0.17	Text	[8]
IMDB-embedding	50000	768	2	0.5	Text	[23]
CIFAR10-embedding	50000	2048	10	0.1	Image	[16]
Table 2: A summary of classification datasets used in the benchmarking analysis. For natural language and image datasets, the pretrained DistilBERT [30] and ResNet50 [10] models are employed to extract an embedding. For 'fried', '2dplanes', and 'pol' datasets, the number indicates their OpenML-Data-ID. All datasets are readily available in OpenDataVal.
algorithm outputs in the form of a CSV file, and our interactive ranking system will periodically update the leaderboard, showcasing the performance of different submissions. Submissions and competition details are available at https://opendataval.github.io/leaderboards.html.
4 Benchmarking Analysis
We present the practical applications of OpenDataVal through benchmarking analysis. We conduct the four machine learning tasks: noisy label data detection, noisy feature data detection, point addition experiment, and point removal experiment. These experiments have been widely used in the literature [7, 42, 13]. We provide details on evaluation metrics in Appendix. All the datasets, data valuation algorithms, prediction models, and downstream tasks employed in the benchmarking analysis are readily accessible within OpenDataVal, ensuring the transparency and reproducibility of all benchmark results.
Experimental settings We use the nine datasets in Table 2. These datasets are standard and widely used within the literature [7, 18]. Due to space limitations in the main text, we primarily focus on the analysis of the six tabular datasets, but our main findings are consistently observed across different modalities and datasets. Results for the text and image datasets are provided in Appendix. We compare the eleven different data valuation algorithms in Table 1. Additionally, we include a baseline algorithm, which assigns random numbers to data values. The InfluenceFunction, Volume-based Shapley, DataBanzhaf, AME, and Data-OOB require training models multiple times, and the number of trained models has a substantial impact on both performance and runtime. To ensure a fair comparison, we have set this parameter to a fixed number of 1000. As for the other algorithms that do not require training models multiple times, we maintain fair comparisons by utilizing hyperparameters similar to those specified in the original papers. As for the base prediction model, we use a logistic regression model. However, it is important to note that the logistic regression model is not compatible with KNNShapley as it is implicitly based on KNN predictors and has a closed-form expression. The sample size for the training dataset is set to  $n = 1000$ . Implementation details are provided in Appendix.
Figure 2: Noisy label data detection. F1-score of different data valuation algorithms on the four noise proportion settings. The higher the F1-score is, the better the data valuation algorithm is. The error bar indicates a  $95\%$  confidence interval based on 50 independent experiments. Data-OOB demonstrates significantly superior performance in detecting mislabeled data points in various situations.
4.1 Noisy data detection
We compare the detection capabilities of different data valuation algorithms using synthetically generated noisy datasets. We consider two types of synthetic noise. The first type is label noise where we flip the original label to its opposite label. The second type is feature noise where we add standard Gaussian random errors to the original features. After selecting one of two noises, we then randomly choose  $p_{\mathrm{noise}}\%$  of the training dataset to perturb. Here, the four different levels of noise proportion  $p_{\mathrm{noise}} \in \{5,10,15,20\}$  is considered.
Data values are computed for this noisy dataset, and we investigate which data valuation algorithm is the most effective to identify these noisy data points. Based on the literature [39, 17, 18], we follow the approach of applying the  $k$ -means clustering algorithm to the data values [2]. This allows us to dichotomize the training dataset into two groups: beneficial and detrimental. We assign the detrimental label to the cluster with a lower average of data values. Given our expectation that mislabeled data points are likely to have low values, we predict data points within the detrimental group as noisy samples. We evaluate the F1-score between this prediction and the ground-truth annotation. Note that we do not use ground-truth annotations for noisy data points when computing data values. We only use them when we evaluate the quality of data valuation algorithms.
Noisy label data detection Figure 2 shows the F1-score of data valuation algorithms for the noisy label data detection tasks. The results can be grouped into three clusters based on their performance. The first cluster consists of a single algorithm, Data-OOB. It achieves significantly better results than other data valuation algorithms on most of the datasets and noise proportions. For instance, on the 'MiniBooNE' dataset with  $p_{\mathrm{noise}} = 20\%$ , Data-OOB shows 0.65 F1-score when the second best algorithm KNNShapley shows 0.53 and the random baseline shows 0.29. The second cluster consists of DVRL, KNNShapley, DataShapley, and BetaShapley. They show reasonably sound detection ability compared to the random baseline. Volume-based Shapley, LAVA, LOO, InfluenceFunction, AME, and DataBanzhaf belong to the third cluster and show similar performance to the random baseline.
Noisy feature data detection Figure 3 shows the F1-score of different data valuation algorithms for the noisy feature data detection tasks. In contrast to the noisy label data detection, LAVA generally outperforms other methods with a substantial performance gap. This difference is potentially due to the transport cost function, one of the key hyperparameters in LAVA [13]. The default choice is designed to be sensitive to noises in features, leading to a good performance in noisy feature
Figure 3: Noisy feature data detection. F1-score of different data valuation algorithms on the four noise proportion settings. The higher the F1-score is, the better the data valuation algorithm is. The error bar indicates a  $95\%$  confidence interval based on 50 independent experiments. LAVA generally outperforms other data valuation algorithms with a big performance gap.
data detection. However, we observe that LAVA's performance varies a lot across different datasets, suggesting that this choice of the transport cost function is important. Following LAVA, Data-OOB and Shapley-based methods demonstrate a better performance than the random baseline, showing their potential efficacy in noisy feature data detection. Volume-based Shapley, DVRL, InfluenceFunction, AME, and LOO show only comparable performance to the random baseline in this task.
4.2 Point removal and addition experiments
Data values can be used to find influential samples. To evaluate this quantitatively, we perform the point addition and removal experiments used in [7, 18]. The point removal experiment is performed with the following steps: For each data valuation algorithm, we remove data points from the entire training dataset in descending order of the data values. Each time the datum is removed, we fit a logistic regression model with the remaining dataset and evaluate its test accuracy on the holdout dataset. As we remove the data points in descending order, in the ideal case we remove the most helpful data points first, and thus model accuracy is expected to decrease. For the point addition experiment, we perform a similar procedure but add data points in ascending order. Similar to the point removal experiment, the model accuracy is expected to be low as we add detrimental data points first. Throughout the experiments, we used a perturbed dataset with label noise. The noise ratio is set to  $p_{\mathrm{noise}} = 20\%$ . The sample size of the holdout test dataset is set to 3000. All the procedures here can be easily performed in OpenDataVal.