SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Data valuation using reinforcement learning (DVRL) proposed by Yoon et al. [42] involves the utilization of reinforcement learning algorithms to compute data values. DVRL trains a model  $h: \mathcal{X} \times \mathcal{Y} \rightarrow [0,1]$  that outputs its importance weight by solving the following objective function.
$$
\min _ {h \in \mathcal {H}} \mathbb {E} [ (Y - f _ {h} (X)) ^ {2} ] \quad \mathrm {s . t .} \quad f _ {h} = \operatorname * {a r g m i n} _ {f \in \mathcal {F}} \mathbb {E} [ h (X, Y) (Y - f (X)) ^ {2} ],
$$
where  $\mathcal{H} := \{h : \mathcal{X} \times \mathcal{Y} \to [0,1]\}$  and  $\mathcal{F} := \{f : \mathcal{X} \to \mathcal{Y}\}$ . Then, the data value of  $(x,y)$  is computed as the importance weight  $h(x,y)$ . In OpenDataVal, we follow the hyperparameters used in Yoon et al. [42].
A.4 Out-of-bag estimate-based methods
Data-OOB by [18] is a distinctive data valuation algorithm, which uses the out-of-bag estimate to describe the quality of data. Suppose we have a bagging model that consists of  $B$  weak learner models: for  $b \in [B]$ , we denote the  $b$ -th weak learner by  $\hat{f}_b : \mathcal{X} \to \mathcal{Y}$ , which is trained on the  $b$ -th bootstrap dataset. It can be expressed as a minimizer of a weighted risk as follows.
$$
\hat {f} _ {b} := \operatorname * {a r g m i n} _ {f \in \mathcal {F}} \frac {1}{n} \sum_ {j = 1} ^ {n} w _ {b j} \ell (y _ {j}, f (x _ {j})),
$$
where  $\ell : \mathcal{Y} \times \mathcal{Y} \to \mathbb{R}$  is a loss function and  $w_{bj} \in \mathbb{Z}$  is the number of times the  $j$ -th datum  $(x_j, y_j)$  is selected in the  $b$ -th bootstrap dataset. With these notations, for  $i \in [n]$ , we propose to use the following quantity as data values.
$$
\psi_ {\mathrm {O O B}} \left(x _ {i}, y _ {i}\right) := \frac {\sum_ {b = 1} ^ {B} \mathbb {1} \left(w _ {b i} = 0\right) T \left(y _ {i} , \hat {f} _ {b} \left(x _ {i}\right)\right)}{\sum_ {b = 1} ^ {B} \mathbb {1} \left(w _ {b i} = 0\right)}. \tag {2}
$$
where  $T: \mathcal{Y} \times \mathcal{Y} \to \mathbb{R}$  is a score function that represents the goodness of a weak learner  $\hat{f}_b$  at the  $i$ -th datum  $(x_i, y_i)$ . For instance, we can use the correctness function  $T(y_i, \hat{f}_b(x_i)) = \mathbb{1}(y_i = \hat{f}_b(x_i))$  in classification settings and the negative Euclidean distance  $T(y_i, \hat{f}_b(x_i)) = -(y_i - \hat{f}_b(x_i))^2$  in regression settings.
B Tasks and their evaluation metrics
We formally introduce the four machine learning tasks that assess the effectiveness of data evaluation algorithms: (i) noisy label data detection, (ii) noisy feature data detection, (iii) point removal experiment, and (iv) point addition experiment.
B.1 Noisy label data detection
The main goal of noisy label data detection is to identify data points that have been mislabeled. We consider a synthetically generated mislabeled dataset: we randomly choose  $p_{\mathrm{noise}}\%$  of the training dataset and replace the original label with a different label. Let  $S^{(\mathrm{mislabeled})} \subseteq [n]$  be a set of indices of mislabeled data points. This annotation is not available to any data valuation algorithms and is only available when we assess the data valuation algorithms.
Let  $\phi : \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$  be a data valuation algorithm and denote a set of computed data values by  $\Phi_n := (\phi(x_1, y_1), \ldots, \phi(x_n, y_n))$ . We divide a set of  $n$  points  $\Phi_n$  into two clusters using the  $k$ -means clustering algorithm [2] and denote these two clusters of indices by  $S_{\phi}^{(\mathrm{low})}$  and  $S_{\phi}^{(\mathrm{high})}$ . That is,  $S_{\phi}^{(\mathrm{low})} \cup S_{\phi}^{(\mathrm{high})} = [n]$  and  $S_{\phi}^{(\mathrm{low})} \cap S_{\phi}^{(\mathrm{high})} = \{\}$ . Without loss of generality, we assume  $S^{(\mathrm{low})}$  has a lower mean than  $S_{\phi}^{(\mathrm{high})}$ , i.e.  $|S_{\phi}^{(\mathrm{low})}|^{-1} \sum_{i \in S_{\phi}^{(\mathrm{low})}} \phi(x_i, y_i) \leq |S_{\phi}^{(\mathrm{high})}|^{-1} \sum_{i \in S_{\phi}^{(\mathrm{high})}} \phi(x_i, y_i)$ . Since it is desirable for mislabeled data points to have lower values, we regard  $S_{\phi}^{(\mathrm{low})}$  as the prediction for the set of mislabeled data points. Then, we compute the F1-score between the prediction set  $S_{\phi}^{(\mathrm{low})}$  and the ground-truth annotation  $S^{(\mathrm{mislabeled})}$  as follows
$$
\mathrm {F 1} (S _ {\phi} ^ {(\mathrm {l o w})}, \mathcal {S} ^ {(\mathrm {m i s l a b e l e d})}) := 2 \frac {| S _ {\phi} ^ {(\mathrm {l o w})} \cap \mathcal {S} ^ {(\mathrm {m i s l a b e l e d})} |}{| S _ {\phi} ^ {(\mathrm {l o w})} | + | \mathcal {S} ^ {(\mathrm {m i s l a b e l e d})} |}.
$$
It is noteworthy that  $\mathrm{F1}(S_{\phi}^{(\mathrm{low})}, S^{\mathrm{(mislabeled)}})$  is the harmonic mean of the precision and recall as the precision is expressed as  $\frac{|S_{\phi}^{(\mathrm{low})} \cap S^{\mathrm{(mislabeled)}}|}{|S_{\phi}^{(\mathrm{low})}|}$  and the recall is expressed as  $\frac{|S_{\phi}^{(\mathrm{low})} \cap S^{\mathrm{(mislabeled)}}|}{|S^{\mathrm{(mislabeled)}}|}$ .
B.2 Noisy feature data detection
The main goal of noisy feature data detection is to identify data points with perturbed features. Similar to Section B.1, we consider a synthetically generated perturbed dataset: we randomly choose  $p_{\mathrm{noise}}\%$  of the training dataset and add some Gaussian random errors to every original feature. Let  $S^{(\mathrm{perturbed})} \subseteq [n]$  be a set of indices of perturbed data points. For each data valuation algorithm  $\phi$ , we compute  $S_{\phi}^{(\mathrm{low})}$ , and then we evaluate  $\mathrm{F1}(S_{\phi}^{(\mathrm{low})}, S^{(\mathrm{perturbed})})$  as in Section B.1.
B.3 Point removal experiment
Let  $\phi : \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$  be a data valuation algorithm and denote a set of computed data values by  $\Phi_n := (\phi(x_1, y_1), \ldots, \phi(x_n, y_n))$ . Suppose  $\pi = (\pi_1, \ldots, \pi_n) \in \mathbb{N}^n$  represents indices of data values in descending order, i.e.,  $\phi(x_{\pi_1}, y_{\pi_1}) \geq \dots \geq \phi(x_{\pi_n}, y_{\pi_n})$ . In the point removal experiment, we remove data points one by one in descending order from the entire training dataset. That is, we remove  $(x_{\pi_1}, y_{\pi_1})$  first and  $(x_{\pi_n}, y_{\pi_n})$  last. Each time the datum is removed, we fit a logistic model with the remaining dataset. For instance, after  $k$  data points are removed, we train a logistic model with  $\mathcal{D} \backslash \{(x_{\pi_j}, y_{\pi_j})\}_{j=1}^k$ . We denote the performance after  $k$  data points are removed by  $\mathrm{perf}_k$ . Then, for preset  $K \in \mathbb{N}$ , the point removal experiment evaluates the average performance after the removal of  $K$  points:
$$
\frac {1}{K} \sum_ {k = 1} ^ {K} \operatorname {p e r f} _ {k}.
$$
Throughout this paper, we consider  $K = \lfloor 0.2 \times n \rfloor$ , and the classification accuracy is used to compute the performance  $\mathrm{perf}_k$ . As we removed the most valuable data points first, a lower value indicates a better performance. The point removal curves in Figures 4 and 9 are  $(\mathrm{perf}_1, \dots, \mathrm{perf}_K)$ , and thus the evaluation metric represents the normalized area under the curve. In experiments, a coarser grid is used that evaluates perf every five-point removal, not one data point.
B.4 Point addition experiment
The point addition experiment is similar to the point removal experiment, but the only difference is that we add data points from the empty set in ascending order. That is, we first add first  $(x_{\pi_n}, y_{\pi_n})$
and  $(x_{\pi_1},y_{\pi_1})$  last. The evaluation metric for the point addition experiment is
$$
\frac {1}{K} \sum_ {k = 1} ^ {K} \operatorname {p e r f} _ {n - k}.
$$
We consider  $K = \lfloor 0.2 \times n \rfloor$ , and the classification accuracy is used for the performance  $\mathrm{perf}_k$ . A lower value indicates a better performance. The curves in Figures 5 and 10 are  $(\mathrm{perf}_{n-1}, \dots, \mathrm{perf}_{n-K})$ . See Section B.3 for more details.
C Additional Benchmarking Analysis
C.1 Missing details on experimental settings
Datasets For tabular datasets, a standard normalization procedure is employed, wherein each feature is centered and normalized to have a mean of zero and a standard deviation of one. However, for non-tabular datasets such as CIFAR10, BBC, and IMDB, no specific normalization procedures are applied except for the feature extraction. We use the pretrained ResNet50 model for CIFAR10 [10] and the pretrained DistilBERT model for texts [30]. Once the preprocessing steps are completed, following the literature [18], we split it into three datasets, namely, a training dataset, a validation dataset, and a test dataset. The data valuation process focuses on assessing the value of data within the training dataset, while the validation dataset is utilized for evaluating the utility function. The sample sizes for the training and validation datasets are set to 1000 and 100, respectively. The size of the test dataset is fixed at 3000 for all datasets, except for the text datasets (BBC and IMDB), where it is set to 500. The test dataset is used for point removal and addition experiments only when evaluating the test accuracy. As for the noisy feature data detection problem, we generate a random error from a Gaussian distribution with a mean of zero and a standard deviation of two. Although our benchmark analysis is fair and captures the gist of different data valuation algorithms, we encourage users to establish various experimental settings with OpenDataVal.
Data valuation algorithms We compare the eleven different data valuation algorithms listed in Table 1. Regarding the hyperparameters, we use the default settings provided by OpenDataVal, with most of the default values derived from the original research papers. There are some hyperparameters we choose for a fair and efficient comparison: for the KNNShapley, we set the number of  $k$ -nearest neighbors to  $10\%$  of training data points. In the case of DVRL, the model is trained for a total of 2000 epochs. Data-OOB, InfluenceFunction, AME, and DataBanzhaf have a shared key parameter 'the number of models to train', and it is set to the same number of 1000, throughout all experiments.
C.2 Additional Experimental Results
We show additional benchmarking analysis results using one image dataset (CIFAR10) and two text datasets (BBC and IMDB). Figures 7 and 8 show the noisy label and feature data detection results on the three non-tabular datasets, respectively. Figures 9 and 10 show the point removal and addition experiment results, respectively.
Overall, the findings presented in Section 4 are consistently observed in the non-tabular datasets. That is, Data-OOB shows promising empirical performance in detecting low-quality data points but turns out to be less effective in identifying beneficial data points. Shapley-based methods exhibit similar performance to each other, which is better than a random baseline. AME performs poorly in identifying low-quality data points, which is likely due to its sparse data values, but shows competitive performance in the point removal and addition experiments. InfluenceFunction and LOO outperform a random baseline in point removal/addition experiments, but they often perform worse than other data valuation algorithms.
C.3 Robustness of findings against different prediction models
We conduct additional numerical experiments to demonstrate our empirical findings in Section 4 are robust against different prediction models. We compare the result with a logistic regression model
Note that Data-OOB does not require the validation dataset and it does not use additional validation data points.
Figure 7: Noisy label data detection on non-tabular datasets. F1-score of different data valuation algorithms on the four noise proportion settings. The higher the F1-score is, the better the data valuation algorithm is. The error bar indicates a  $95\%$  confidence interval based on 50 independent experiments. Data-OOB significantly outperforms other methods in detecting unlabeled data points in various situations.
Figure 8: Noisy feature data detection on non-tabular datasets. F1-score of different data valuation algorithms on the four noise proportion settings. The higher the F1-score is, the better the data valuation algorithm is. The error bar indicates a  $95\%$  confidence interval based on 50 independent experiments. As in tabular datasets, LAVA shows remarkable performance compared to other data valuation algorithms.
with that of two prediction models: a decision tree model, denoted by TREE, and a support vector machine, denoted by SVM. As for the datasets, we used 2dplanes, pol, fried, and IMDB. We consider every data valuation algorithm used in Section 4, but we excluded DataShapley and BetaShapley for IMDB due to their expensive computational costs.
Figures 11 and 12 show that the F1-score comparison on the four datasets in noisy label data detection and noisy feature data detection, respectively. Note that the performances of both KNNShapley and LAVA do not vary across different prediction models as their algorithms do not rely on a prediction model. As for other data valuation algorithms, while there are some variations, we observe an overall consistent trend across different prediction models: Data-OOB (resp. LAVA) outperforms other methods with a large margin in the noisy label (resp. feature) data detection task. Another finding is that the performance of DVRL is highly variable compared to other methods. This could be because of its dependence on complicated models and the instability of the reinforcement learning algorithm.
Every data valuation algorithm has various hyperparameters that could affect the performance by a large margin. In our experiments, we followed the literature, and the default or suggested hyperparameters were employed. However, we acknowledge that it is possible that carefully tuned hyperparameters can yield better or different results. Comprehensive and extensive comparisons can be done with OpenDataVal, and it will be an important future research topic.
D Available APIs in OpenDataVal
We separate the preprocessing of a raw dataset (acquisition and parsing) and the preparation for a numerical experiment (splitting, noisifying) using a Register and DataFetcher object, respectively. A Register object downloads, caches, and preprocesses the raw dataset. The DataFetcher, which
Figure 9: Point removal experiment on non-tabular datasets. Test accuracy curves as a function of the most valuable data points removed. Lower curve indicates better data valuation algorithm. The error bar indicates a  $95\%$  confidence interval based on 50 independent experiments. AME and Shapley-based methods, especially KNNShapley, exhibit superior performance.
Figure 10: Point addition experiment on non-tabular datasets. Test accuracy curves as a function of the least valuable data points added. Lower curve indicates better data valuation algorithm. The error bar indicates a  $95\%$  confidence interval based on 50 independent experiments. Data-OOB exhibits superior performance.
is named to prevent collision with the PyTorch DataLoader, prepares the dataset into the training, testing, and validation datasets for experiments.
Register The Register object allows us to assign a unique string identifier to each dataset. For instance, OpenDataVal provides various image datasets ('mnist10', 'cifar10', 'cifar10-embeddings'), text datasets ('bbc-embeddings', 'imdb-embeddings'), and tabular datasets ('iris', 'digits', 'breast_cancer', '2dplanes', 'electricity', 'MiniBooNE', 'pol', 'fried', 'nomao', and 'creditcard', just name a few). We will keep updating a list of available registered datasets. With the registered object, we can easily load the raw dataset, which allows us to have reproducible preprocessing. Also, as Code snippet 3 demonstrates, users can register and preprocess their own datasets. All the registered datasets are in the 'opendataval/dataloger/datasets' directory.
We highlight that two options are available to access the image and text datasets. The first option is the raw data using a PyTorch data loader [26], which allows direct handling and processing of the data. Alternatively, we also offer pretrained ResNet50 embeddings for image datasets [10] and pretrained DistilBERT embeddings for text datasets [30]. These embeddings provide a compact representation of the data, enabling efficient analysis and experimentation.
DataFetter Once a dataset is registered, it becomes accessible through the DataFetter. The Code Snippet 4 demonstrates how to retrieve a registered dataset using a string identifier and split it into training, validation, and test datasets.
Additionally, we offer an alternative approach to retrieve datasets without the need for registration, as depicted in Code Snippets 5. For short-lived tasks, DataFetcher.from_data allows you to add data from numpy arrays.
To add synthetic noise, we can use one of the functions from the 'noisify' file in Code snippet 6. With this API, we can specify how to prepare datasets where the preprocessing needs to be consistent across experiments.
from opendataval.dataloader import Register #Register a dataset @Register(name  $=$  "custom_dataset",onehot  $\equiv$  True，cacheable  $\equiv$  True) def readCustom_dataset(): #readaraw dataset data  $=$  pd.read_csv('data_path.csv') #preprocess covariates and labels. covariates，labels  $=$  data.loc[:,-1]，data.loc[：,-1] return covariates，labels
Code snippet 3: The Register feature simplifies the process of reading and preprocessing raw datasets. The following code snippet demonstrates how to read a custom CSV dataset from my working directory and preprocess it to have covariates and labels. The function component can be easily modified. Additionally, users have the flexibility to download datasets from the internet and apply various normalization techniques as needed.
from opendataval.dataloader import DataFetter   
#Access a registered dataset   
fetcher  $=$  DataFetter(dataset_name  $\equiv$  "custom_dataset")   
# Splitting and loading the data   
fetcher.split_dataset_by_count(300,100,100)   
x_train，y_train，x_valid，y_valid，x_test，y_test  $=$  fetcher.datapoints
Code snippet 4: With Datachter, users can conveniently access registered datasets and randomly split them into training, validation, and test sets, thereby facilitating smooth and reliable execution of subsequent experiments.
D.2 Prediction Models
Data values are often computed in the context of how well they improve the prediction model performance. As such, many data valuation algorithms require a class of prediction models. In OpenDataVal, one can take a prediction model from a comprehensive collection of PyTorch classifiers and regressors, empowering users with a wide range of machine learning models for their data valuation tasks. Additionally, the framework provides convenient wrappers for scikit-learn models, enabling seamless integration of scikit-learn models into the OpenDataVal ecosystem. These wrappers ensure that scikit-learn models can be readily utilized and leveraged within the framework's data valuation pipeline.
To ease of use, OpenDataVal provides a ModelFactory that will load a prediction model from a string identifier of its name. The Code snippet 7 shows how to select and initialize a model.
D.3 Data valuation algorithms
OpenDataVal provides eleven state-of-the-art data valuation algorithms in Table 1 with a unified API called DataEvaluator. All the data valuation algorithms are importable in the 'opendataval/dataval/' directory. One can develop a new data valuation algorithm by simply inheriting from the provided abstract DataEvaluator base class. DataEvaluator often requires a performance metric and a prediction model to be specified in order to determine data values. However, KNNShapley only produces data values for a KNN classifier. The injected prediction model will not be used to compute the data values but will be retained for evaluation tasks.
from opendataval.dataloader import DataFetter   
#Quick random regression task covariates, labels  $=$  np.random RAND(500,5),np.random rand(500,2)   
fetcher  $\equiv$  DataFetter.from_data(covariates, labels, one-hot  $\equiv$  False)   
# Splitting and loading the data   
fetcher.split_dataset_by Indices([i for i in range(300)], [i for i in range(300,400)], [i for i in range(400,500)])   
x_train,y_train,x_valid,y_valid,x_test,y_test  $\equiv$  fetcher.datapoints
Code snippet 5: Even if a dataset is not registered, it can be readily loaded. This code snippet demonstrates how to create a DataFetcher object using 'numpy' arrays for covariates and labels allowing seamless access to the dataset. Also, this code snippet shows how to split the dataset by preset indices.
from opendataval.dataloader.noisify import add_gauss_noise, mix_labels
# Split data
fetcher.split_dataset_by_count(300, 100, 100)
# Adds noise
fetcher.noisify(mix_labels, noise_rate=0.2)
fetcher.noisify(add_gauss_noise, noise_rate=0.2, mu=0, sigma=1)
Code snippet 6: In our benchmarking analysis, we utilized a synthetically generated noisy dataset to assess the quality of data values. By employing the 'noisify' function, one can introduce label noise or feature noise.
from opendataval.model import ModelFactory
# Example 1: Loading PyTorch-based Logistic regression model
logistic_model = ModelFactory("logisticregression", docker, device="cuda")
# Example 2: Loading scikit-learn model
linear_model = ModelFactory("sklinreg", docker, fit_intercept=True)
Code snippet 7: Two examples of loading prediction models from PyTorch and scikit-learn using a string identifier. Available string identifiers include 'logisticregression', 'classifiermlp', 'regressionmlp', 'lenet', and 'sklogreg', to name a few. A full list of identifiers is defined in the 'opendataval/model/_init.py' file.
Dataset: 2dplanes, Noise type: label noise
Prediction Model
Data-OOB
DVRL
BetaShapley
KNNShapley
DataShapley
VolumeShapley
DataBanzhaf
InfluenceFunction
LAVA
AME
LeaveOneOut
Random
Dataset: pol, Noise type: label noise
Prediction Model
Dataset: fried, Noise type: label_noise
Prediction Model
Dataset: imdb-embeddings, Noise type: label noise
Prediction Model
Figure 11: Noisy label data detection across different prediction models. F1-score of different data valuation algorithms on the four datasets. The higher the F1-score is, the better the data valuation algorithm is. Across different prediction models, Data-OOB generally outperforms other data valuation algorithms.
Data-OOB
DVRL
KNNShapley
VolumeSha
DataBanzhaf
InfluenceFunction
LAVA
AME
LeaveOneOut
Random
Figure 12: Noisy feature data detection across different prediction models. F1-score of different data valuation algorithms on the four datasets. The higher the F1-score is, the better the data valuation algorithm is. LAVA significantly outperforms other data valuation algorithms.