SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The function round rounds a noisy value to the nearest integer. If the noisy value is out of range (e.g.,  $< 0$ ), then it is rounded to the nearest possible value within the valid range. By subsampling the data points, the privacy guarantee is amplified. This is because the subsampling process itself is a randomized operation that provides a level of privacy, making it harder to infer information about individual data points. When combined with the Gaussian mechanism, the overall privacy guarantee is improved.
The subsampling technique can be incorporated easily for DP-TKNN-Shapley as above, where the subsampling technique only improves the computational efficiency. As a comparison, incorporating subsampling technique for DP-KNN-Shapley will introduce a significantly higher computational cost, as we discussed in Appendix B.3.2.
D.3 Differentially private release of  $\phi_{z_i}^{\mathrm{TKNN}}(v_{D^{(\mathrm{val})}}^{\mathrm{TKNN}})$  (Privacy Accounting)
Recall that the TKNN-Shapley corresponding to the full validation set  $D^{(\mathrm{val})}$  is  $\phi_{z_i}^{\mathrm{TKNN}}(v_{D^{(\mathrm{val})}}^{\mathrm{TKNN}}) = \sum_{z^{(\mathrm{val})} \in D^{(\mathrm{val})}} \phi_{z_i}^{\mathrm{TKNN}}(D_{-z_i}; z^{(\mathrm{val})})$ . We can compute privatized  $\phi_{z_i}^{\mathrm{TKNN}}(v_{D^{(\mathrm{val})}}^{\mathrm{TKNN}})$  by simply releasing  $\phi_{z_i}^{\mathrm{TKNN}}(D_{-z_i}; z^{(\mathrm{val})})$  for all  $z^{(\mathrm{val})} \in D^{(\mathrm{val})3}$ . To better keep track of the overall privacy cost, we use the current state-of-the-art privacy accounting technique based on the notion of the Privacy Loss Random Variable (PRV) [17]. Here, we provide more details about the background of the composition of differential privacy and the privacy accounting techniques we use.
Background: Composition of Differential Privacy ("Privacy Accounting"). In practice, multiple differentially private mechanisms may be applied to the same dataset, denoted as  $\mathcal{M}(D) = \mathcal{M}_1 \circ \mathcal{M}_2(D) \coloneqq (\mathcal{M}_1(D), \mathcal{M}_2(D))$ .<sup>4</sup> Differential privacy offers strong composition guarantees, and the guarantees are derived by various composition theorems or privacy accounting techniques, including the basic composition theorem [14], advanced composition theorem [18], and Moments Accountant [1]. For example, the basic composition theorem states that if  $\mathcal{M}_1$  is  $(\varepsilon_1, \delta_1)$ -DP and  $\mathcal{M}_2$  is  $(\varepsilon_2, \delta_2)$ -DP, then the composition of  $\mathcal{M}_1$  and  $\mathcal{M}_2$  is  $(\varepsilon_1 + \varepsilon_2, \delta_1 + \delta_2)$ -DP. In our scenario, each individual mechanism is the release of  $\widehat{\phi}_{z_i}^{\mathrm{TKNN}}(D_{-z_i}; z^{(\mathrm{val})})$  for each of  $z^{(\mathrm{val})} \in D^{(\mathrm{val})}$ , and we would like to keep track of the privacy loss of releasing all of them.
Privacy Accounting Techniques based on Privacy Loss Random Variable (PRV). To better keep track of the privacy cost, we use the current state-of-the-art privacy accounting based on the notion of the Privacy Loss Random Variable (PRV) [17]. The PRV accountant was introduced in [37] and later refined in [36, 25]. For any DP algorithm, one can easily compute its  $(\varepsilon, \delta)$  privacy guarantee based on the distribution of its PRV. The key property of PRVs is that, under (adaptive) composition, they simply add up; the PRV  $Y$  of the composition  $\mathcal{M} = \mathcal{M}_1 \circ M_2 \circ \dots \circ M_k$  is given by  $Y = \sum_{i=1}^{k} Y_i$ , where  $Y_i$  is the PRV of  $\mathcal{M}_i$ . Therefore, one can then find the distribution of  $Y$  by convolving the distributions of  $Y_1, Y_2, \ldots, Y_k$ . Prior works [36, 25] approximate the distribution of PRVs by truncating and discretizing them, then using the Fast Fourier Transform (FFT) to efficiently convolve the distributions. In our experiment, we use the FFT-based accountant from [25], the current state-of-the-art privacy accounting technique.
${}^{3}$  Directly privatizing  ${\phi }_{{z}_{i}}^{\mathrm{{TKNN}}}\left( {v}_{{D}^{\left( \text{ val }\right) }}^{\mathrm{{TKNN}}}\right)$  is very difficult. Releasing more privatized statistics for the ease of privacy analysis is common in DP, e.g., DP-SGD [1] releases all privatized gradients.
Multiple DP mechanisms can be adaptively composed in the sense that the output of one mechanism can be used as an input to another mechanism, i.e.,  $\mathcal{M}(D) = \mathcal{M}_1\circ \mathcal{M}_2(D)\coloneqq (\mathcal{M}_1(D),\mathcal{M}_2(D,\mathcal{M}_1(D)))$
E Additional Experiment Settings & Results
E.1 Datasets
A comprehensive list of datasets and sources is summarized in Table 7. Similar to the existing data valuation literature [23, 39, 30, 60], we preprocess datasets for ease of training. For Fraud, Creditcard, and all datasets from OpenML, we subsample the dataset to balance positive and negative labels. For these datasets, if they have multi-class, we binarize the label by considering  $\mathbb{1}[y = 1]$ . For the image dataset MNIST, CIFAR10, we apply a ResNet50 [27] that is pre-trained on the ImageNet dataset as the feature extractor. This feature extractor produces a 1024-dimensional vector for each image. We employ sentence embedding models [54] to extract features for the text classification dataset AGNews and DBPedia, and the extracted features are 1024-dimensional vectors for each text instance. We perform L2 normalization on the extracted features as part of the pre-processing step.
The size of each dataset we use is shown in Table 7. For some of the datasets, we use a subset of the full set. We stress that we are using a much larger size of the dataset compared with prior data valuation literature [30, 39, 60] (these works usually only pick a very small subset of the full dataset, e.g., 2000 data points for binarized CIFAR10). The validation data size we use is  $10\%$  of the training data size.
Dataset	Number of classes	Size of dataset	Source
MNIST	10	50000	[41]
CIFAR10	10	50000	[38]
AGnews	4	10000	[65]
DBPedia	14	10000	[2]
Click	2	2000	https://www.openml.org/d/1218
Fraud	2	2000	[11]
Creditcard	2	2000	[71]
Apsfail	2	2000	https://www.openml.org/d/41138
Phoneme	2	2000	https://www.openml.org/d/1489
Wind	2	2000	https://www.openml.org/d/847
Pol	2	2000	https://www.openml.org/d/722
CPU	2	2000	https://www.openml.org/d/761
2DPlanes	2	2000	https://www.openml.org/d/727
Table 7: A summary of datasets used in Section 6's experiments.
E.2 Settings & Additional Experiments for Runtime Experiments
For the runtime comparison experiment in Section 6.1, we follow similar experiment settings from prior study [40] and use a synthetic binary classification dataset. To generate the synthetic dataset, we sample data points from a  $d$ -dimensional standard Gaussian distribution. The labels are assigned based on the sign of the sum of the two features.
We choose a range of training data sizes  $N$ , and compare the runtime of both TKNN/KNN-Shapley at each  $N$ . We use 100 validation data points. For Figure 2 in the maintext, the data dimension  $d = 10$ . Here, we show additional experiment results when  $d = 100$ . As we can see from Figure 5, again TKNN-Shapley achieves better computational efficiency than KNN-Shapley across all training data sizes, and is around  $30\%$  faster than KNN-Shapley for large  $N$ .
E.3 Settings & Additional Experiments for Distinguishing Data Quality Experiments
E.3.1 Details for Mislabel/Noisy Data Detection Experiment
In the experiment of mislabeled (or noisy) data detection, we randomly choose  $10\%$  of the data points and flip their labels (or add strong noise to their features). For mislabel data detection, we flip  $10\%$  of the labels by picking an alternative label from the rest of the classes uniformly at random. For noisy data detection, we add zero-mean Gaussian noise to data features, where the standard deviation of the
Figure 5: Runtime comparison of TKNN-Shapley and KNN-Shapley across various training data sizes  $N$  when data dimension  $d = 100$ . The plot shows the average runtime based on 5 independent runs. Experiments were conducted with an AMD 64-Core CPU Processor.
Gaussian noise added to each feature dimension is equal to the average absolute value of the feature dimension across the full dataset.
E.3.2 Additional Experiment Results for Private Setting
Table 8 provides a comprehensive comparison of DP-TKNN-Shapley and DP-KNN-Shapley. We conducted the experiments on DP-KNN-Shapley with subsampling on only half of the datasets due to the considerable computational cost (at least  $30 \times$  longer to execute compared to its non-subsampled counterpart). Additionally, because DP-KNN-Shapley injects Gaussian noise independently into the value score of each data holder, it does not have the property of being robust against collusion.
As evidenced by the results, DP-TKNN-Shapley consistently demonstrates a markedly superior privacy-utility tradeoff compared to DP-KNN-Shapley without subsampling across all datasets. Even when comparing with DP-KNN-Shapley with subsampling, DP-TKNN-Shapley still shows superior performance, and often significantly outshining this time-intensive and collusion-susceptible baseline. Notably, DP-TKNN-Shapley manages to maintain high AUROC values even when  $\varepsilon \approx 0.1$ . The low performance of DP-KNN-Shapley can be attributed to its relatively high global sensitivity, as we discussed in Appendix B.3.1.
E.3.3 Additional Experiment Results for Non-private Setting
Here, we show full experimental results for various data valuation techniques in mislabel/noisy data detection scenarios, summarized in Tables 9 and 10. In line with the main text findings, TKNN-Shapley exhibits performance largely on par with KNN-Shapley across the majority of datasets, demonstrating that TKNN-Shapley effectively mirrors KNN-Shapley's ability to discern data quality. In addition, both KNN-Shapley and TKNN-Shapley significantly outperform traditional retrain-based data valuation techniques (LOO, Data Shapley, and Data Banzhaf), corroborating observations made in existing studies [31, 52]. The inferior performance of Data Shapley can be attributed to sample inefficiency and the inherent randomness during the retraining process [60].
Settings for LOO, Data Shapley and Data Banzhaf. Following the experiment settings in [23, 39] we use logistic regression as the ML model for these techniques. For Data Shapley, we use permutation sampling [48] for estimation. For Data Banzhaf, we use the maximum-sample-reuse (MSR) algorithm [60] for estimation. For each dataset, we train models on 10,000 subsets (which takes more than  $1000 \times$  longer time compared with both TKNN/KNN-Shapley).
E.3.4 Ablation Study on the Choice of Threshold for TKNN-Shapley
KNN-Shapley is known for being effective on a wide range of choices of  $K$  [29]. In this experiment, we study the impact of different choices of  $\tau$  on the performance of TKNN-Shapley on the tasks of mislabeled/noisy data detection. The results are shown in Table 11 and 12. As we can see, the
performance of TKNN-Shapley is quite stable on a wide range of choices of  $\tau$ . For example, for 2dPlanes dataset, the performance of TKNN-Shapley only varies  $< 0.032$  between  $\tau \in [-0.1, -0.6]$ . Moreover,  $\tau \approx -0.5$  is a relatively robust choice of the threshold value. Hence, hyperparameter tuning is not difficult for TKNN-Shapley. This is not too surprising; the formula of TKNN-Shapley (in Equation (7)) shows that regardless of the choice of  $\tau$ , if a training data point is within the threshold but has a different label from the validation data, then its value is necessarily negative. At the same time, the points that share the same label as validation data are guaranteed to have a positive value. The choice of  $\tau$  only decides the magnitude but not the sign. As data valuation methods only rely on the rank of data values to perform data selection, the performance of TKNN-Shapley is relatively robust to the choice of  $\tau$  in identifying mislabeled data points (who get negative values). A similar argument also applies to the task of noisy data detection: noisy data is typically far from the population; hence, as long as  $\tau$  is not too large, the noisy data will receive a value score close to 0. On the other hand, clean data will receive a positive value score, and the choice of  $\tau$  only affects the magnitude of the data value but not the sign.
Ideally, we would like to set a threshold such that for most of the queries, there are a certain amount of neighbors whose distance to the query is below the threshold. One possible way to do this is simply picking a  $\tau$  that optimizes the final validation accuracy of the TKNN classifier (note that TKNN is very efficient in hyperparameter tuning).
Dataset	TKNN-Shapley	KNN-Shapley	Data Shapley	LOO	Data Banzhaf
2DPlanes	0.919	0.913	0.552	0.458	0.548
Phoneme	0.826	0.873	0.525	0.484	0.604
CPU	0.946	0.932	0.489	0.496	0.513
Fraud	0.96	0.967	0.488	0.495	0.534
Creditcard	0.662	0.646	0.517	0.487	0.536
Apsfail	0.958	0.948	0.496	0.485	0.506
Click	0.572	0.568	0.474	0.504	0.528
Wind	0.889	0.896	0.469	0.456	0.512
Pol	0.871	0.928	0.512	0.538	0.473
MNIST	0.962	0.974	-	-	-
CIFAR10	0.957	0.991	-	-	-
AG News	0.956	0.971	-	-	-
DBPedia	0.981	0.991	-	-	-
Table 9: Full Table for the AUROC of Mislabel Data Detection (non-private setting). The results for LOO, Data Shapley, Data Banzhaf are omitted for MNIST, CIFAR10, AGNews and DBPedia as they require a significant amount of runtime (prior works typically use a subset for MNIST and CIFAR10 [23, 39, 60]).
Dataset	TKNN-Shapley	KNN-Shapley	Data Shapley	LOO	Data Banzhaf
2DPlanes	0.705	0.718	0.545	0.504	0.656
Phoneme	0.71	0.706	0.49	0.531	0.631
CPU	0.751	0.726	0.527	0.459	0.531
Fraud	0.818	0.794	0.502	0.526	0.582
Creditcard	0.611	0.64	0.535	0.513	0.506
Apsfail	0.841	0.666	0.533	0.465	0.566
Click	0.558	0.547	0.461	0.503	0.572
Wind	0.691	0.794	0.495	0.488	0.54
Pol	0.68	0.79	0.526	0.503	0.588
MNIST	0.613	0.815	-	-	-
CIFAR10	0.85	0.935	-	-	-
AG News	0.804	0.746	-	-	-
DBPedia	0.719	0.88	-	-	-
Table 10: Full Table for the AUROC of Noisy Data Detection (non-private setting). The results for LOO, Data Shapley, Data Banzhaf are omitted for MNIST, CIFAR10, AGNews and DBPedia as they require a significant amount of runtime (prior works typically use a subset for MNIST and CIFAR10 [23, 39, 60]).
For completeness, we also show the impact of different choices of  $K$  on the performance of KNN-Shapley in Table 13 and 14. As we can see, KNN-Shapley's performance is quite stable against the different choices of  $K$ .
Dataset	τ = -0.1	τ = -0.2	τ = -0.3	τ = -0.4	τ = -0.5	τ = -0.6	τ = -0.7	τ = -0.8	τ = -0.9
2dPlanes	0.889	0.902	0.911	0.918	0.92	0.887	0.862	0.725	0.553
CPU	0.952	0.956	0.954	0.952	0.95	0.943	0.917	0.895	0.834
Phoneme	0.79	0.799	0.805	0.811	0.825	0.834	0.844	0.859	0.861
Fraud	0.961	0.956	0.951	0.954	0.959	0.957	0.931	0.884	0.75
Creditcard	0.653	0.671	0.668	0.669	0.662	0.652	0.648	0.606	0.539
Apsfail	0.959	0.952	0.962	0.967	0.962	0.952	0.924	0.899	0.851
Click	0.576	0.552	0.557	0.56	0.568	0.556	0.547	0.538	0.522
Wind	0.875	0.879	0.882	0.885	0.888	0.889	0.879	0.888	0.775
Pol	0.847	0.854	0.864	0.871	0.875	0.878	0.89	0.878	0.86
Table 11: Impact of different choices of  $\tau$  for TKNN-Shapley on the performance of mislabel data detection.
Dataset	τ = -0.1	τ = -0.2	τ = -0.3	τ = -0.4	τ = -0.5	τ = -0.6	τ = -0.7	τ = -0.8	τ = -0.9
2dPlanes	0.671	0.691	0.704	0.696	0.705	0.694	0.678	0.626	0.528
CPU	0.691	0.687	0.696	0.709	0.751	0.777	0.789	0.786	0.75
Phoneme	0.681	0.688	0.691	0.698	0.71	0.718	0.716	0.725	0.72
Fraud	0.758	0.781	0.804	0.809	0.818	0.837	0.847	0.814	0.664
Creditcard	0.579	0.576	0.585	0.597	0.611	0.624	0.642	0.626	0.574
Apsfail	0.771	0.822	0.834	0.855	0.841	0.842	0.828	0.792	0.749
Click	0.53	0.544	0.537	0.539	0.558	0.563	0.521	0.521	0.539
Wind	0.592	0.612	0.637	0.663	0.691	0.727	0.747	0.754	0.719
Pol	0.594	0.638	0.643	0.661	0.68	0.682	0.712	0.721	0.699
Table 12: Impact of different choices of  $\tau$  for TKNN-Shapley on the performance of noisy data detection.
Dataset	K=1	K=2	K=3	K=4	K=5	K=6	K=7	K=8	K=9	K=10	K=11	K=12	K=13	K=14	K=15
2dPlanes	0.914	0.914	0.915	0.915	0.916	0.915	0.915	0.914	0.915	0.915	0.914	0.91	0.905	0.898	0.896
CPU	0.936	0.936	0.937	0.937	0.937	0.937	0.938	0.937	0.936	0.934	0.932	0.929	0.928	0.927	0.921
Phoneme	0.859	0.861	0.862	0.862	0.864	0.865	0.868	0.868	0.869	0.871	0.871	0.871	0.871	0.874	0.86
Fraud	0.972	0.972	0.971	0.971	0.972	0.972	0.972	0.971	0.971	0.971	0.97	0.967	0.968	0.968	0.966
Creditcard	0.652	0.652	0.653	0.653	0.653	0.651	0.65	0.651	0.651	0.649	0.646	0.642	0.638	0.637	0.634
Apsfall	0.952	0.952	0.952	0.952	0.951	0.951	0.949	0.948	0.947	0.947	0.948	0.949	0.949	0.95	0.95
Click	0.56	0.562	0.564	0.563	0.566	0.566	0.563	0.562	0.562	0.563	0.567	0.566	0.557	0.547	0.553
Wind	0.9	0.9	0.9	0.9	0.9	0.899	0.899	0.899	0.899	0.898	0.896	0.897	0.896	0.891	0.89
Pol	0.931	0.931	0.931	0.931	0.93	0.929	0.929	0.93	0.929	0.927	0.925	0.922	0.919	0.914	0.917
Table 13: Impact of different choices of  $K$  for KNN-Shapley on the performance of mislabel data detection.
Dataset	K=1	K=2	K=3	K=4	K=5	K=6	K=7	K=8	K=9	K=10	K=11	K=12	K=13	K=14	K=15
2dPlanes	0.719	0.719	0.718	0.718	0.718	0.717	0.716	0.715	0.714	0.712	0.712	0.709	0.704	0.703	0.703
CPU	0.7	0.702	0.703	0.705	0.706	0.706	0.709	0.711	0.717	0.723	0.73	0.736	0.744	0.751	0.766
Phoneme	0.718	0.721	0.722	0.724	0.726	0.729	0.732	0.735	0.738	0.741	0.739	0.74	0.739	0.74	0.742
Fraud	0.786	0.788	0.789	0.792	0.794	0.794	0.795	0.797	0.797	0.799	0.803	0.805	0.811	0.815	0.822
Creditcard	0.637	0.638	0.636	0.638	0.64	0.642	0.643	0.644	0.646	0.646	0.645	0.642	0.636	0.636	0.64
Apsfail	0.667	0.667	0.667	0.666	0.666	0.666	0.666	0.668	0.671	0.674	0.678	0.683	0.69	0.711	0.763
Click	0.544	0.547	0.552	0.549	0.547	0.549	0.549	0.547	0.548	0.544	0.544	0.543	0.543	0.544	0.545
Wind	0.796	0.796	0.795	0.794	0.794	0.793	0.792	0.791	0.791	0.79	0.788	0.788	0.788	0.789	0.791
Pol	0.789	0.788	0.788	0.788	0.79	0.79	0.788	0.787	0.786	0.786	0.787	0.79	0.786	0.786	0.785
Table 14: Impact of different choices of  $K$  for KNN-Shapley on the performance of noisy data detection.