SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Considering the privacy concerns and privatization challenges associated with the original KNN-Shapley method, we introduce TKNN-Shapley, a privacy-friendly alternative of KNN-Shapley which also achieves improved computational efficiency. At the core of this novel method is Threshold-KNN (TKNN) classifier, a simple variant of the KNN classifier. We will discuss how to incorporate DP for TKNN-Shapley in Section 5.
4.1 Threshold-based Nearest Neighbor Classifier (TKNN)
Threshold-KNN (TKNN) [5, 72] is a variant of KNN classifier that considers neighbors within a pre-specified threshold of the query example, rather than exclusively focusing on the exact  $K$  nearest neighbors. Formally, for a training set  $S$  and a validation data point  $z^{(\mathrm{val})} = (x^{(\mathrm{val})},y^{(\mathrm{val})})$ , we denote  $\mathbb{N}\mathbb{B}_{x^{(\mathrm{val})},\tau}(S)\coloneqq \{(x,y)|(x,y)\in S,d(x,x^{(\mathrm{val})})\leq \tau \}$  the set of neighbors of  $x^{(\mathrm{val})}$  in  $S$  within a pre-specified threshold  $\tau$ . Similar to the utility function for KNN, we define the utility function for TKNN classifier when using a validation set  $D^{(\mathrm{val})}$  as the aggregated prediction accuracy  $v_{D^{(\mathrm{val})}}^{\mathrm{TKNN}}(S)\coloneqq \sum_{z^{(\mathrm{val})}\in D^{(\mathrm{val})}}v_{z^{(\mathrm{val})}}^{\mathrm{TKNN}}(S)$  where
$$
v _ {z (\text {v a l})} ^ {\mathrm {T K N N}} (S) := \left\{ \begin{array}{l l} \text {C o n s t a n t} & | \mathrm {N B} _ {x (\text {v a l}), \tau} (S) | = 0 \\ \frac {1}{| \mathrm {N B} _ {x (\text {v a l}) , \tau} (S) |} \sum_ {(x, y) \in \mathrm {N B} _ {x (\text {v a l}) , \tau} (S)} \mathbb {1} [ y = y ^ {(\text {v a l})} ] & | \mathrm {N B} _ {x (\text {v a l}), \tau} (S) | > 0 \end{array} \right. \tag {3}
$$
where Constant can be the trivial accuracy of random guess.
Comparison with standard KNN. (1) Robustness to outliers. Compared with KNN, TKNN is better equipped to deal with prediction phase outliers [5]. When predicting an outlier that is far from the entire training set, TKNN prevents the influence of distant, potentially irrelevant neighbors, leading to a more reliable prediction score for outliers. (2) Inference Efficiency. TKNN has slightly better computational efficiency compared to KNN, as it has  $O(N)$  instead of  $O(N\log N)$  inference time. This improvement is achieved because TKNN only requires the computation of neighbors within the threshold  $\tau$ , rather than searching for the exact  $K$  nearest neighbors. (3) TKNN is also a consistent estimator. The consistency of standard KNN is a well-known theoretical result [26]. That is, for any target function that satisfies certain regularity conditions, KNN binary classifier/regressor is guaranteed to converge to the target function as the size of the training set grows to infinite. In Appendix C.1, we derived a similar consistency result for TKNN binary classifier/regressor.
Remark 3 (Intuition: Why we consider TKNN?). The 'recursive form' of KNN-Shapley is due to the sorting operation in the prediction of the standard KNN. The recursive form of the formula causes difficulties in incorporating KNN-Shapley with differential privacy. In contrast, TKNN avoids the recursive formula for its Data Shapley value; the intuition is that for TKNN, the selection of neighbors for prediction solely depends on the queried example and the validation data, and is independent of the other training data points.
4.2 Data Shapley for TKNN (TKNN-Shapley)
With the introduction of TKNN classifier and its utility function, we now present our main result, the closed-form, efficiently computable Data Shapley formula for the TKNN classifier.
Theorem 5 (TKNN-Shapley (simplified version)). Consider the utility function  $v_{z(\mathrm{val})}^{TKNN}$  in (3). Given a validation data point  $z^{\mathrm{(val)}} = (x^{\mathrm{(val)}},y^{\mathrm{(val)}})$  and a distance metric  $d(\cdot ,\cdot)$ , the Shapley value  $\phi_{z_i}^{TKNN}$  of each training point  $z_{i} = (x_{i},y_{i})\in D$  corresponding to utility function  $v_{z(\mathrm{val})}^{TKNN}$  can be calculated as
$$
\phi_ {z _ {i}} ^ {T K N N} = f \left(\boldsymbol {C} _ {z (\text {v a l})} (D _ {- z _ {i}})\right) \tag {4}
$$
where  $C_{z^{(\mathrm{val})}}\coloneqq (\mathbf{c},\mathbf{c}_{x^{(\mathrm{val})},\tau},\mathbf{c}_{z^{(\mathrm{val})},\tau}^{(+)})$  is a 3-dimensional function/vector s.t.
$$
\mathbf {c} = \mathbf {c} \left(D _ {- z _ {i}}\right) := \left| D _ {- z _ {i}} \right| \quad \left(\text {s i z e o f} D _ {- z _ {i}}\right)
$$
$$
\mathbf {c} _ {x ^ {(\text {v a l})}, \tau} = \mathbf {c} _ {x ^ {(\text {v a l})}, \tau} (D _ {- z _ {i}}) := 1 + | N B _ {x ^ {(\text {v a l})}, \tau} (D _ {- z _ {i}}) | \quad (1 + \# \text {n e i g h b o r s o f} x ^ {(\text {v a l})} \text {i n} D _ {- z _ {i}})
$$
$$
\mathbf {c} _ {z ^ {(\text {v a l})}, \tau} ^ {(+)} = \mathbf {c} _ {z ^ {(\text {v a l})}, \tau} ^ {(+)} (D _ {- z _ {i}}) := \sum_ {(x, y) \in N B _ {x ^ {(\text {v a l})}, \tau} (D _ {- z _ {i}})} \mathbb {1} [ y = y ^ {(\text {v a l})} ] \quad (\# \text {s a m e - l a b e l n e i g h b o r s i n} D _ {- z _ {i}})
$$
and  $f(\cdot)$  is a function whose exact form can be found in Appendix C.2.
Runtime: the computation of all  $(\phi_{z_1}^{TKNN},\dots,\phi_{z_N}^{TKNN})$  can be achieved in  $O(N)$  runtime (see Appendix C.2.1).
The Shapley value when using full validation set: The Shapley value corresponding to the utility function  $v_{D(\mathrm{val})}^{\mathrm{TKNN}}$  can be calculated as  $\phi_{z_i}^{\mathrm{TKNN}}(v_{D(\mathrm{val})}^{\mathrm{TKNN}}) = \sum_{z(\mathrm{val}) \in D(\mathrm{val})} \phi_{z_i}^{\mathrm{TKNN}}(v_{z(\mathrm{val})}^{\mathrm{TKNN}})$ .
The main technical challenges of proving Theorem 5 lies in showing  $\mathbf{C}_{z^{(\mathrm{val})}} = (\mathbf{c},\mathbf{c}_{x^{(\mathrm{val})},\tau},\mathbf{c}_{z^{(\mathrm{val})},\tau}^{(+)})$ , three simple counting queries on  $D_{-z_i}$ , are the key quantities for computing  $\phi_{z_i}^{\mathrm{TKNN}}$ . In later part of the paper, we will often view  $\phi_{z_i}^{\mathrm{TKNN}}$  as a function of  $\mathbf{C}_{z^{(\mathrm{val})}}$  and denote  $\phi_{z_i}^{\mathrm{TKNN}}[\mathbf{C}_{z^{(\mathrm{val})}}] := f(\mathbf{C}_{z^{(\mathrm{val})}})$  when we want to stress this dependency. The full version and the proof for TKNN-Shapley can be found in Appendix C.2.1. Here, we briefly discuss why TKNN-Shapley can achieve  $O(N)$  runtime in total.
Efficient Computation of TKNN-Shapley. As we can see, all of the quantities in  $\mathbf{C}_{z^{(\mathrm{val})}}$  are simply counting queries on  $D_{-z_i}$ , and hence  $\mathbf{C}_{z^{(\mathrm{val})}}(D_{-z_i})$  can be computed in  $O(N)$  runtime for any  $z_i \in D$ . A more efficient way to compute  $\mathbf{C}_{z^{(\mathrm{val})}}(D_{-z_i})$  for all  $z_i \in D$ , however, is to first compute  $\mathbf{C}_{z^{(\mathrm{val})}}(D)$  on the full dataset  $D$ , and we can easily show that each of  $\mathbf{C}_{z^{(\mathrm{val})}}(D_{-z_i})$  can be computed from that in  $O(1)$  runtime (see Appendix C.2.2 for details). Hence, we can compute TKNN-Shapley  $\phi_{z_i}^{\mathrm{TKNN}}(v_{z^{(\mathrm{val})}}^{\mathrm{TKNN}})$  for all  $z_i \in D$  within an overall computational cost of  $O(N)$ .
Comparison with KNN-Shapley. TKNN-Shapley offers several advantages over the original KNN-Shapley. (1) Non-recursive: In contrast to the KNN-Shapley formula (Theorem 2), which is recursive, TKNN-Shapley has an explicit formula for computing the Shapley value of every point  $z_{i}$ . This non-recursive nature not only simplifies the implementation, but also makes it straightforward to incorporate techniques like subsampling. (2) Computational efficiency: TKNN-Shapley has  $O(N)$  runtime in total, which is better than the  $O(N\log N)$  runtime for KNN-Shapley.
5 Differentially Private TKNN-Shapley
Having established TKNN-Shapley as an alternative to KNN-Shapley which shares many advantages, our next step is to develop a new version of TKNN-Shapley that incorporates differential privacy. In this section, we introduce our proposed Differentially Private TKNN-Shapley (DP-TKNN-Shapley).
5.1 Differentially private release of  $\phi_{z_i}^{\mathrm{TKNN}}(v_{z(\mathrm{val})}^{\mathrm{TKNN}})$
As  $\mathbf{C}_{z^{(\mathrm{val})}} = (\mathbf{c},\mathbf{c}_{x^{(\mathrm{val})},\tau},\mathbf{c}_{z^{(\mathrm{val})},\tau}^{(+)})$  are the only quantities that depend on the dataset  $D_{-z_i}$  in (4), privatizing these quantities is sufficient for overall privacy preservation, as DP guarantee is preserved under any post-processing [16]. Since all of  $\mathbf{c},\mathbf{c}_{x^{(\mathrm{val})},\tau}$  and  $\mathbf{c}_{z^{(\mathrm{val})},\tau}^{(+)}$  are just counting queries, the function of  $\mathbf{C}_{z^{(\mathrm{val})}}(\cdot)$  has global sensitivity  $\sqrt{3}$  in  $\ell_2$ -norm. This allows us to apply the commonly used Gaussian mechanism (Theorem 4) to release  $\mathbf{C}_{z^{(\mathrm{val})}}$  in a differentially private way, i.e.,  $\widehat{\mathbf{C}}_{z^{(\mathrm{val})}}(D_{-z_i}) := \text{round}(\mathbf{C}_{z^{(\mathrm{val})}}(D_{-z_i}) + \mathcal{N}(0,\sigma^2\mathbb{1}_3))$  where  $\widehat{\mathbf{C}}_{z^{(\mathrm{val})}}$  is the privatized version of  $\mathbf{C}_{z^{(\mathrm{val})}}$ , and the function round rounds each entry of noisy value to the nearest integer. The differentially private version of TKNN-Shapley value  $\widehat{\phi}_{z_i}^{\mathrm{TKNN}}(D_{-z_i})$  is simply the value score computed by
(4) but use the privatized quantities  $\widehat{\mathbf{C}}_{z(\mathrm{val})}$ . The privacy guarantee of releasing DP-TKNN-Shapley  $\widehat{\phi}_{z_i}^{\mathrm{TKNN}}(D_{-z_i})$  follows from the guarantee of Gaussian mechanism (see Appendix D.1 for proof).
Theorem 6. For any  $z^{\mathrm{(val)}}\in D^{\mathrm{(val)}}$ , releasing  $\widehat{\phi}_{z_i}^{TKNN}(v_{z(\mathrm{val})}^{TKNN})\coloneqq \phi_{z_i}^{TKNN}\left[\widehat{C}_{z(\mathrm{val})}(D_{-z_i})\right]$  to data owner  $i$  is  $(\varepsilon ,\delta)$ - $DP$  with  $\sigma = \sqrt{3}\cdot \sqrt{\log(1.25 / \delta)} /\varepsilon$ .
While our approach may seem simple, we stress that simplicity is appreciated in DP as complex mechanisms pose challenges for correct implementation and auditing [47, 21].
5.2 Advantages of DP-TKNN-Shapley (Overview)
We give an overview (detailed in Appendix D.2) of several advantages of DP-KNN-Shapley here, including the efficiency, collusion resistance, and simplicity in incorporating subsampling.
By reusing privatized statistics,  $\widehat{\phi}_{z_i}^{\mathrm{TKNN}}$  can be efficiently computed for all  $z_i \in D$ . Recall that in practical data valuation scenarios, it is often desirable to compute the data value scores for all of  $z_i \in D$ . As we detailed in Section 4.2, for TKNN-Shapley, such a computation only requires  $O(N)$  runtime in total if we first compute  $\mathbf{C}_{z(\mathrm{val})}(D)$  and subsequently  $\mathbf{C}_{z(\mathrm{val})}(D_{-z_i})$  for each  $z_i \in D$ . In a similar vein, we can efficiently compute the DP variant  $\widehat{\phi}_{z_i}^{\mathrm{TKNN}}$  for all  $z_i \in D$ . To do so, we first calculate  $\widehat{\mathbf{C}}_{z(\mathrm{val})}(D)$  and then, for each  $z_i \in D$ , we compute  $\widehat{\mathbf{C}}_{z(\mathrm{val})}(D_{-z_i})$ . It is important to note that when releasing  $\widehat{\phi}_{z_i}^{\mathrm{TKNN}}$  to individual  $i$ , the data point they hold,  $z_i$ , is not private to the individual themselves. Therefore, as long as  $\widehat{\mathbf{C}}_{z(\mathrm{val})}(D)$  is privatized,  $\widehat{\mathbf{C}}_{z(\mathrm{val})}(D_{-z_i})$  and hence  $\widehat{\phi}_{z_i}^{\mathrm{TKNN}}$  also satisfy the same privacy guarantee due to DP's post-processing property.
By reusing privatized statistics, DP-TKNN-Shapley is collusion resistance. In Section 5.1, we consider the single Shapley value  $\phi_{z_i}^{\mathrm{TKNN}}(D_{-z_i})$  as the function to privatize. However, when we consider the release of all Shapley values  $(\widehat{\phi}_{z_1}^{\mathrm{TKNN}},\ldots ,\widehat{\phi}_{z_N}^{\mathrm{TKNN}})$  as a unified mechanism, one can show that such a mechanism satisfies joint differential privacy (JDP) [34] if we reuse the privatized statistic  $\widehat{\mathbf{C}}_{z(\mathrm{val})}(D)$  for the release of all  $\widehat{\phi}_{z_i}^{\mathrm{TKNN}}$ . The consequence of satisfying JDP is that our mechanism is resilient against collusion among groups of individuals without any privacy degradation. That is, even if an arbitrary group of individuals in  $[N]\setminus i$  colludes (i.e., shares their respective  $\widehat{\phi}_{z_j}^{\mathrm{TKNN}}$  values within the group), the privacy of individual  $i$  remains uncompromised. Our method also stands resilient in scenarios where a powerful adversary sends multiple data points and receives multiple value scores.
Incorporating Privacy Amplification by Subsampling. In contrast to KNN-Shapley, the nonrecursive nature of DP-TKNN-Shapley allows for the straightforward incorporation of subsampling. Besides the privacy guarantee, subsampling also significantly boosts the computational efficiency of DP-TKNN-Shapley.
5.3 Differentially private release of  $\phi_{z_i}^{\mathrm{TKNN}}(v_{D(\mathrm{val})}^{\mathrm{TKNN}})$
Recall that the TKNN-Shapley corresponding to the full validation set  $D^{(\mathrm{val})}$  is  $\phi_{z_i}^{\mathrm{TKNN}}(v_{D^{(\mathrm{val})}}^{\mathrm{TKNN}}) = \sum_{z^{(\mathrm{val})} \in D^{(\mathrm{val})}} \phi_{z_i}^{\mathrm{TKNN}}(D_{-z_i}; z^{(\mathrm{val})})$ . We can compute privatized  $\phi_{z_i}^{\mathrm{TKNN}}(v_{D^{(\mathrm{val})}}^{\mathrm{TKNN}})$  by simply releasing  $\phi_{z_i}^{\mathrm{TKNN}}(D_{-z_i}; z^{(\mathrm{val})})$  for all  $z^{(\mathrm{val})} \in D^{(\mathrm{val})}$ . To better keep track of the privacy cost, we use the current state-of-the-art privacy accounting technique based on the notion of the Privacy Loss Random Variable (PRV) [17]. We provide more details about the background of privacy accounting in Appendix D.3. We note that PRV-based privacy accountant computes the privacy loss numerically, and hence the final privacy loss has no closed-form expressions.
In this section, we systematically evaluate the practical effectiveness of our proposed TKNN-Shapley method. Our evaluation aims to demonstrate the following points: (1) TKNN-Shapley offers improved runtime efficiency compared with KNN-Shapley. (2) The differentially private version of TKNN-Shapley (DP-TKNN-Shapley) achieves significantly better privacy-utility tradeoff compared to naively privatized KNN-Shapley in discerning data quality. (3) Non-private TKNN-Shapley maintains a comparable performance to the original KNN-Shapley. These observations highlight TKNN-Shapley's potential for data valuation in real-life applications. Detailed settings for our experiments are provided in Appendix E.
Remark 4. Given that differential privacy offers a provable guarantee against any potential adversaries, our experiments prioritize evaluating the utility of DP-TKNN-Shapley rather than its privacy properties. However, for readers interested in understanding the efficacy of DP in safeguarding training data, we have included an evaluation of the proposed MI attack on DP-TKNN-Shapley in Appendix B.2.2, where it shows a significant drop in attack performance compared to non-DP version.
6.1 Computational Efficiency
We evaluate the runtime efficiency of TKNN-Shapley in comparison to KNN-Shapley (see Appendix E.2 for details as well as more experiments). We choose a range of training data sizes  $N$ , and compare the runtime of both methods at each  $N$ . As demonstrated in Figure 2, TKNN-Shapley achieves better computational efficiency than KNN-Shapley across all training data sizes. In particular, TKNN-Shapley is around  $30\%$  faster than KNN-Shapley for large  $N$ . This shows the computational advantage of TKNN-Shapley mentioned in Section 4.2.
6.2 Discerning Data Quality
In this section, we evaluate the performance of both DP and non-DP version of TKNN-Shapley in discerning data quality. Tasks: We evaluate the performance of TKNN-Shapley on two standard tasks: mislabeled data detection and noisy data detection, which are tasks that are often used for evaluating the performance of data valuation tech-
Figure 2: Runtime comparison of TKNN-Shapley and KNN-Shapley across various training data sizes  $N$ . The plot shows the average runtime based on 5 independent runs. Experiments were conducted with an AMD 64-Core CPU Processor.
niques in prior works [39, 60]. Since mislabeled/noisy data often negatively affect the model performance, it is desirable to assign low values to these data points. In the experiment of mislabeled (or noisy) data detection, we randomly choose  $10\%$  of the data points and flip their labels (or add strong noise to their features). Datasets: We conduct our experiments on a diverse set of 13 datasets, where 11 of them have been used in previous data valuation studies [39, 60]. Additionally, we experiment on 2 NLP datasets (AG News [65] and DBPedia [2]) that have been rarely used in the past due to their high-dimensional nature and the significant computational resources required. Settings & Hyperparameters of TKNN-/KNN-Shapley: for both TKNN/KNN-Shapley, we use the popular cosine distance as the distance measure [53], which is always bounded in  $[-1, +1]$ . Throughout all experiments, we use  $\tau = -0.5$  and  $K = 5$  for TKNN-/KNN-Shapley, respectively, as we found the two choices consistently work well across all datasets. We conduct ablation studies on the choice of hyperparameters in Appendix E.
6.2.1 Experiment for Private Setting
Baselines: (1) DP-KNN-Shapley without subsampling. Recall from Section 3.1, the original KNN-Shapley has a large global sensitivity. Nevertheless, we can still use Gaussian mechanism to privatize it based on its global sensitivity bound. We call this approach as DP-KNN-Shapley. (2) DP-KNN-Shapley with subsampling. Recall from Section 3.1, it is computationally expensive to incorporate subsampling techniques for DP-KNN-Shapley (detailed in Appendix B.3.2). For instance, subsampled DP-KNN-Shapley with subsampling rate  $q = 0.01$  generally takes  $30 \times$  longer time compared with non-subsampled counterpart. Nevertheless, we still compare with subsampled DP-KNN-Shapley for completeness. These two baselines are detailed in Appendix B.4. We also note that an unpublished manuscript [67] proposed a DP version of Data Shapley. However, their DP guarantee requires a hard-to-verify assumption of uniform stability (see Appendix A). Hence, we do not compare with [67].
Results: We evaluate the privacy-utility tradeoff of DP-TKNN-Shapley. Specifically, for a fixed  $\delta$ , we examine the AUROC of mislabeled/noisy data detection tasks at different values of  $\varepsilon$ , where  $\varepsilon$  is adjusted by changing the magnitude of Gaussian noise. In the experiment, we set the subsampling rate  $q = 0.01$  for TKNN-Shapley and subsampled KNN-Shapley. Table 2 shows the results on 3 datasets, and we defer the results for the rest of 10 datasets to Appendix E.3.2. As we can see, DP-TKNN-Shapley shows a substantially better privacy-utility tradeoff compared to both DP-KNN-Shapley with/without subsampling. In particular, DP-TKNN-Shapley maintains a high
Mist labeled Data Detection	Noisy Data Detection
Dataset	ε	DP-TKNN-Shapley (ours)	DP-KNN-Shapley (no subsampling)	DP-KNN-Shapley (with subsampling)	DP-TKNN-Shapley (ours)	DP-KNN-Shapley (no subsampling)	DP-KNN-Shapley (with subsampling)
2dPlanes	0.1	0.883 (0.017)	0.49 (0.024)	0.733 (0.011)	0.692 (0.014)	0.494 (0.023)	0.615 (0.01)
0.5	0.912 (0.009)	0.488 (0.022)	0.815 (0.006)	0.706 (0.004)	0.494 (0.012)	0.66 (0.004)
1	0.913 (0.009)	0.504 (0.019)	0.821 (0.005)	0.705 (0.007)	0.495 (0.011)	0.665 (0.004)
Phoneme	0.1	0.816 (0.011)	0.5 (0.014)	0.692 (0.011)	0.648 (0.028)	0.475 (0.042)	0.566 (0.01)
0.5	0.826 (0.007)	0.497 (0.011)	0.738 (0.003)	0.683 (0.014)	0.536 (0.033)	0.588 (0.004)
1	0.826 (0.005)	0.486 (0.01)	0.741 (0.002)	0.685 (0.016)	0.494 (0.071)	0.59 (0.005)
CPU	0.1	0.932 (0.007)	0.49 (0.028)	0.881 (0.005)	0.805 (0.037)	0.42 (0.074)	0.709 (0.011)
0.5	0.946 (0.004)	0.507 (0.029)	0.928 (0.002)	0.838 (0.007)	0.472 (0.092)	0.746 (0.003)
1	0.948 (0.002)	0.512 (0.008)	0.931 (0.002)	0.839 (0.003)	0.455 (0.079)	0.748 (0.002)
Table 2: Privacy-utility tradeoff of DP-TKNN-Shapley and DP-KNN-Shapley for mislabeled/noisy data detection task on 3 datasets we use (see Appendix E.3.2 for the results on the rest of 10 datasets). We set  $\delta = 10^{-4}$  and show the AUROC at different privacy budgets  $\varepsilon$ ; the higher the AUROC is, the better the method is. We show the standard deviation of AUROC across 5 independent runs in().
AUROC even when  $\varepsilon \approx 0.1$ . The poor performance of DP-KNN-Shapley is due to its relatively high global sensitivity.
6.2.2 Experiment for Non-private Setting
Baselines: Our main baseline for comparison is KNN-Shapley. For completeness, we also compare with other classic, yet much less efficient data valuation techniques, such as Data Shapley [23], Data Banzhaf [60], and leave-one-out error (LOO) [35]. Due to space constraints, we only show the results for the famous Data Shapley here and defer other methods' results to Appendix E.3.3.
Results: We use AUROC as the performance metric on mislabeled/noisy data detection tasks. Due to space constraints, we defer the results for the task of noisy data detection to Appendix E. Table 3 shows the results for the task of mislabeled data detection across all 13 datasets we use. As we can see, TKNN-Shapley shows a comparable performance as KNN-Shapley across almost all datasets, demonstrating that TKNN-Shapley matches the effectiveness of KNN-Shapley in discerning data quality. Moreover, KNN-Shapley (and TKNN-Shapley) has a significantly better performance compared to Data Shapley, which is consistent with the observations in many existing studies [31, 52]. The poor performance of Data Shapley is attributable to the sample inefficiency and stochasticity during retraining [60].
Dataset	TKNN-Shapley	KNN-Shapley	Data Shapley
2dPlanes	0.919 (+0.006)	0.913	0.552
Phoneme	0.826 (-0.047)	0.873	0.525
CPU	0.946 (+0.014)	0.932	0.489
Fraud	0.96 (-0.007)	0.967	0.488
Creditcard	0.662 (+0.016)	0.646	0.517
Apsfail	0.958 (+0.01)	0.948	0.496
Click	0.572 (+0.004)	0.568	0.474
Wind	0.889 (-0.007)	0.896	0.469
Pol	0.871 (-0.057)	0.928	0.512
MNIST	0.962 (-0.012)	0.974	-
CIFAR10	0.957 (-0.034)	0.991	-
AG News	0.956 (-0.015)	0.971	-
DBPedia	0.981 (-0.01)	0.991	-
Table 3: AUROC scores of TKNN/KNN-Shapley and Data Shapley for mislabeled data detection tasks on various datasets. The higher the AUROC score is, the better the method is. In the column for TKNN-Shapley, we highlight its performance difference with KNN-Shapley in(). The results for Data Shapley on the last 4 datasets are omitted due to computational constraints.
In this work, we uncover the inherent privacy risks associated with data value scores and introduce TKNN-Shapley, a privacy-friendly alternative to the widely-used KNN-Shapley. We demonstrate that TKNN-Shapley outperforms KNN-Shapley in terms of computational efficiency, and is as good as discerning data quality. Moreover, the privatized version of TKNN-Shapley significantly surpasses the naively privatized KNN-Shapley on tasks such as unlabeled data detection.