SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiachen T. Wang*
tianhaowang@princeton.edu
Yuqing Zhu
UC Santa Barbara
yuqingzhu@ucsb.edu
Yu-Xiang Wang
yuxiangw@cs.ucsb.edu
Ruoxi Jia*
ruoxijia@vt.edu
Prateek Mittal
pmittal@princeton.edu
Data valuation aims to quantify the usefulness of individual data sources in training machine learning (ML) models, and is a critical aspect of data-centric ML research. However, data valuation faces significant yet frequently overlooked privacy challenges despite its importance. This paper studies these challenges with a focus on KNN-Shapley, one of the most practical data valuation methods nowadays. We first emphasize the inherent privacy risks of KNN-Shapley, and demonstrate the significant technical difficulties in adapting KNN-Shapley to accommodate differential privacy (DP). To overcome these challenges, we introduce TKNN-Shapley, a refined variant of KNN-Shapley that is privacy-friendly, allowing for straightforward modifications to incorporate DP guarantee  $(DP$ -TKNN-Shapley). We show that DP-TKNN-Shapley has several advantages and offers a superior privacy-utility tradeoff compared to naively privatized KNN-Shapley in discerning data quality. Moreover, even non-private TKNN-Shapley achieves comparable performance as KNN-Shapley. Overall, our findings suggest that TKNN-Shapley is a promising alternative to KNN-Shapley, particularly for real-world applications involving sensitive data.
Data valuation is an emerging research area that seeks to measure the contribution of individual data sources to the training of machine learning (ML) models. Data valuation is essential in data marketplaces for ensuring equitable compensation for data owners, and in explainable ML for identifying influential training data. The importance of data valuation is underscored by the DASHBOARD Act introduced in the U.S. Senate in 2019 [66], which mandates companies to provide users with an assessment of their data's economic value. Moreover, OpenAI's Future Plan explicitly highlights "the fair distribution of AI-generated benefits" as a key question moving forward [51].
Data Shapley. Inspired by cooperative game theory, [23, 30] initiated the study of using the Shapley value as a principled approach for data valuation. The Shapley value [56] is a famous solution concept from game theory for fairly assigning the total revenue to each player. In a typical scenario of data valuation, training data points are collected from different sources, and the data owner of each training data point can be regarded as "player" in a cooperative game. "Data Shapley" refers to the method of using the Shapley value as the contribution measure for each data owner. Since its introduction, many
* Correspondence to Jiachen T. Wang and Ruoxi Jia.
different variants of Data Shapley have been proposed [29, 22, 63, 6, 39, 44, 69, 33, 60], reflecting its effectiveness in quantifying data point contributions to ML model training.
KNN-Shapley. While being a principled approach for data valuation, the exact calculation of the Shapley value is computationally prohibitive [30]. Various approximation algorithms for Data Shapley have been proposed [30, 28, 50, 64, 7, 48, 44, 62], but these approaches still require substantial computational resources due to model retraining. Fortunately, a breakthrough by [29] showed that computing the exact Data Shapley for K-Nearest Neighbors (KNN), one of the oldest yet still popular ML algorithms, is surprisingly easy and efficient. KNN-Shapley quantifies data value based on KNN's Data Shapley score; it can be applied to large, high-dimensional CV/NLP datasets by calculating the value scores on the last-layer neural network embeddings. Owing to its superior efficiency and effectiveness in discerning data quality, KNN-Shapley is recognized as one of the most practical data valuation techniques nowadays [52]. It has been applied to various ML domains including active learning [24], continual learning [57], NLP [43, 42], and semi-supervised learning [9].
Motivation: privacy risks in data valuation. In this work, we study a critical, yet often overlooked concern in the deployment of data valuation: privacy leakage associated with data value scores released to data holders. The value of a single data point is always relative to other data points in the training set. This, however, can potentially reveal sensitive information about the rest of data holders in the dataset. This problem becomes even more complex when considering a strong threat model where multiple data holders collude, sharing their received data values to determine the membership of a particular individual. As data valuation techniques such as KNN-Shapley become increasingly popular and relevant in various applications, understanding and addressing the privacy challenges of data valuation methods is of utmost importance. In this work, we study this critical issue through the lens of differential privacy (DP) [15], a de-facto standard for privacy-preserving applications.
Our technical contributions are listed as follows.
Privacy Risks & Challenges of Privatization for KNN-Shapley (Section 3). We demonstrate that data value scores (specifically KNN-Shapley) indeed serve as a new channel for private information leakage, potentially exposing sensitive information about individuals in the dataset. In particular, we explicitly design a privacy attack (in Appendix B.2.2) where an adversary could infer the presence/absence of certain data points based on the variations in the KNN-Shapley scores, analogous to the classic membership inference attack on ML model [58]. Additionally, we highlight the technical challenges in incorporating the current KNN-Shapley technique with differential privacy, such as its large global sensitivity. All these results emphasize the need for a new, privacy-friendly approach to data valuation.
TKNN-Shapley: an efficient, privacy-friendly data valuation technique (Section 4.2). To address the privacy concerns, we derive a novel variant of KNN-Shapley. This new method considers the Data Shapley of an alternative form of KNN classifier called Threshold-KNN (TKNN) [5], which takes into account all neighbors within a pre-specified threshold of a test example, rather than the exact  $K$  nearest neighbors. We derive the closed-form formula of Data Shapley for TKNN (i.e., TKNN-Shapley). We show that it can be computed exactly with exact linear-time computational efficiency over the original KNN-Shapley. DP-TKNN-Shapley (Section 5). Importantly, we recognize that TKNN-Shapley only depends on three simple counting queries. Hence, TKNN-Shapley can be conveniently transformed into a differentially private version by DP's post-processing property. Moreover, we prove that such a DP variant satisfies several favorable properties, including (1) the high computational efficiency, (2) the capability to withstand collusion among data holders without compromising the privacy guarantee, and (3) the ease of integrating subsampling for privacy amplification.
Numerical experiments (Section 6). We evaluate the performance of TKNN-Shapley across 11 commonly used benchmark datasets and 2 NLP datasets. Key observations include: (1) TKNN-Shapley surpasses KNN-Shapley in terms of computational efficiency; (2) DP-TKNN-Shapley significantly outperforms the naively privatized KNN-Shapley in terms of privacy-utility tradeoff in discerning data quality; (3) even non-private TKNN-Shapley achieves comparable performance as KNN-Shapley.
Overall, our work suggests that TKNN-Shapley, being a privacy-friendly, yet more efficient and effective alternative to the original KNN-Shapley, signifies a milestone toward practical data valuation.
2 Background of Data Valuation
In this section, we formalize the data valuation problem for ML, and review the method of Data Shapley and KNN-Shapley.
Setup & Goal. Consider a dataset  $D := \{z_i\}_{i=1}^N$  consisting of  $N$  data points where each data point  $z_i := (x_i, y_i)$  is collected from a data owner  $i$ . The objective of data valuation is to attribute a score to each training data point  $z_i$ , reflecting its importance or quality in ML model training. Formally, we aim to determine a score vector  $(\phi_{z_i})_{i=1}^N$ , wherein  $\phi_{z_i}$  represents the value of data point  $z_i$ . For any reasonable data valuation method, the value of a data point is always relative to other data points in the dataset. For instance, if a data point has many duplicates in the dataset, its value will likely be lower. Hence,  $\phi_{z_i}$  is a function of the leave-one-out dataset  $D_{-z_i} := D \setminus \{z_i\}$ . We write  $\phi_{z_i}(D_{-z_i})$  when we want to stress the dependency of a data value score with the rest of the data points.
Utility Function. Most of the existing data valuation techniques are centered on the concept of utility function, which maps an input dataset to a score indicating the usefulness of the training set. A common choice for utility function is the validation accuracy of a model trained on the input training set. Formally, for a training set  $S$ , a utility function  $v(S) \coloneqq \mathrm{acc}(\mathcal{A}(S))$ , where  $\mathcal{A}$  is a learning algorithm that takes a dataset  $S$  as input and returns a model;  $\mathrm{acc}(\cdot)$  is a metric function that evaluates the performance of a given model, e.g., the classification accuracy on a hold-out validation set.
2.1 The Shapley Value
The Shapley value (SV) [56] is a classic concept from game theory to attribute the total gains generated by the coalition of all players. At a high level, it appraises each point based on the (weighted) average utility change caused by adding the point into different subsets of the training set. Formally, given a utility function  $v(\cdot)$  and a training set  $D$ , the Shapley value of a data point  $z \in D$  is defined as
$$
\phi_ {z} (D _ {- z}; v) := \frac {1}{N} \sum_ {k = 1} ^ {N} \binom {N - 1} {k - 1} ^ {- 1} \sum_ {S \subseteq D _ {- z}, | S | = k - 1} [ v (S \cup \{z \}) - v (S) ] \tag {1}
$$
For notation simplicity, when the context is clear, we omit the utility function and/or leave-one-out dataset, and write  $\phi_z(D_{-z})$ ,  $\phi_z(v)$  or  $\phi_z$  depending on the specific dependency we want to stress.
The popularity of the Shapley value is attributable to the fact that it is the unique data value notion satisfying four axioms: Dummy player, Symmetry, Linearity, and Efficiency. We refer the readers to [23], [30] and the references therein for a detailed discussion about the interpretation and necessity of the four axioms in the ML context. Here, we introduce the linearity axiom which will be used later.
Theorem 1 (Linearity of the Shapley value [56]). For any of two utility functions  $v_{1}, v_{2}$  and any  $\alpha_{1}, \alpha_{2} \in \mathbb{R}$ , we have  $\phi_z(\alpha_1v_1 + \alpha_2v_2) = \alpha_1\phi_z(v_1) + \alpha_2\phi_z(v_2)$ .
2.2 KNN-Shapley
Formula (1) suggests that the exact Shapley value can be computationally prohibitive in general, as it requires evaluating  $v(S)$  for all possible subsets  $S \subseteq D$ . Surprisingly, [29, 61] showed that for  $K$ -Nearest Neighbor (KNN), the computation of the exact Data Shapley score is highly efficient. Following its introduction, KNN-Shapley has rapidly gained attention and follow-up works across diverse areas of machine learning [24, 43, 42, 9]. In particular, it has been recognized by recent studies as "the most practical data valuation technique capable of handling large-scale data effectively" [52, 33].
Specifically, the performance of an unweighted KNN classifier is typically evaluated by its validation accuracy. For a given validation set  $D^{(\mathrm{val})} = \{z_i^{(\mathrm{val})}\}_{i=1}^{N_{\mathrm{val}}}$ , we can define KNN's utility function  $v_{D^{(\mathrm{val})}}^{\mathrm{KNN}}$  on a non-empty training set  $S$  as  $v_{D^{(\mathrm{val})}}^{\mathrm{KNN}}(S) := \sum_{z^{(\mathrm{val})} \in D^{(\mathrm{val})}} v_{z^{(\mathrm{val})}}^{\mathrm{KNN}}(S)$ , where
$$
v _ {z (\text {v a l})} ^ {\mathrm {K N N}} (S) := \frac {1}{\min  (K , | S |)} \sum_ {j = 1} ^ {\min  (K, | S |)} \mathbb {1} \left[ y _ {\pi^ {(S)} (j; x ^ {(\text {v a l})})} = y ^ {(\text {v a l})} \right] \tag {2}
$$
is the probability of a (soft-label) KNN classifier in predicting the correct label for a validation point  $z^{(\mathrm{val})} = (x^{(\mathrm{val})},y^{(\mathrm{val})})\in D^{(\mathrm{val})}$ . In (2),  $\pi^{(S)}(i;x^{\mathrm{(val)}})$  is the index of the  $i$ th closest data point in
Figure 1: The potential privacy risks in data valuation arise from the dependency of the data value score  $\phi_{z_i}$  on the rest of the dataset  $D_{-z_i}$ . Our goal is to privatize  $\phi_{z_i}(D_{-z_i})$  such that it provides strong differential privacy guarantee for the rest of the dataset  $D_{-z_i}$ .
$S$  to  $x^{\mathrm{(val)}}$ . When  $|S| = 0$ ,  $v_{z^{(\mathrm{val})}}^{\mathrm{KNN}}(S)$  is set to the accuracy by random guessing. The distance is measured through suitable metrics such as  $\ell_2$  distance. Here is the main result in [29, 61]:
Theorem 2 (KNN-Shapley [30, 61] (simplified version)). Consider the utility function in (2). Given a validation data point  $z^{\mathrm{(val)}} = (x^{\mathrm{(val)}}, y^{\mathrm{(val)}})$  and a distance metric  $d(\cdot, \cdot)$ , if we sort the training set  $D = \{z_i = (x_i, y_i)\}_{i=1}^N$  according to  $d(x_i, x^{\mathrm{(val)}})$  in ascending order, then the Shapley value of each data point  $\phi_{z_i}^{KNN}$  corresponding to utility function  $v_{z^{\mathrm{(val)}}}^{KNN}$  can be computed recursively as follows:
$$
\phi_ {z _ {N}} ^ {K N N} := f _ {N} (D), \text {a n d} \phi_ {z _ {i}} ^ {K N N} := \phi_ {z _ {i + 1}} ^ {K N N} + f _ {i} (D) \text {f o r} i = 1, \dots , N - 1
$$
where the exact form of functions  $f_{i}(D)$  can be found in Appendix B.1.
Runtime: the computation of all Shapley values  $(\phi_{z_1}^{KNN},\ldots ,\phi_{z_N}^{KNN})$  can be achieved in  $O(N\log N)$  runtime in total (dominated by the sorting data points in  $D$ ).
The Shapley value corresponding to full validation set: for a validation set  $D^{(\mathrm{val})}$ , recall that  $v_{D^{(\mathrm{val})}}^{KNN}(S) \coloneqq \sum_{z^{(\mathrm{val})} \in D^{(\mathrm{val})}} v_{z^{(\mathrm{val})}}^{KNN}(S)$ . One can compute the Shapley value corresponding to  $v_{D^{(\mathrm{val})}}^{KNN}$  by summing each  $\phi_{z_i}^{KNN}\left(v_{z^{(\mathrm{val})}}^{KNN}\right)$ , i.e.,  $\phi_{z_i}^{KNN}\left(v_{D^{(\mathrm{val})}}^{KNN}\right) = \sum_{z^{(\mathrm{val})} \in D^{(\mathrm{val})}} \phi_{z_i}^{KNN}\left(v_{z^{(\mathrm{val})}}^{KNN}\right)$ .
Theorem 2 tells that for any validation data point  $z^{(\mathrm{val})}$ , we can compute the exact Shapley value  $\phi_{z_i}^{\mathrm{KNN}}\left(D_{-z_i};v_{z(\mathrm{val})}^{\mathrm{KNN}}\right)$  for all  $z_i \in D$  by using a recursive formula within a total runtime of  $O(N\log N)$ . After computing the Shapley value  $\phi_{z_i}^{\mathrm{KNN}}\left(v_{z(\mathrm{val})}^{\mathrm{KNN}}\right)$  for each  $z^{(\mathrm{val})} \in D^{(\mathrm{val})}$ , one can compute the Shapley value corresponding to the full validation set by simply taking the sum of each  $\phi_{z_i}^{\mathrm{KNN}}\left(v_{z(\mathrm{val})}^{\mathrm{KNN}}\right)$  due to the linearity property (Theorem 1) of the Shapley value.
Remark 1 (Criteria of Computational Efficiency). As prior data valuation literature [23], we focus on the total runtime required to release all data value scores  $(\phi_{z_1}^{KNN},\dots,\phi_{z_N}^{KNN})$ . This is because, in a practical data valuation scenario (e.g., profit allocation), we rarely want to compute the data value score for only a single data point; instead, a typical objective is to compute the data value scores for all data points within the training set.
3 Privacy Risks & Challenges of Privatization for KNN-Shapley
Scenario. Figure 1 illustrates the data valuation scenario and potential privacy leakages considered in our paper. Specifically, a centralized, trusted server collects data point  $z_{i}$  from data owner  $i$  for each  $i \in [N]$ . The central server's role is to provide each data owner  $i$  with an assessment of the value of their data  $z_{i}$ , e.g., the KNN-Shapley value  $\phi_{z_i}^{\mathrm{KNN}}$ . A real life example: Mayo Clinic has created a massive digital health patient data marketplace platform [68], where the patients submit part of their medical records onto the platform, and life science companies/labs pay a certain amount of money to purchase patients' data. The platform's responsibility is to gauge the worth of the data of each patient (i.e., the data owner) to facilitate fair compensation.
The privacy risks associated with KNN-Shapley (as well as other data valuation techniques) arise from the fact that  $\phi_{z_i}^{\mathrm{KNN}}(D_{-z_i})$  depends on other data owners' data  $D_{-z_i}$ . Consequently, the data value score  $\phi_{z_i}^{\mathrm{KNN}}$  may inadvertently reveal private information (e.g., membership) about the rest of the dataset. The dependency of a data value score on the rest of the dataset is an unavoidable aspect of data valuation, as the value of a data point is inherently a relative quantity determined by its role within the complete dataset.
Remark 2 (Other privacy risks in data valuation). It is important to note that in this work, we do not consider the privacy risks of revealing individuals' data to the central server. This is a different type of privacy risk that needs to be addressed using secure multi-party computation (MPC) technique [70], and it should be used together with differential privacy in practice. In addition, to use KNN-Shapley or many other data valuation techniques, the central server needs to maintain a clean, representative validation set, the privacy of which is not considered by this paper.
A Simple Membership Inference (MI) Attack on KNN-Shapley (detailed in Appendix B.2.2).
We further illustrate the privacy risks of revealing data value scores with a concrete example. Analogous to the classic membership inference attack on ML model [58], in Appendix B.2.2 we show an example of privacy attack where an adversary could infer the presence/absence of certain data points in the dataset based on the variations in the KNN-Shapley scores. The design is analogous to the membership inference attack against ML models via the likelihood ratio test [8]. The AUROC score of the attack results is shown in Table 1. As we can see, our
MIA attack can achieve a detection performance that is better than the random guess (0.5) for most of the settings. On some datasets, the attack performance can achieve  $>0.7$  AUROC. This demonstrates that privacy leakage in data value scores can indeed lead to non-trivial privacy attacks, and underscores the need for privacy safeguards in data valuation. $^{1}$
K=1	K=3	K=5	K=7	K=9	K=11	K=13	K=15
2DPlanes	0.56	0.595	0.518	0.52	0.57	0.55	0.515	0.6
Phoneme	0.692	0.54	0.513	0.505	0.505	0.588	0.512	0.502
CPU	0.765	0.548	0.52	0.572	0.588	0.512	0.612	0.615
Fraud	0.625	0.645	0.6	0.592	0.622	0.532	0.538	0.558
Creditcard	0.542	0.643	0.628	0.665	0.503	0.67	0.602	0.66
Apsfail	0.532	0.595	0.625	0.6	0.53	0.645	0.532	0.52
Click	0.61	0.525	0.588	0.538	0.588	0.582	0.622	0.618
Wind	0.595	0.51	0.518	0.528	0.558	0.562	0.505	0.577
Pol	0.725	0.695	0.7	0.62	0.57	0.522	0.535	0.532
Table 1: Results of the AUROC of the MI attack proposed in Appendix B.2.2 on KNN-Shapley. The higher the AUROC score is, the larger the privacy leakage is. The detailed algorithm description and experiment settings can be found in Appendix B.2.2.
3.1 Privatizing KNN-Shapley is Difficult
The growing popularity of data valuation techniques, particularly KNN-Shapley, underscores the critical need to mitigate the inherent privacy risks. In the quest for privacy protection, differential privacy (DP) [15] has emerged as the leading framework. DP has gained considerable recognition for providing robust, quantifiable privacy guarantees, thereby becoming the de-facto choice in privacy protection. In this section, we introduce the background of DP and highlight the technical difficulties in constructing a differentially private variant for the current version of KNN-Shapley.
Background of Differential Privacy. We use  $D$ ,  $D' \in \mathbb{N}^{\mathcal{X}}$  to denote two datasets with an unspecified size over space  $\mathcal{X}$ . We call two datasets  $D$  and  $D'$  adjacent (denoted as  $D \sim D'$ ) if we can construct one by adding/removing one data point from the other, e.g.,  $D = D' \cup \{z\}$  for some  $z \in \mathcal{X}$ .
Definition 3 (Differential Privacy [15]). For  $\varepsilon, \delta \geq 0$ , a randomized algorithm  $\mathcal{M}:\mathbb{N}^{\mathcal{X}}\to \mathcal{Y}$  is  $(\varepsilon, \delta)$ -differentially private if for every pair of adjacent datasets  $D\sim D^{\prime}$  and for every subset of possible outputs  $E\subseteq \mathcal{V}$ , we have  $\operatorname*{Pr}_{\mathcal{M}}[\mathcal{M}(D)\in E]\leq e^{\varepsilon}\operatorname*{Pr}_{\mathcal{M}}[\mathcal{M}(D^{\prime})\in E] + \delta$
That is,  $(\varepsilon, \delta)$ -DP requires that for all adjacent datasets  $D, D'$ , the output distribution  $\mathcal{M}(D)$  and  $\mathcal{M}(D')$  are close, where the closeness is measured by the parameters  $\varepsilon$  and  $\delta$ . In our scenario, we would like to modify data valuation function  $\phi_{z_i}(D_{-z_i})$  to satisfy  $(\varepsilon, \delta)$ -DP in order to protect the privacy of the rest of the dataset  $D_{-z_i}$ . Gaussian mechanism [14] is a common way for privatizing a function; it introduces Gaussian noise aligned with the function's global sensitivity, which is the maximum output change when a single data point is added/removed from any given dataset.
Definition 4 (Gaussian Mechanism [14]). Define the global sensitivity of a function  $f: \mathbb{N}^{\mathcal{X}} \to \mathbb{R}^{d}$  as  $\Delta_2(f) \coloneqq \sup_{D \sim D'} \|f(D) - f(D')\|_2$ . The Gaussian mechanism  $\mathcal{M}$  with noise level  $\sigma$  is then given by  $\mathcal{M}(D) \coloneqq f(D) + \mathcal{N}\left(0, \sigma^2 \mathbb{1}_d\right)$  where  $\mathcal{M}$  is  $(\varepsilon, \delta)$ -DP with  $\sigma = \Delta_2(f) \sqrt{\log(1.25 / \delta)} / \varepsilon$ .
<sup>1</sup>We stress that the goal here is to demonstrate that the data value scores can indeed serve as another channel of privacy leakage. We do not claim any optimality of the attack we construct here. Improving MI attacks for data valuation is an interesting future work.
Challenges in making KNN-Shapley being differentially private (overview). Here, we give an overview of the inherent difficulties in making the KNN-Shapley  $\phi_{z_i}^{\mathrm{KNN}}(D_{-z_i})$  (Theorem 2) to be differentially private, and we provide a more detailed discussion in Appendix B.3. (1) Large global sensitivity: In Appendix B.3, we show that the global sensitivity of  $\phi_{z_i}^{\mathrm{KNN}}(D_{-z_i})$  can significantly exceed the magnitude of  $\phi_{z_i}^{\mathrm{KNN}}$ . Moreover, we prove that the global sensitivity bound cannot be further improved by constructing a specific pair of neighboring datasets that matches the bound. Hence, if we introduce random noise proportional to the global sensitivity bound, the resulting privatized data value score could substantially deviate from its non-private counterpart, thereby compromising the utility of the privatized data value scores. (2) Computational challenges in incorporating privacy amplification by subsampling: "Privacy amplification by subsampling" [4] is a technique where the subsampling of a dataset amplifies the privacy guarantees due to the reduced probability of an individual's data being included. Being able to incorporate such a technique is often important for achieving a decent privacy-utility tradeoff. However, in Appendix B.3 we show that the recursive nature of KNN-Shapley computation causes a significant increase in computational demand compared to non-private KNN-Shapley.
These challenges underscore the pressing need for new data valuation techniques that retain the efficacy and computational efficiency of KNN-Shapley, while being amenable to privatization.
4 The Shapley Value for Threshold-based Nearest Neighbor