SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Future Work. (1) Privacy risks of data revelation to central server: in this work, we assume the existence of a trusted central server, and we do not consider the privacy risks associated with revealing individuals' data to the central server. Future work should consider integrating secure multi-party computation (MPC) techniques to mitigate this risk [59]. MPC can allow the computation of KNN-Shapley without revealing individual data to the central server, thereby preserving privacy. We envision an end-to-end privacy-preserving data valuation framework that combines both DP and MPC. (2) Impact of Randomization on Payment Fairness: the incorporation of differential privacy necessarily adds a degree of randomness to the data value scores. This randomization could potentially impact the fairness of payments to data providers [3]. The influence of this randomness, and its potential implications for payment fairness, remain areas for further investigation.
This work was supported in part by the National Science Foundation under grants CNS-2131938, CNS-1553437, CNS-1704105, CNS-2048091, IIS-2312794, IIS-2313130, OAC-2239622, the ARL's Army Artificial Intelligence Innovation Institute (A2I2), the Office of Naval Research Young Investigator Award, the Army Research Office Young Investigator Prize, Schmidt DataX award, Princeton Effliates Award, Amazon-Virginia Tech Initiative in Efficient and Robust Machine Learning, the Commonwealth Cyber Initiative, a Google PhD Fellowship, and a Princeton's Gordon Y. S. Wu Fellowship. We are grateful to anonymous reviewers at NeurIPS for their valuable feedback.
[1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-318, 2016.
[2] Soren Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. Dbpedia: A nucleus for a web of open data. In *The Semantic Web: 6th International Semantic Web Conference*, 2nd Asian Semantic Web Conference, ISWC 2007+ ASWC 2007, Busan, Korea, November 11-15, 2007. Proceedings, pages 722-735. Springer, 2007.
[3] Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. Differential privacy has disparate impact on model accuracy. Advances in neural information processing systems, 32, 2019.
[4] Borja Balle, Gilles Barthe, and Marco Gaboardi. Privacy amplification by subsampling: Tight analyses via couplings and divergences. Advances in Neural Information Processing Systems, 31, 2018.
[5] Jon L Bentley. Survey of techniques for fixed radius near neighbor searching. Technical report, Stanford Linear Accelerator Center, Calif.(USA), 1975.
[6] Yatao Bian, Yu Rong, Tingyang Xu, Jiaxiang Wu, Andreas Krause, and Junzhou Huang. Energy-based learning for cooperative games, with applications to valuation problems in machine learning. arXiv preprint arXiv:2106.02938, 2021.
[7] Mark Alexander Burgess and Archie C Chapman. Approximating the shapley value using stratified empirical bernstein sampling. In *IJCAI*, pages 73–81, 2021.
[8] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897-1914. IEEE, 2022.
[9] Christie Courtnage and Evgueni Smirnov. Shapley-value data valuation for semi-supervised learning. In Discovery Science: 24th International Conference, DS 2021, Halifax, NS, Canada, October 11–13, 2021, Proceedings 24, pages 94–108. Springer, 2021.
[10] Rachel Cummings, Hadi Elzayn, Emmanouil Pountourakis, Vasilis Gkatzelis, and Juba Ziani. Optimal data acquisition with privacy-aware agents. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 210-224. IEEE, 2023.
[11] Andrea Dal Pozzolo, Olivier Caelen, Reid A Johnson, and Gianluca Bontempi. Calibrating probability with undersampling for unbalanced classification. In 2015 IEEE Symposium Series on Computational Intelligence, pages 159-166. IEEE, 2015.
[12] Irit Dinur and Kobbi Nissim. Revealing information while preserving privacy. In Proceedings of the twenty-second ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, pages 202-210, 2003.
[13] Pradeep Dubey, Abraham Neyman, and Robert James Weber. Value theory without efficiency. Mathematics of Operations Research, 6(1):122-128, 1981.
[70] Andrew C Yao. Protocols for secure computations. In 23rd annual symposium on foundations of computer science (sfcs 1982), pages 160-164. IEEE, 1982.
[71] I-Cheng Yeh and Che-hui Lien. The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert systems with applications, 36(2):2473-2480, 2009.
[72] Yuqing Zhu, Xuandong Zhao, Chuan Guo, and Yu-Xiang Wang. Private prediction strikes back! Private kernelized nearest neighbors with individual Rényi filter. In Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, pages 2586-2596, 2023.
KNN-Shapley & Its Applications. The Shapley value is known for being computationally expensive. Fortunately, [29] found that computing the Data Shapley for K-Nearest Neighbors (KNN), one of the most classic yet still popular ML algorithms, is surprisingly easy and efficient. To the best of our knowledge, unweighted KNN is the only commonly-used ML model for which the exact Data Shapley can be efficiently computed, referred to as 'KNN-Shapley'. Owing to its superior computational efficiency and effectiveness in distinguishing data quality, KNN-Shapley has become one of the most popular and practical data valuation techniques. In the realm of ML research, for instance, [24] extends KNN-Shapley to active learning, [57] employs it in a continual learning setting, [43, 42] utilize KNN-Shapley for removing confusing samples in NLP applications, and [9] adopts KNN-Shapley for data valuation in semi-supervised learning. Furthermore, KNN-Shapley has also proven to be highly practical in real-life applications: [52] demonstrates that KNN-Shapley is the only practical data valuation technique for valuing large amounts of healthcare data, and [33] builds the first data debugging system for end-to-end ML pipelines based on KNN-Shapley.
Remark 5 (KNN-Shapley vs General Data Shapley). In comparison to the work of general Data Shapley [23] (including Beta Shapley [39] as well as Data Banzhaf [60]), KNN-Shapley may have the following differences: (1) KNN-Shapley focuses on KNN classifiers. As a result, the applicability of KNN-Shapley scores as a proxy of data points' value with respect to other ML models may not be straightforward. However, it is noteworthy that KNN is asymptotically Bayes optimal, implying that KNN-Shapley scores can be justified as a proxy for the data's value relative to the best possible model, i.e., the Bayes classifier, under certain asymptotic conditions. (2) For high-dimensional data, such as images, KNN-Shapley requires a public model to first map the original data into data embeddings, and evaluates the value of these embeddings rather than the original data. While this is indeed a constraint in certain scenario, it is important to recognize that utilizing a publicly available foundation model to convert original data into embeddings, followed by the fine-tuning of the model's last layer, has become a common practice. Therefore, in many situations, it might be more desirable to evaluate the value of data embeddings instead of the original data.
Privacy and Data Valuation. Few studies in the literature consider the privacy risks of data valuation. [59] explores a scenario where a trusted server does not exist, and different data holders collaboratively compute each other's Shapley values without actually examining the data holder's data points, utilizing Multi-party Computation (MPC) techniques. The privacy risks addressed in [59] are orthogonal to those in our paper, and we can combine both techniques for end-to-end privacy protection. Another orthogonal line of works [45, 20, 10, 32] studies the scenario of a central platform collecting private data from privacy-aware agents and offering a differentially private statistic computed from the submitted data as a service in return. Agents consider the privacy costs and benefits of obtaining the statistic when deciding whether to participate and reveal their data truthfully. [46] proposes a privacy attack on the Shapley value for feature attribution, while in this work we study the privacy risks when using the Shapley value for data valuation.
The unpublished manuscript by [67] is the work most closely related to ours. [67] explores how to make the Shapley values of a data point to be differentially private against the rest of the dataset. However, instead of focusing on KNN-Shapley, they study the privatization of the less practical, retrain-based Data Shapley [23, 30]. Furthermore, their algorithm is highly restrictive in that the differential privacy guarantee relies on the "uniform stability" assumption, which is not verifiable for modern learning algorithms such as neural networks. Moreover, while [67] argues the "uniform stability" assumption holds for Logistic regression on bounded data domain, it is unclear whether the uniform stability assumption still holds when the Logistic regression is trained by SGD (which may involve training stochasticity and early stopping). In addition, [67] does not release the implementation. Hence, in our experiment, we do not compare with it.
Remark 6 (Brief background for the privacy risks in releasing aggregated statistics). Since 1998, researchers have observed that a lot of seemingly benevolent aggregate statistics of a dataset can be used to reveal sensitive information about individuals [55]. A classic example is Netflix Prize fiasco, where the researchers show that an anonymized dataset can leak many sensitive information about individuals [49]. Dinur and Nissim [12] proved that "revealing too many statistics too accurately leads to data privacy breach". A great amount of discussion and practical realization of these privacy attacks on aggregated statistics can be found in [19]. In 2020, the US Census Bureau used these privacy attacks to justify its use of differential privacy.
KNN-Shapley score for an individual is one kind of aggregated statistic that depends on the rest of the dataset. Hence, KNN-Shapley score intrinsically reveals private information about the rest of the dataset (where we use membership inference attack as a concrete example in our paper). In addition, when users collude, their KNN-shapley values can be combined to make joint inferences about the rest of the dataset.
B Details about KNN-Shapley, its Privacy Risks & Challenges of Privatization
B.1 Full version of KNN-Shapley
KNN-Shapley was originally proposed in [29] and was later refined in [61]. Specifically, [61] considers the KNN's utility function formula (2) we present in the main text. Here is the main result of [61]:
Theorem 7 (KNN-Shapley [61]). Consider the utility function in (2). Given a validation data point  $z^{\mathrm{(val)}} = (x^{\mathrm{(val)}},y^{\mathrm{(val)}})$  and a distance metric  $d(\cdot ,\cdot)$ , if we sort the training set  $D = \{z_{i} = (x_{i},y_{i})\}_{i = 1}^{N}$  according to  $d(x_{i},x^{\mathrm{(val)}})$  in ascending order, then the Shapley value of each data point  $\phi_{z_i}^{KNN}$  corresponding to utility function  $v_{z(\mathrm{val})}^{KNN}$  can be computed recursively as follows:
$$
\phi_ {z _ {N}} ^ {K N N} = \frac {\mathbb {1} [ N \geq 2 ]}{N} \left(\mathbb {1} \left[ y _ {N} = y ^ {(\text {v a l})} \right] - \frac {\sum_ {i = 1} ^ {N - 1} \mathbb {1} \left[ y _ {i} = y ^ {(\text {v a l})} \right]}{N - 1}\right) \left(\sum_ {j = 1} ^ {\min  (K, N) - 1} \frac {1}{j + 1}\right) + \frac {1}{N} \left(\mathbb {1} \left[ y _ {N} = y ^ {(\text {v a l})} \right] - \frac {1}{C}\right)
$$
$$
\phi_ {z _ {i}} ^ {K N N} = \phi_ {z _ {i + 1}} ^ {K N N} + \frac {\mathbb {1} [ y _ {i} = y ^ {(\mathrm {v a l})} ] - \mathbb {1} [ y _ {i + 1} = y ^ {(\mathrm {v a l})} ]}{N - 1} \left[ \sum_ {j = 1} ^ {\min (K, N)} \frac {1}{j} + \frac {\mathbb {1} [ N \geq K ]}{K} \left(\frac {\min (i , K) \cdot (N - 1)}{i} - K\right) \right]
$$
where  $C$  denotes the number of classes for the classification task.
B.1.1 Older version of KNN-Shapley from [29]
Prior to the current formulation of KNN-Shapley introduced by [61], [29] proposed an older version of the algorithm. There is only a small distinction between the updated version by [61] and the older version from [29]. Specifically, the difference between the versions proposed by [61] and [29] lies in the utility function considered when computing the Shapley value. [61] use the utility function (2) as presented in the main text, while the utility function used in [29] is slightly less interpretable:
$$
v _ {z ^ {(\mathrm {v a l})}} ^ {\mathrm {K N N - O L D}} (S) := \frac {1}{K} \sum_ {j = 1} ^ {\min  (K, | S |)} \mathbb {1} \left[ y _ {\pi^ {(S)} (j; x ^ {(\mathrm {v a l})})} = y ^ {(\mathrm {v a l})} \right] \tag {5}
$$
That is, (2) in the maintext divides the number of correct predictions by  $\min(K, |S|)$ , which can be interpreted as the likelihood of the soft-label KNN classifier predicting the correct label  $y^{(\mathrm{val})}$  for  $x^{(\mathrm{val})}$ . On the other hand, the function above (5) divides the number of correct predictions by  $K$ , which is less interpretable when  $|S| < K$ .
Nevertheless, the main result in [29] shows the following:
Theorem 8 (Older Version of KNN-Shapley from  $[29]^2$ ). Consider the utility function in (5). Given a validation data point  $z^{\mathrm{(val)}} = (x^{\mathrm{(val)}},y^{\mathrm{(val)}})$  and a distance metric  $d(\cdot ,\cdot)$ , if we sort the training set  $D = \{z_{i} = (x_{i},y_{i})\}_{i = 1}^{N}$  according to  $d(x_{i},x^{\mathrm{(val)}})$  in ascending order, then the Shapley value of each data point  $\phi_{z_i}^{KNN - OLD}$  corresponding to utility function  $v_{z^{\mathrm{(val)}}}^{KNN - OLD}$  can be computed recursively as follows:
$$
\phi_ {z _ {N}} ^ {K N N - O L D} = \frac {\mathbb {1} \left[ y _ {N} = y _ {\text {v a l}} \right]}{\max (K , N)}
$$
$$
\phi_ {z _ {i}} ^ {K N N - O L D} = \phi_ {z _ {i + 1}} ^ {K N N - O L D} + \frac {\mathbb {1} [ y _ {i} = y _ {\text {v a l}} ] - \mathbb {1} [ y _ {i + 1} = y _ {\text {v a l}} ]}{K} \frac {\min (K , i)}{i}
$$
As we can see, both versions of KNN-Shapley have recursive forms. Since the utility functions they consider are very similar to each other except for the normalization term for small subsets, the two versions of KNN-Shapley have very close value scores in practice and as shown in the experiments of [61], the two versions of KNN-Shapley perform very similarly.
Remark 7 (The use of older version of KNN-Shapley for DP-related experiments). In the experiments, we use the more advanced version of the KNN-Shapley from [61] except for the DP-related experiments. This is because the global sensitivity of KNN-Shapley is difficult to derive, and we can only derive the global sensitivity of the older version of KNN-Shapley from [29] (as we will
2We state a more generalized version which does not require  $N\geq K$
show in Appendix B.3). Hence, we can only use the older version of KNN-Shapley from [29] as the baseline in DP-related experiments. It is important to note that for non-DP experiments, the performance of KNN-Shapley and its older variant is very close to each other, and which version of the KNN-Shapley we use for non-DP experiments does not affect the final conclusion.
B.2 Settings & Additional Experiments for Privacy Risks for KNN-Shapley
B.2.1 Experiment for the changes of data value after eliminating nearby points
We first investigate the impact of removing a data point on the KNN-Shapley score of another data point that is close to the removed one. Specifically, we calculate the KNN-Shapley score for a chosen data point in the dataset, and then repeat the process after eliminating one of its nearby points from the dataset.
Settings. We use commonly used datasets in the past literature, and the details for data preprocessing can be found in Appendix E.1. We calculate the KNN-Shapley score for a randomly selected data point in the dataset, and then repeat the computation of the KNN-Shapley score after we eliminate its nearest neighbor from the dataset.
Results. The results for a variety of other datasets are shown in Figure 3. We can see a significant difference in the KNN-Shapley score of the investigated data point depending on whether the nearest data point has been removed. We remark that how the data value change (increases or decreases) when the nearest data point is excluded depends on the label of the nearest data point as well as the validation data being used.
Figure 3: Curves of KNN-Shapley of a fixed data point as a function of  $K$  (the hyperparameter for KNN), with and without a particular nearby data point in the training set.
For completeness, we also plot the same figures but for the older version of KNN-Shapley from [29], which shows similar results.
Figure 4: Curves of KNN-Shapley (old version from [29]) of a fixed data point as a function of  $K$  (the hyperparameter for KNN), with and without a particular nearby data point in the training set.
B.2.2 An Instantiation of Membership Inference Attack via Data Value Scores
The observation in Appendix B.2.1, demonstrates the potential for KNN-Shapley scores to leak membership information of other data points in the dataset. To further demonstrate such privacy risk, in this section we show an example of a privacy attack where an adversary could infer the presence/absence of certain data points in the dataset based on the variations in the KNN-Shapley scores, analogous to the classic membership inference attack on ML model [58]. The attack result further highlights the need for privacy protection when applying data valuation techniques.
Remark 8 (Motivation of Membership Inference Attack). Membership inference attack, i.e., confirming one's membership in a database, could pose significant privacy risks. For example, if a medical database is known to contain data from patients with specific conditions, this could disclose members' health status. We choose MI attack as our main example of the privacy risk as it has long been recognized as a fundamental privacy challenge across various domains, and such attacks have been extended well into the field of machine learning.
Remark 9. We stress that our goal here is the proof-of-concept, where we demonstrate that the data value scores can indeed serve as another channel of privacy leakage, and can indeed lead to the design of membership inference attacks. We do not claim any optimality of the attack we construct here. Designing the best membership inference attack on data value scores can be the topic of a single paper on its own, and is an interesting future work.
Membership Inference Attack via KNN-Shapley Scores (Algorithm 1). Our membership inference attack technique leverages KNN-Shapley scores to detect the presence or absence of specific data points in a dataset. The design is analogous to the membership inference attack against ML models via the likelihood ratio test [8]. Threat model: The threat model we consider here is that the attacker can compute the value of any data point among any datasets (analogue to the setting of MIA against ML models where the attacker can train models on any datasets he/she constructs). Moreover,
Algorithm 1 - Membership Inference Attack via KNN-Shapley Scores. We collect data value scores of a copy of the target example on datasets with and without the target example, estimate mean and variance of the loss distributions, and compute a likelihood ratio test.
Require: dataset  $D$  query example  $z\coloneqq (x,y)$  data distribution  $\mathbb{D}$    
1:KNNSVin  $\equiv$  {   
2:KNNSVout  $\equiv$  {   
3:for  $T$  times do   
4:  $D_{\mathrm{shadow}}\leftarrow \mathbb{S}^{\mathbb{D}} / /$  Sample a shadow dataset   
5: KNNSVin  $\leftarrow$  KNNSVin  $\cup \{\phi_z(D_{\mathrm{shadow}}\cup z)\} /$  collect IN data value   
6: KNNSVin  $\leftarrow$  KNNSVin  $\cup \{\phi_z(D_{\mathrm{shadow}}\}$  / collect OUT data value   
7: end for   
8:  $\mu_{\mathrm{in}}\gets$  mean(KNNSVin)   
9:  $\mu_{\mathrm{out}}\gets$  mean(KNNSVout)   
10:  $\sigma_{\mathrm{in}}^{2}\gets$  var(KNNSVin)   
11:  $\sigma_{\mathrm{out}}^{2}\gets$  var(KNNSVout)   
12:  $\phi_{\mathrm{obs}}\gets \phi_z(D)$  // query target data value   
13: Return  $\Lambda = \frac{p(\phi_{\mathrm{obs}}|\mathcal{N}(\mu_{\mathrm{in}},\sigma_{\mathrm{in}}^{2}))}{p(\phi_{\mathrm{obs}}|\mathcal{N}(\mu_{\mathrm{out}},\sigma_{\mathrm{out}}^{2}))}$
the attacker can craft the data point it owns, send it to the server and obtain the data value score of its own data point.
The attack goes as follows: firstly, we create a shadow dataset  $D_{\mathrm{shadow}}$  by randomly sampling from the data distribution  $\mathbb{D}$ .  $D_{\mathrm{shadow}}$  serves as the function of  $D_{-z}$  in the maintext. We repeat this sampling process  $T$  times, where  $T$  is a predefined number of iterations. For each iteration, we calculate the KNN-Shapley scores of a query example  $z = (x,y)$  both when it is included in the shadow dataset (IN) and when it is excluded (OUT). We thus collect sets of IN and OUT scores for the query example. Upon collecting all these scores, we calculate their respective mean and variance. Finally, we query the server about KNN-Shapley score of the query example  $z$  on the actual dataset  $D_{-z}$ , which we refer to as the target data value (i.e., the server takes a copy the target data point as the data it holds, and send to the central server). We perform a likelihood ratio test using the distributions of the IN and OUT scores. This test involves comparing the probability of the observed target data value given the Normal distribution of the IN scores with the probability of the same given the Normal distribution of the OUT scores.
Intuition of the attack. The intuition of the attack is as follows: if the target data point is in the dataset, then if the attacker makes a copy of the target data point and send it to the server, the server effectively has two exactly the same data point, which will result in a lower value of each data point. On the other hand, if the target data point is a non-member, then the data value queried by the attacker will be higher.
Experiment Settings For each dataset we experiment on, we select 200 data points (members) as the private dataset held by the central server. We pick another 200 data points which serve as non-members. Moreover, we leverage another 400 data points where we can subsample "shadow dataset" in Algorithm 1. We set the number of shadow datasets we sample as 32.
Results on KNN-Shapley and TKNN-Shapley. Table 1 in the main paper shows the AUROC score of the attack results on KNN-Shapley. For completeness, we additionally conducted the experiment of the proposed MIA against (non-private) TKNN-Shapley and the results are shown in Table 4. As we can see, our MIA attack can achieve a detection performance that is better than the random guess (0.5) for most of the settings. On some datasets, the attack performance can achieve  $>0.7$  AUROC. This demonstrates that privacy leakage in data value scores can indeed lead to non-trivial privacy attacks.
Results on DP-TKNN-Shapley. To directly demonstrate how DP-TKNN-Shapley can mitigate the privacy risk, we additionally conducted the experiment of evaluating the proposed MI attack on DP-TKNN-Shapley (see Table 5 and 6). Compared with the result on non-private TKNN-Shapley, we can see that the overall attack performance drops to around 0.5 (the performance of random guess).
The result shows that DP-TKNN-Shapley is indeed very effective against membership inference attacks.