Title: A Privacy-Friendly Approach to Data Valuation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Data Valuation

Differential Privacy
Abstract: Data valuation, a growing field that aims at quantifying the usefulness of individual data sources for training machine learning (ML) models, faces notable yet often overlooked privacy challenges. This paper studies these challenges with a focus on KNN-Shapley, one of the most practical data valuation methods nowadays. We first emphasize the inherent privacy risks of KNN-Shapley, and demonstrate the significant technical challenges in adapting KNN-Shapley to accommodate differential privacy (DP). To overcome these challenges, we introduce TKNN-Shapley, a refined variant of KNN-Shapley that is privacy-friendly, allowing for straightforward modifications to incorporate DP guarantee (DP-TKNN-Shapley). We show that DP-TKNN-Shapley has several advantages and offers a superior privacy-utility tradeoff compared to naively privatized KNN-Shapley. Moreover, even non-private TKNN-Shapley matches KNN-Shapley's performance in discerning data quality. Overall, our findings suggest that TKNN-Shapley is a promising alternative to KNN-Shapley, particularly for real-world applications involving sensitive data.

Threshold KNN-Shapley: A Linear-Time and Privacy-Friendly Approach to Data Valuation

Abstract

Data valuation aims to quantify the usefulness of individual data sources in training machine learning (ML) models, and is a critical aspect of data-centric ML research. However, data valuation faces significant yet frequently overlooked privacy challenges despite its importance. This paper studies these challenges with a focus on KNN-Shapley, one of the most practical data valuation methods nowadays. We first emphasize the inherent privacy risks of KNN-Shapley, and demonstrate the significant technical difficulties in adapting KNN-Shapley to accommodate differential privacy (DP). To overcome these challenges, we introduce TKNNShapley , a refined variant of KNN-Shapley that is privacy-friendly, allowing for straightforward modifications to incorporate DP guarantee ( DPTKNN-Shapley). We show that DP-TKNN-Shapley has several advantages and offers a superior privacy-utility tradeoff compared to naively privatized KNN-Shapley in discerning data quality. Moreover, even non-private TKNN-Shapley achieves comparable performance as KNN-Shapley. Overall, our findings suggest that TKNN-Shapley is a promising alternative to KNN-Shapley, particularly for real-world applications involving sensitive data.

1 Introduction

Data valuation is an emerging research area that seeks to measure the contribution of individual data sources to the training of machine learning (ML) models. Data valuation is essential in data marketplaces for ensuring equitable compensation for data owners, and in explainable ML for identifying influential training data. The importance of data valuation is underscored by the DASHBOARD Act introduced in the U.S. Senate in 2019 [66], which mandates companies to provide users with an assessment of their data's economic value. Moreover, OpenAI's Future Plan explicitly highlights 'the fair distribution of AI-generated benefits' as a key question moving forward [51].

Data Shapley. Inspired by cooperative game theory, [23, 30] initiated the study of using the Shapley value as a principled approach for data valuation. The Shapley value [56] is a famous solution concept from game theory for fairly assigning the total revenue to each player. In a typical scenario of data valuation, training data points are collected from different sources, and the data owner of each training data point can be regarded as 'player' in a cooperative game. 'Data Shapley' refers to the method of using the Shapley value as the contribution measure for each data owner. Since its introduction, many

Yuqing Zhu

different variants of Data Shapley have been proposed [29, 22, 63, 6, 39, 44, 69, 33, 60], reflecting its effectiveness in quantifying data point contributions to ML model training.

KNN-Shapley. While being a principled approach for data valuation, the exact calculation of the Shapley value is computationally prohibitive [30]. Various approximation algorithms for Data Shapley have been proposed [30, 28, 50, 64, 7, 48, 44, 62], but these approaches still require substantial computational resources due to model retraining. Fortunately, a breakthrough by [29] showed that computing the exact Data Shapley for K-Nearest Neighbors (KNN), one of the oldest yet still popular ML algorithms, is surprisingly easy and efficient. KNN-Shapley quantifies data value based on KNN's Data Shapley score; it can be applied to large, high-dimensional CV/NLP datasets by calculating the value scores on the last-layer neural network embeddings. Owing to its superior efficiency and effectiveness in discerning data quality, KNN-Shapley is recognized as one of the most practical data valuation techniques nowadays [52]. It has been applied to various ML domains including active learning [24], continual learning [57], NLP [43, 42], and semi-supervised learning [9].

Motivation: privacy risks in data valuation. In this work, we study a critical, yet often overlooked concern in the deployment of data valuation: privacy leakage associated with data value scores released to data holders. The value of a single data point is always relative to other data points in the training set. This, however, can potentially reveal sensitive information about the rest of data holders in the dataset. This problem becomes even more complex when considering a strong threat model where multiple data holders collude, sharing their received data values to determine the membership of a particular individual. As data valuation techniques such as KNN-Shapley become increasingly popular and relevant in various applications, understanding and addressing the privacy challenges of data valuation methods is of utmost importance. In this work, we study this critical issue through the lens of differential privacy (DP) [15], a de-facto standard for privacy-preserving applications.

Our technical contributions are listed as follows.

Privacy Risks & Challenges of Privatization for KNN-Shapley (Section 3). We demonstrate that data value scores (specifically KNN-Shapley) indeed serve as a new channel for private information leakage, potentially exposing sensitive information about individuals in the dataset. In particular, we explicitly design a privacy attack (in Appendix B.2.2) where an adversary could infer the presence/absence of certain data points based on the variations in the KNN-Shapley scores, analogous to the classic membership inference attack on ML model [58]. Additionally, we highlight the technical challenges in incorporating the current KNN-Shapley technique with differential privacy, such as its large global sensitivity. All these results emphasize the need for a new, privacy-friendly approach to data valuation.

TKNN-Shapley: an efficient, privacy-friendly data valuation technique (Section 4.2). To address the privacy concerns, we derive a novel variant of KNN-Shapley. This new method considers the Data Shapley of an alternative form of KNN classifier called Threshold-KNN (TKNN) [5], which takes into account all neighbors within a pre-specified threshold of a test example, rather than the exact K nearest neighbors. We derive the closed-form formula of Data Shapley for TKNN (i.e., TKNN-Shapley). We show that it can be computed exactly with exact linear-time computational efficiency over the original KNN-Shapley. DP-TKNN-Shapley (Section 5). Importantly, we recognize that TKNN-Shapley only depends on three simple counting queries. Hence, TKNN-Shapley can be conveniently transformed into a differentially private version by DP's post-processing property. Moreover, we prove that such a DP variant satisfies several favorable properties, including (1) the high computational efficiency, (2) the capability to withstand collusion among data holders without compromising the privacy guarantee, and (3) the ease of integrating subsampling for privacy amplification.

Numerical experiments (Section 6). We evaluate the performance of TKNN-Shapley across 11 commonly used benchmark datasets and 2 NLP datasets. Key observations include: (1) TKNNShapley surpasses KNN-Shapley in terms of computational efficiency; (2) DP-TKNN-Shapley significantly outperforms the naively privatized KNN-Shapley in terms of privacy-utility tradeoff in discerning data quality; (3) even non-private TKNN-Shapley achieves comparable performance as KNN-Shapley.

Overall, our work suggests that TKNN-Shapley, being a privacy-friendly, yet more efficient and effective alternative to the original KNN-Shapley, signifies a milestone toward practical data valuation.

2 Background of Data Valuation

In this section, we formalize the data valuation problem for ML, and review the method of Data Shapley and KNN-Shapley.

Setup & Goal. Consider a dataset D := { z i } N i =1 consisting of N data points where each data point z i := ( x i , y i ) is collected from a data owner i . The objective of data valuation is to attribute a score to each training data point z i , reflecting its importance or quality in ML model training. Formally, we aim to determine a score vector ( ϕ z i ) N i =1 , wherein ϕ z i represents the value of data point z i . For any reasonable data valuation method, the value of a data point is always relative to other data points in the dataset. For instance, if a data point has many duplicates in the dataset, its value will likely be lower. Hence, ϕ z i is a function of the leave-one-out dataset D -z i := D \ { z i } . We write ϕ z i ( D -z i ) when we want to stress the dependency of a data value score with the rest of the data points.

Utility Function. Most of the existing data valuation techniques are centered on the concept of utility function , which maps an input dataset to a score indicating the usefulness of the training set. A common choice for utility function is the validation accuracy of a model trained on the input training set. Formally, for a training set S , a utility function v ( S ) := acc ( A ( S )) , where A is a learning algorithm that takes a dataset S as input and returns a model; acc ( · ) is a metric function that evaluates the performance of a given model, e.g., the classification accuracy on a hold-out validation set.

2.1 The Shapley Value

The Shapley value (SV) [56] is a classic concept from game theory to attribute the total gains generated by the coalition of all players. At a high level, it appraises each point based on the (weighted) average utility change caused by adding the point into different subsets of the training set. Formally, given a utility function v ( · ) and a training set D , the Shapley value of a data point z ∈ D is defined as

For notation simplicity, when the context is clear, we omit the utility function and/or leave-one-out dataset, and write ϕ z ( D -z ) , ϕ z ( v ) or ϕ z depending on the specific dependency we want to stress.

The popularity of the Shapley value is attributable to the fact that it is the unique data value notion satisfying four axioms: Dummy player, Symmetry, Linearity, and Efficiency. We refer the readers to [23], [30] and the references therein for a detailed discussion about the interpretation and necessity of the four axioms in the ML context. Here, we introduce the linearity axiom which will be used later.

Theorem 1 (Linearity of the Shapley value [56]) . For any of two utility functions v 1 , v 2 and any α 1 , α 2 ∈ R , we have ϕ z ( α 1 v 1 + α 2 v 2 ) = α 1 ϕ z ( v 1 ) + α 2 ϕ z ( v 2 ) .

2.2 KNN-Shapley

Formula (1) suggests that the exact Shapley value can be computationally prohibitive in general, as it requires evaluating v ( S ) for all possible subsets S ⊆ D . Surprisingly, [29, 61] showed that for K -Nearest Neighbor (KNN), the computation of the exact Data Shapley score is highly efficient. Following its introduction, KNN-Shapley has rapidly gained attention and follow-up works across diverse areas of machine learning [24, 43, 42, 9]. In particular, it has been recognized by recent studies as 'the most practical data valuation technique capable of handling large-scale data effectively' [52, 33].

Specifically, the performance of an unweighted KNN classifier is typically evaluated by its validation accuracy. For a given validation set D (val) = { z (val) i } N val i =1 , we can define KNN's utility function v KNN D (val) on a non-empty training set S as v KNN D (val) ( S ) := ∑ z (val) ∈ D (val) v KNN z (val) ( S ) , where

is the probability of a (soft-label) KNN classifier in predicting the correct label for a validation point z (val) = ( x (val) , y (val) ) ∈ D (val) . In (2), π ( S ) ( i ; x (val) ) is the index of the i th closest data point in

Φ2,(D-z1)

Trusted Central Party

Z1

Z1

G

(z-a)zΦ

Φ22(D-22)

Z2=(x2,y2)

2(D-2)

DataValuation

ΦzN(D-2N)

ZN =(xN,yN)

ZN

Algorithm

ZN

(Nz-a)NzΦ

(e.g., KNN-Shapley)

S

Collected

DataValue

ReturnValue

Dataset

Scores

D-z1

Owners

S to x (val) . When | S | = 0 , v KNN z (val) ( S ) is set to the accuracy by random guessing. The distance is measured through suitable metrics such as ℓ 2 distance. Here is the main result in [29, 61]:

Theorem 2 (KNN-Shapley [30, 61] (simplified version)) . Consider the utility function in (2). Given a validation data point z (val) = ( x (val) , y (val) ) and a distance metric d ( · , · ) , if we sort the training set D = { z i = ( x i , y i ) } N i =1 according to d ( x i , x (val) ) in ascending order, then the Shapley value of each data point ϕ KNN z i corresponding to utility function v KNN z (val) can be computed recursively as follows:

where the exact form of functions f i ( D ) can be found in Appendix B.1.

Runtime: the computation of all Shapley values ( ϕ KNN z 1 , . . . , ϕ KNN z N ) can be achieved in O ( N log N ) runtime in total (dominated by the sorting data points in D ).

The Shapley value corresponding to full validation set: for a validation set D (val) , recall that v KNN D (val) ( S ) := ∑ z (val) ∈ D (val) v KNN z (val) ( S ) . One can compute the Shapley value corresponding to v KNN D (val) by summing each ϕ KNN z i ( v KNN z (val) ) , i.e., ϕ KNN z i ( v KNN D (val) ) = ∑ z (val) ∈ D (val) ϕ KNN z i ( v KNN z (val) ) .

Theorem 2 tells that for any validation data point z (val) , we can compute the exact Shapley value ϕ KNN z i ( D -z i ; v KNN z (val) ) for all z i ∈ D by using a recursive formula within a total runtime of O ( N log N ) . After computing the Shapley value ϕ KNN z i ( v KNN z (val) ) for each z (val) ∈ D (val) , one can compute the Shapley value corresponding to the full validation set by simply taking the sum of each ϕ KNN z i ( v KNN z (val) ) , due to the linearity property (Theorem 1) of the Shapley value.

Remark 1 ( Criteria of Computational Efficiency ) . As prior data valuation literature [23], we focus on the total runtime required to release all data value scores ( ϕ KNN z 1 , . . . , ϕ KNN z N ) . This is because, in a practical data valuation scenario (e.g., profit allocation), we rarely want to compute the data value score for only a single data point; instead, a typical objective is to compute the data value scores for all data points within the training set.

3 Privacy Risks & Challenges of Privatization for KNN-Shapley

Scenario. Figure 1 illustrates the data valuation scenario and potential privacy leakages considered in our paper. Specifically, a centralized, trusted server collects data point z i from data owner i for each i ∈ [ N ] . The central server's role is to provide each data owner i with an assessment of the value of their data z i , e.g., the KNN-Shapley value ϕ KNN z i . A real life example: Mayo Clinic has created a massive digital health patient data marketplace platform [68], where the patients submit part of their medical records onto the platform, and life science companies/labs pay a certain amount of money to purchase patients' data. The platform's responsibility is to gauge the worth of the data of each patient (i.e., the data owner) to facilitate fair compensation.

The privacy risks associated with KNN-Shapley (as well as other data valuation techniques) arise from the fact that ϕ KNN z i ( D -z i ) depends on other data owners' data D -z i . Consequently, the data value score ϕ KNN z i may inadvertently reveal private information (e.g., membership) about the rest of the dataset. The dependency of a data value score on the rest of the dataset is an unavoidable aspect of data valuation, as the value of a data point is inherently a relative quantity determined by its role within the complete dataset.

Remark 2 ( Other privacy risks in data valuation ) . It is important to note that in this work, we do not consider the privacy risks of revealing individuals' data to the central server. This is a different type of privacy risk that needs to be addressed using secure multi-party computation (MPC) technique [70], and it should be used together with differential privacy in practice. In addition, to use KNN-Shapley or many other data valuation techniques, the central server needs to maintain a clean, representative validation set, the privacy of which is not considered by this paper.

A Simple Membership Inference (MI) Attack on KNN-Shapley (detailed in Appendix B.2.2). We further illustrate the privacy risks of revealing data value scores with a concrete example. Analogous to the classic membership inference attack on ML model [58], in Appendix B.2.2 we show an example of privacy attack where an adversary could infer the presence/absence of certain data points in the dataset based on the variations in the KNN-Shapley scores. The design is analogous to the membership inference attack against ML models via the likelihood ratio test [8]. The AUROC score of the attack results is shown in Table 1. As we can see, our

K

= 1

K

= 3

K

= 5

K

= 7

K

= 9

K

= 11

K

= 13

K

= 15

2DPlanes

0.56

0.595

0.518

0.52

0.57

0.55

0.515

0.6

Phoneme

0.692

0.54

0.513

0.505

0.505

0.588

0.512

0.502

CPU

0.765

0.548

0.52

0.572

0.588

0.512

0.612

0.615

Fraud

0.625

0.645

0.6

0.592

0.622

0.532

0.538

0.558

Creditcard

0.542

0.643

0.628

0.665

0.503

0.67

0.602

0.66

Apsfail

0.532

0.595

0.625

0.6

0.53

0.645

0.532

0.52

Click

0.61

0.525

0.588

0.538

0.588

0.582

0.622

0.618

Wind

0.595

0.51

0.518

0.528

0.558

0.562

0.505

0.577

Pol

0.725

0.695

0.7

0.62

0.57

0.522

0.535

0.532

MIA attack can achieve a detection performance that is better than the random guess (0.5) for most of the settings. On some datasets, the attack performance can achieve > 0 . 7 AUROC. This demonstrates that privacy leakage in data value scores can indeed lead to non-trivial privacy attacks, and underscores the need for privacy safeguards in data valuation. 1

3.1 Privatizing KNN-Shapley is Difficult

The growing popularity of data valuation techniques, particularly KNN-Shapley, underscores the critical need to mitigate the inherent privacy risks. In the quest for privacy protection, differential privacy (DP) [15] has emerged as the leading framework. DP has gained considerable recognition for providing robust, quantifiable privacy guarantees, thereby becoming the de-facto choice in privacy protection. In this section, we introduce the background of DP and highlight the technical difficulties in constructing a differentially private variant for the current version of KNN-Shapley.

Background of Differential Privacy. Weuse D,D ′ ∈ N X to denote two datasets with an unspecified size over space X . We call two datasets D and D ′ adjacent (denoted as D ∼ D ′ ) if we can construct one by adding/removing one data point from the other, e.g., D = D ′ ∪ { z } for some z ∈ X .

Definition 3 (Differential Privacy [15]) . For ε, δ ≥ 0 , a randomized algorithm M : N X → Y is ( ε, δ ) -differentially private if for every pair of adjacent datasets D ∼ D ′ and for every subset of possible outputs E ⊆ Y , we have Pr M [ M ( D ) ∈ E ] ≤ e ε Pr M [ M ( D ′ ) ∈ E ] + δ .

That is, ( ε, δ ) -DP requires that for all adjacent datasets D,D ′ , the output distribution M ( D ) and M ( D ′ ) are close, where the closeness is measured by the parameters ε and δ . In our scenario, we would like to modify data valuation function ϕ z i ( D -z i ) to satisfy ( ε, δ ) -DP in order to protect the privacy of the rest of the dataset D -z i . Gaussian mechanism [14] is a common way for privatizing a function; it introduces Gaussian noise aligned with the function's global sensitivity , which is the maximum output change when a single data point is added/removed from any given dataset.

Definition 4 (Gaussian Mechanism [14]) . Define the global sensitivity of a function f : N X → R d as ∆ 2 ( f ) := sup D ∼ D ′ ∥ f ( D ) -f ( D ′ ) ∥ 2 . The Gaussian mechanism M with noise level σ is then given by M ( D ) := f ( D ) + N ( 0 , σ 2 d ) where M is ( ε, δ ) -DP with σ = ∆ 2 ( f ) √ log(1 . 25 /δ ) /ε .

Challenges in making KNN-Shapley being differentially private (overview). Here, we give an overview of the inherent difficulties in making the KNN-Shapley ϕ KNN z i ( D -z i ) (Theorem 2) to be differentially private, and we provide a more detailed discussion in Appendix B.3. (1) Large global sensitivity: In Appendix B.3, we show that the global sensitivity of ϕ KNN z i ( D -z i ) can significantly exceed the magnitude of ϕ KNN z i . Moreover, we prove that the global sensitivity bound cannot be further improved by constructing a specific pair of neighboring datasets that matches the bound. Hence, if we introduce random noise proportional to the global sensitivity bound, the resulting privatized data value score could substantially deviate from its non-private counterpart, thereby compromising the utility of the privatized data value scores. (2) Computational challenges in incorporating privacy amplification by subsampling: 'Privacy amplification by subsampling' [4] is a technique where the subsampling of a dataset amplifies the privacy guarantees due to the reduced probability of an individual's data being included. Being able to incorporate such a technique is often important for achieving a decent privacy-utility tradeoff. However, in Appendix B.3 we show that the recursive nature of KNN-Shapley computation causes a significant increase in computational demand compared to non-private KNN-Shapley.

These challenges underscore the pressing need for new data valuation techniques that retain the efficacy and computational efficiency of KNN-Shapley, while being amenable to privatization.

4 The Shapley Value for Threshold-based Nearest Neighbor

Considering the privacy concerns and privatization challenges associated with the original KNNShapley method, we introduce TKNN-Shapley , a privacy-friendly alternative of KNN-Shapley which also achieves improved computational efficiency. At the core of this novel method is Threshold-KNN (TKNN) classifier, a simple variant of the KNN classifier. We will discuss how to incorporate DP for TKNN-Shapley in Section 5.

4.1 Threshold-based Nearest Neighbor Classifier (TKNN)

Threshold-KNN (TKNN) [5, 72] is a variant of KNN classifier that considers neighbors within a pre-specified threshold of the query example, rather than exclusively focusing on the exact K nearest neighbors. Formally, for a training set S and a validation data point z (val) = ( x (val) , y (val) ) , we denote NB x (val) ,τ ( S ) := { ( x, y ) | ( x, y ) ∈ S, d ( x, x (val) ) ≤ τ } the set of neighbors of x (val) in S within a pre-specified threshold τ . Similar to the utility function for KNN, we define the utility function for TKNN classifier when using a validation set D (val) as the aggregated prediction accuracy v TKNN D (val) ( S ) := ∑ z (val) ∈ D (val) v TKNN z (val) ( S ) where

Comparison with standard KNN. (1) Robustness to outliers. Compared with KNN, TKNN is better equipped to deal with prediction phase outliers [5]. When predicting an outlier that is far from the entire training set, TKNN prevents the influence of distant, potentially irrelevant neighbors, leading to a more reliable prediction score for outliers. (2) Inference Efficiency. TKNN has slightly better computational efficiency compared to KNN, as it has O ( N ) instead of O ( N log N ) inference time. This improvement is achieved because TKNN only requires the computation of neighbors within the threshold τ , rather than searching for the exact K nearest neighbors. (3) TKNN is also a consistent estimator. The consistency of standard KNN is a well-known theoretical result [26]. That is, for any target function that satisfies certain regularity conditions, KNN binary classifier/regressor is guaranteed to converge to the target function as the size of the training set grows to infinite. In Appendix C.1, we derived a similar consistency result for TKNN binary classifier/regressor.

Remark 3 ( Intuition: Why we consider TKNN? ) . The 'recursive form' of KNN-Shapley is due to the sorting operation in the prediction of the standard KNN. The recursive form of the formula causes difficulties in incorporating KNN-Shapley with differential privacy. In contrast, TKNN avoids the recursive formula for its Data Shapley value; the intuition is that for TKNN, the selection of neighbors for prediction solely depends on the queried example and the validation data, and is independent of the other training data points.

4.2 Data Shapley for TKNN (TKNN-Shapley)

With the introduction of TKNN classifier and its utility function, we now present our main result, the closed-form, efficiently computable Data Shapley formula for the TKNN classifier.

Theorem 5 (TKNN-Shapley (simplified version)) . Consider the utility function v TKNN z (val) in (3). Given a validation data point z (val) = ( x (val) , y (val) ) and a distance metric d ( · , · ) , the Shapley value ϕ TKNN z i of each training point z i = ( x i , y i ) ∈ D corresponding to utility function v TKNN z (val) can be calculated as

where C z (val) := ( c , c x (val) ,τ , c (+) z (val) ,τ ) is a 3-dimensional function/vector s.t.

and f ( · ) is a function whose exact form can be found in Appendix C.2.

Runtime: the computation of all ( ϕ TKNN z 1 , . . . , ϕ TKNN z N ) can be achieved in O ( N ) runtime (see Appendix C.2.1).

The Shapley value when using full validation set: The Shapley value corresponding to the utility function v TKNN D (val) can be calculated as ϕ TKNN z i ( v TKNN D (val) ) = ∑ z (val) ∈ D (val) ϕ TKNN z i ( v TKNN z (val) ) .

The main technical challenges of proving Theorem 5 lies in showing C z (val) = ( c , c x (val) ,τ , c (+) z (val) ,τ ) , three simple counting queries on D -z i , are the key quantities for computing ϕ TKNN z i . In later part of the paper, we will often view ϕ TKNN z i as a function of C z (val) and denote ϕ TKNN z i [ C z (val) ] := f ( C z (val) ) when we want to stress this dependency. The full version and the proof for TKNN-Shapley can be found in Appendix C.2.1. Here, we briefly discuss why TKNN-Shapley can achieve O ( N ) runtime in total .

Efficient Computation of TKNN-Shapley. As we can see, all of the quantities in C z (val) are simply counting queries on D -z i , and hence C z (val) ( D -z i ) can be computed in O ( N ) runtime for any z i ∈ D . A more efficient way to compute C z (val) ( D -z i ) for all z i ∈ D , however, is to first compute C z (val) ( D ) on the full dataset D , and we can easily show that each of C z (val) ( D -z i ) can be computed from that in O (1) runtime (see Appendix C.2.2 for details). Hence, we can compute TKNN-Shapley ϕ TKNN z i ( v TKNN z (val) ) for all z i ∈ D within an overall computational cost of O ( N ) .

Comparison with KNN-Shapley. TKNN-Shapley offers several advantages over the original KNNShapley. (1) Non-recursive: In contrast to the KNN-Shapley formula (Theorem 2), which is recursive, TKNN-Shapley has an explicit formula for computing the Shapley value of every point z i . This non-recursive nature not only simplifies the implementation, but also makes it straightforward to incorporate techniques like subsampling. (2) Computational efficiency: TKNN-Shapley has O ( N ) runtime in total, which is better than the O ( N log N ) runtime for KNN-Shapley.

5 Differentially Private TKNN-Shapley

Having established TKNN-Shapley as an alternative to KNN-Shapley which shares many advantages, our next step is to develop a new version of TKNN-Shapley that incorporates differential privacy. In this section, we introduce our proposed Differentially Private TKNN-Shapley (DP-TKNN-Shapley).

5.1 Differentially private release of ϕ TKNN z i ( v TKNN z (val) )

As C z (val) = ( c , c x (val) ,τ , c (+) z (val) ,τ ) are the only quantities that depend on the dataset D -z i in (4), privatizing these quantities is sufficient for overall privacy preservation, as DP guarantee is preserved under any post-processing [16]. Since all of c , c x (val) ,τ and c (+) z (val) ,τ are just counting queries, the function of C z (val) ( · ) has global sensitivity √ 3 in ℓ 2 -norm. This allows us to apply the commonly used Gaussian mechanism (Theorem 4) to release C z (val) in a differentially private way, i.e., ̂ C z (val) ( D -z i ) := round ( C z (val) ( D -z i ) + N ( 0 , σ 2 ✶ 3 )) where ̂ C z (val) is the privatized version of C z (val) , and the function round rounds each entry of noisy value to the nearest integer. The differentially private version of TKNN-Shapley value ̂ ϕ TKNN z i ( D -z i ) is simply the value score computed by

(4) but use the privatized quantities ̂ C z (val) . The privacy guarantee of releasing DP-TKNN-Shapley ̂ ϕ TKNN z i ( D -z i ) follows from the guarantee of Gaussian mechanism (see Appendix D.1 for proof).

While our approach may seem simple, we stress that simplicity is appreciated in DP as complex mechanisms pose challenges for correct implementation and auditing [47, 21].

5.2 Advantages of DP-TKNN-Shapley (Overview)

We give an overview (detailed in Appendix D.2) of several advantages of DP-KNN-Shapley here, including the efficiency, collusion resistance, and simplicity in incorporating subsampling.

By reusing privatized statistics, ̂ ϕ TKNN z i can be efficiently computed for all z i ∈ D . Recall that in practical data valuation scenarios, it is often desirable to compute the data value scores for all of z i ∈ D . As we detailed in Section 4.2, for TKNN-Shapley, such a computation only requires O ( N ) runtime in total if we first compute C z (val) ( D ) and subsequently C z (val) ( D -z i ) for each z i ∈ D . In a similar vein, we can efficiently compute the DP variant ̂ ϕ TKNN z i for all z i ∈ D . To do so, we first calculate ̂ C z (val) ( D ) and then, for each z i ∈ D , we compute ̂ C z (val) ( D -z i ) . It is important to note that when releasing ̂ ϕ TKNN z i to individual i , the data point they hold, z i , is not private to the individual themselves. Therefore, as long as ̂ C z (val) ( D ) is privatized, ̂ C z (val) ( D -z i ) and hence ̂ ϕ TKNN z i also satisfy the same privacy guarantee due to DP's post-processing property.

By reusing privatized statistics, DP-TKNN-Shapley is collusion resistance. In Section 5.1, we consider the single Shapley value ϕ TKNN z i ( D -z i ) as the function to privatize. However, when we consider the release of all Shapley values ( ̂ ϕ TKNN z 1 , . . . , ̂ ϕ TKNN z N ) as a unified mechanism, one can show that such a mechanism satisfies joint differential privacy (JDP) [34] if we reuse the privatized statistic ̂ C z (val) ( D ) for the release of all ̂ ϕ TKNN z i . The consequence of satisfying JDP is that our mechanism is resilient against collusion among groups of individuals without any privacy degradation. That is, even if an arbitrary group of individuals in [ N ] \ i colludes (i.e., shares their respective ̂ ϕ TKNN z j values within the group), the privacy of individual i remains uncompromised. Our method also stands resilient in scenarios where a powerful adversary sends multiple data points and receives multiple value scores.

Incorporating Privacy Amplification by Subsampling. In contrast to KNN-Shapley, the nonrecursive nature of DP-TKNN-Shapley allows for the straightforward incorporation of subsampling. Besides the privacy guarantee, subsampling also significantly boosts the computational efficiency of DP-TKNN-Shapley.

5.3 Differentially private release of ϕ TKNN z i ( v TKNN D (val) )

Recall that the TKNN-Shapley corresponding to the full validation set D (val) is ϕ TKNN z i ( v TKNN D (val) ) = ∑ z (val) ∈ D (val) ϕ TKNN z i ( D -z i ; z (val) ) . We can compute privatized ϕ TKNN z i ( v TKNN D (val) ) by simply releasing ϕ TKNN z i ( D -z i ; z (val) ) for all z (val) ∈ D (val) . To better keep track of the privacy cost, we use the current state-of-the-art privacy accounting technique based on the notion of the Privacy Loss Random Variable (PRV) [17]. We provide more details about the background of privacy accounting in Appendix D.3. We note that PRV-based privacy accountant computes the privacy loss numerically, and hence the final privacy loss has no closed-form expressions.

6 Numerical Experiments

In this section, we systematically evaluate the practical effectiveness of our proposed TKNN-Shapley method. Our evaluation aims to demonstrate the following points: (1) TKNN-Shapley offers improved runtime efficiency compared with KNN-Shapley. (2) The differentially private version of TKNN-Shapley (DP-TKNN-Shapley) achieves significantly better privacy-utility tradeoff compared to naively privatized KNN-Shapley in discerning data quality. (3) Non-private TKNN-Shapley maintains a comparable performance to the original KNN-Shapley. These observations highlight TKNN-Shapley's potential for data valuation in real-life applications. Detailed settings for our experiments are provided in Appendix E.

6.1 Computational Efficiency

We evaluate the runtime efficiency of TKNN-Shapley in comparison to KNN-Shapley (see Appendix E.2 for details as well as more experiments). We choose a range of training data sizes N , and compare the runtime of both methods at each N . As demonstrated in Figure 2, TKNN-Shapley achieves better computational efficiency than KNN-Shapley across all training data sizes. In particular, TKNN-Shapley is around 30% faster than KNNShapley for large N . This shows the computational advantage of TKNN-Shapley mentioned in Section 4.2.

6.2 Discerning Data Quality

In this section, we evaluate the performance of both DP and non-DP version of TKNN-Shapley in discerning data quality. Tasks: We evaluate the performance of TKNNShapley on two standard tasks: mislabeled data detection and noisy data detection, which are tasks that are often used for evaluating the performance of data valuation tech-

niques in prior works [39, 60]. Since mislabeled/noisy data often negatively affect the model performance, it is desirable to assign low values to these data points. In the experiment of mislabeled (or noisy) data detection, we randomly choose 10% of the data points and flip their labels (or add strong noise to their features). Datasets: We conduct our experiments on a diverse set of 13 datasets, where 11 of them have been used in previous data valuation studies [39, 60]. Additionally, we experiment on 2 NLP datasets (AG News [65] and DBPedia [2]) that have been rarely used in the past due to their high-dimensional nature and the significant computational resources required. Settings & Hyperparameters of TKNN-/KNN-Shapley: for both TKNN/KNN-Shapley, we use the popular cosine distance as the distance measure [53], which is always bounded in [ -1 , +1] . Throughout all experiments, we use τ = -0 . 5 and K = 5 for TKNN-/KNN-Shapley, respectively, as we found the two choices consistently work well across all datasets. We conduct ablation studies on the choice of hyperparameters in Appendix E.

6.2.1 Experiment for Private Setting

Baselines: (1) DP-KNN-Shapley without subsampling. Recall from Section 3.1, the original KNN-Shapley has a large global sensitivity. Nevertheless, we can still use Gaussian mechanism to privatize it based on its global sensitivity bound. We call this approach as DP-KNN-Shapley. (2) DP-KNN-Shapley with subsampling. Recall from Section 3.1, it is computationally expensive to incorporate subsampling techniques for DP-KNN-Shapley (detailed in Appendix B.3.2). For instance, subsampled DP-KNN-Shapley with subsampling rate q = 0 . 01 generally takes 30 × longer time compared with non-subsampled counterpart. Nevertheless, we still compare with subsampled DP-KNN-Shapley for completeness. These two baselines are detailed in Appendix B.4. We also note that an unpublished manuscript [67] proposed a DP version of Data Shapley. However, their DP guarantee requires a hard-to-verify assumption of uniform stability (see Appendix A). Hence, we do not compare with [67].

Results: We evaluate the privacy-utility tradeoff of DP-TKNN-Shapley. Specifically, for a fixed δ , we examine the AUROC of mislabeled/noisy data detection tasks at different values of ε , where ε is adjusted by changing the magnitude of Gaussian noise. In the experiment, we set the subsampling rate q = 0 . 01 for TKNN-Shapley and subsampled KNN-Shapley. Table 2 shows the results on 3 datasets, and we defer the results for the rest of 10 datasets to Appendix E.3.2. As we can see, DP-TKNN-Shapley shows a substantially better privacy-utility tradeoff compared to both DP-KNN-Shapley with/without subsampling. In particular, DP-TKNN-Shapley maintains a high

AUROC even when ε ≈ 0 . 1 . The poor performance of DP-KNN-Shapley is due to its relatively high global sensitivity.

6.2.2 Experiment for Non-private Setting

Baselines: Our main baseline for comparison is KNN-Shapley. For completeness, we also compare with other classic, yet much less efficient data valuation techniques, such as Data Shapley [23], Data Banzhaf [60], and leave-one-out error (LOO) [35]. Due to space constraints, we only show the results for the famous Data Shapley here and defer other methods' results to Appendix E.3.3.

Results: We use AUROC as the performance metric on mislabeled/noisy data detection tasks. Due to space constraints, we defer the results for the task of noisy data detection to Appendix E. Table 3 shows the results for the task of mislabeled data detection across all 13 datasets we use. As we can see, TKNN-Shapley shows a comparable performance as KNN-Shapley across almost all datasets, demonstrating that TKNN-Shapley matches the effectiveness of KNN-Shapley in discerning data quality. Moreover, KNN-Shapley (and TKNN-Shapley) has a significantly better performance compared to Data Shapley, which is consistent with the observations in many existing studies [31, 52]. The poor performance of Data Shapley is attributable to the sample inefficiency and stochasticity during retraining [60].

7 Conclusion

In this work, we uncover the inherent privacy risks associated with data value scores and introduce TKNN-Shapley, a privacy-friendly alternative to the widely-used KNN-Shapley. We demonstrate that TKNN-Shapley outperforms KNN-Shapley in terms of computational efficiency, and is as good as discerning data quality. Moreover, the privatized version of TKNN-Shapley significantly surpasses the naively privatized KNN-Shapley on tasks such as mislabeled data detection.

Future Work. (1) Privacy risks of data revelation to central server: in this work, we assume the existence of a trusted central server, and we do not consider the privacy risks associated with revealing individuals' data to the central server. Future work should consider integrating secure multi-party computation (MPC) techniques to mitigate this risk [59]. MPC can allow the computation of KNN-Shapley without revealing individual data to the central server, thereby preserving privacy. We envision an end-to-end privacy-preserving data valuation framework that combines both DP and MPC. (2) Impact of Randomization on Payment Fairness: the incorporation of differential privacy necessarily adds a degree of randomness to the data value scores. This randomization could potentially impact the fairness of payments to data providers [3]. The influence of this randomness, and its potential implications for payment fairness, remain areas for further investigation.

Acknowledgments

This work was supported in part by the National Science Foundation under grants CNS-2131938, CNS1553437, CNS-1704105, CNS-2048091, IIS-2312794, IIS-2313130, OAC-2239622, the ARL's Army Artificial Intelligence Innovation Institute (A2I2), the Office of Naval Research Young Investigator Award, the Army Research Office Young Investigator Prize, Schmidt DataX award, Princeton Effiliates Award, Amazon-Virginia Tech Initiative in Efficient and Robust Machine Learning, the Commonwealth Cyber Initiative, a Google PhD Fellowship, and a Princeton's Gordon Y. S. Wu Fellowship. We are grateful to anonymous reviewers at NeurIPS for their valuable feedback.

References

Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pages 308-318, 2016.

Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. Dbpedia: A nucleus for a web of open data. In The Semantic Web: 6th International Semantic Web Conference, 2nd Asian Semantic Web Conference, ISWC 2007+ ASWC 2007, Busan, Korea, November 11-15, 2007. Proceedings , pages 722-735. Springer, 2007.

Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. Differential privacy has disparate impact on model accuracy. Advances in neural information processing systems , 32, 2019.

Borja Balle, Gilles Barthe, and Marco Gaboardi. Privacy amplification by subsampling: Tight analyses via couplings and divergences. Advances in Neural Information Processing Systems , 31, 2018.

Jon L Bentley. Survey of techniques for fixed radius near neighbor searching. Technical report, Stanford Linear Accelerator Center, Calif.(USA), 1975.

Yatao Bian, Yu Rong, Tingyang Xu, Jiaxiang Wu, Andreas Krause, and Junzhou Huang. Energybased learning for cooperative games, with applications to valuation problems in machine learning. arXiv preprint arXiv:2106.02938 , 2021.

Mark Alexander Burgess and Archie C Chapman. Approximating the shapley value using stratified empirical bernstein sampling. In IJCAI , pages 73-81, 2021.

Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP) , pages 1897-1914. IEEE, 2022.

Christie Courtnage and Evgueni Smirnov. Shapley-value data valuation for semi-supervised learning. In Discovery Science: 24th International Conference, DS 2021, Halifax, NS, Canada, October 11-13, 2021, Proceedings 24 , pages 94-108. Springer, 2021.

Rachel Cummings, Hadi Elzayn, Emmanouil Pountourakis, Vasilis Gkatzelis, and Juba Ziani. Optimal data acquisition with privacy-aware agents. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) , pages 210-224. IEEE, 2023.

Andrea Dal Pozzolo, Olivier Caelen, Reid A Johnson, and Gianluca Bontempi. Calibrating probability with undersampling for unbalanced classification. In 2015 IEEE Symposium Series on Computational Intelligence , pages 159-166. IEEE, 2015.

Irit Dinur and Kobbi Nissim. Revealing information while preserving privacy. In Proceedings of the twenty-second ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems , pages 202-210, 2003.

Pradeep Dubey, Abraham Neyman, and Robert James Weber. Value theory without efficiency. Mathematics of Operations Research , 6(1):122-128, 1981.

Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Advances in CryptologyEUROCRYPT 2006: 24th Annual International Conference on the Theory and Applications of Cryptographic Techniques, St. Petersburg, Russia, May 28-June 1, 2006. Proceedings 25 , pages 486-503. Springer, 2006.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference , pages 265-284. Springer, 2006.

Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science , 9(3-4):211-407, 2014.

Cynthia Dwork and Guy N. Rothblum. Concentrated differential privacy. CoRR , abs/1603.01887, 2016.

Cynthia Dwork, Guy N Rothblum, and Salil Vadhan. Boosting and differential privacy. In 2010 IEEE 51st Annual Symposium on Foundations of Computer Science , pages 51-60. IEEE, 2010.

Cynthia Dwork, Adam Smith, Thomas Steinke, and Jonathan Ullman. Exposed! a survey of attacks on private data. Annual Review of Statistics and Its Application , 4:61-84, 2017.

Alireza Fallah, Ali Makhdoumi, Azarakhsh Malekian, and Asuman Ozdaglar. Optimal and differentially private data acquisition: Central and local mechanisms. In Proceedings of the 23rd ACM Conference on Economics and Computation , pages 1141-1141, 2022.

Marco Gaboardi, Michael Hay, and Salil Vadhan. A programming framework for opendp. Manuscript, May , 2020.

Amirata Ghorbani, Michael Kim, and James Zou. A distributional framework for data valuation. In International Conference on Machine Learning , pages 3535-3544. PMLR, 2020.

Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning , pages 2242-2251. PMLR, 2019.

Amirata Ghorbani, James Zou, and Andre Esteva. Data shapley valuation for efficient batch active learning. In 2022 56th Asilomar Conference on Signals, Systems, and Computers , pages 1456-1462. IEEE, 2022.

Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz. Numerical composition of differential privacy. Advances in Neural Information Processing Systems , 34:11631-11642, 2021.

László Györfi, Michael Köhler, Adam Krzy˙ zak, and Harro Walk. A distribution-free theory of nonparametric regression , volume 1. Springer, 2002.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Ferenc Illés and Péter Kerényi. Estimation of the shapley value by ergodic sampling. arXiv preprint arXiv:1906.05224 , 2019.

Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas J Spanos, and Dawn Song. Efficient task-specific data valuation for nearest neighbor algorithms. Proceedings of the VLDB Endowment , 2019.

Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J Spanos. Towards efficient data valuation based on the shapley value. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1167-1176. PMLR, 2019.

Ruoxi Jia, Fan Wu, Xuehui Sun, Jiacen Xu, David Dao, Bhavya Kailkhura, Ce Zhang, Bo Li, and Dawn Song. Scalability vs. utility: Do we have to sacrifice one for the other in data importance quantification? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8239-8247, 2021.

Justin Kang, Ramtin Pedarsani, and Kannan Ramchandran. The fair value of data under heterogeneous privacy constraints. arXiv preprint arXiv:2301.13336 , 2023.

Bojan Karlaš, David Dao, Matteo Interlandi, Bo Li, Sebastian Schelter, Wentao Wu, and Ce Zhang. Data debugging with shapley importance over end-to-end machine learning pipelines. arXiv preprint arXiv:2204.11131 , 2022.

Michael Kearns, Mallesh Pai, Aaron Roth, and Jonathan Ullman. Mechanism design in large games: Incentives and privacy. In Proceedings of the 5th conference on Innovations in theoretical computer science , pages 403-410, 2014.

Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning , pages 1885-1894. PMLR, 2017.

Antti Koskela and Antti Honkela. Computing differential privacy guarantees for heterogeneous compositions using fft. CoRR , abs/2102.12412, 2021.

Antti Koskela, Joonas Jälkö, and Antti Honkela. Computing tight differential privacy guarantees using fft. In International Conference on Artificial Intelligence and Statistics , pages 2560-2569. PMLR, 2020.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Yongchan Kwon and James Zou. Beta shapley: a unified and noise-reduced data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics , pages 8780-8802. PMLR, 2022.

Yongchan Kwon and James Zou. Data-oob: Out-of-bag estimate as a simple and efficient data value. ICML , 2023.

Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/ , 1998.

Weixin Liang, Kai-Hui Liang, and Zhou Yu. Herald: An annotation efficient method to detect user disengagement in social conversations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 3652-3665, 2021.

Weixin Liang, James Zou, and Zhou Yu. Beyond user self-reported likert scale ratings: A comparison model for automatic dialog evaluation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 1363-1374, 2020.

Jinkun Lin, Anqi Zhang, Mathias Lécuyer, Jinyang Li, Aurojit Panda, and Siddhartha Sen. Measuring the effect of training data on deep learning predictions via randomized experiments. In International Conference on Machine Learning , pages 13468-13504. PMLR, 2022.

Jinfei Liu, Jian Lou, Junxu Liu, Li Xiong, Jian Pei, and Jimeng Sun. Dealer: an end-to-end model marketplace with differential privacy. Proceedings of the VLDB Endowment , 14(6), 2021.

Xinjian Luo, Yangfan Jiang, and Xiaokui Xiao. Feature inference attack on shapley values. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security , pages 2233-2247, 2022.

Min Lyu, Dong Su, and Ninghui Li. Understanding the sparse vector technique for differential privacy. Proceedings of the VLDB Endowment , 10(6), 2017.

Rory Mitchell, Joshua Cooper, Eibe Frank, and Geoffrey Holmes. Sampling permutations for shapley value estimation. 2022.

Arvind Narayanan and Vitaly Shmatikov. How to break anonymity of the netflix prize dataset. arXiv preprint cs/0610105 , 2006.

Ramin Okhrati and Aldo Lipani. A multilinear sampling algorithm to estimate shapley values. In 2020 25th International Conference on Pattern Recognition (ICPR) , pages 7992-7999. IEEE, 2021.

OpenAI. Planning for agi and beyond, 2023.

Konstantin D Pandl, Fabian Feiland, Scott Thiebes, and Ali Sunyaev. Trustworthy machine learning for health care: scalable data valuation with the shapley value. In Proceedings of the Conference on Health, Inference, and Learning , pages 47-57, 2021.

Sabita Rajbanshi. Chapter 1: K nearest neighbors (supervised machine learning algorithm), 2021.

Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bertnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, 11 2019.

Pierangela Samarati and Latanya Sweeney. Protecting privacy when disclosing information: k-anonymity and its enforcement through generalization and suppression. 1998.

Lloyd S Shapley. A value for n-person games. Contributions to the Theory of Games , 2(28):307317, 1953.

Dongsub Shim, Zheda Mai, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang. Online class-incremental continual learning with adversarial shapley value. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 9630-9638, 2021.

Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP) , pages 3-18. IEEE, 2017.

Zhihua Tian, Jian Liu, Jingyu Li, Xinle Cao, Ruoxi Jia, and Kui Ren. Private data valuation and fair payment in data marketplaces. arXiv preprint arXiv:2210.08723 , 2022.

Jiachen T Wang and Ruoxi Jia. Data banzhaf: A robust data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics , pages 6388-6421. PMLR, 2023.

Jiachen T Wang and Ruoxi Jia. A note on" efficient task-specific data valuation for nearest neighbor algorithms". arXiv preprint arXiv:2304.04258 , 2023.

Jiachen T Wang and Ruoxi Jia. A note on" towards efficient data valuation based on the shapley value'. arXiv preprint arXiv:2302.11431 , 2023.

Tianhao Wang, Johannes Rausch, Ce Zhang, Ruoxi Jia, and Dawn Song. A principled approach to data valuation for federated learning. In Federated Learning , pages 153-167. Springer, 2020.

Tianhao Wang, Yu Yang, and Ruoxi Jia. Improving cooperative game theory-based data valuation via data utility learning. ICLR 2022 Workshop on Socially Responsible Machine Learning , 2022.

Zhen Wang, Xu Shan, Xiangxie Zhang, and Jie Yang. N24news: A new dataset for multimodal news classification. arXiv preprint arXiv:2108.13327 , 2021.

Mark Warner. Warner & hawley introduce bill to force social media companies to disclose how they are monetizing user data, 2019.

Lauren Watson, Rayna Andreeva, Hao-Tsung Yang, and Rik Sarkar. Differentially private shapley values for data evaluation. arXiv preprint arXiv:2206.00511 , 2022.

Cynthia Weiss. Mayo clinic platform: A patient's experience and beyond.

Zhaoxuan Wu, Yao Shu, and Bryan Kian Hsiang Low. Davinz: Data valuation using deep neural networks at initialization. In International Conference on Machine Learning , pages 24150-24176. PMLR, 2022.

Andrew C Yao. Protocols for secure computations. In 23rd annual symposium on foundations of computer science (sfcs 1982) , pages 160-164. IEEE, 1982.

I-Cheng Yeh and Che-hui Lien. The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert systems with applications , 36(2):2473-2480, 2009.

Yuqing Zhu, Xuandong Zhao, Chuan Guo, and Yu-Xiang Wang. Private prediction strikes back! Private kernelized nearest neighbors with individual Rényi filter. In Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence , pages 2586-2596, 2023.

A Extended Related Work

KNN-Shapley & Its Applications. The Shapley value is known for being computationally expensive. Fortunately, [29] found that computing the Data Shapley for K-Nearest Neighbors (KNN), one of the most classic yet still popular ML algorithms, is surprisingly easy and efficient. To the best of our knowledge, unweighted KNN is the only commonly-used ML model for which the exact Data Shapley can be efficiently computed, referred to as 'KNN-Shapley'. Owing to its superior computational efficiency and effectiveness in distinguishing data quality, KNN-Shapley has become one of the most popular and practical data valuation techniques. In the realm of ML research, for instance, [24] extends KNN-Shapley to active learning, [57] employs it in a continual learning setting, [43, 42] utilize KNN-Shapley for removing confusing samples in NLP applications, and [9] adopts KNN-Shapley for data valuation in semi-supervised learning. Furthermore, KNN-Shapley has also proven to be highly practical in real-life applications: [52] demonstrates that KNN-Shapley is the only practical data valuation technique for valuing large amounts of healthcare data, and [33] builds the first data debugging system for end-to-end ML pipelines based on KNN-Shapley.

Remark 5 (KNN-Shapley vs General Data Shapley) . In comparison to the work of general Data Shapley [23] (including Beta Shapley [39] as well as Data Banzhaf [60]), KNN-Shapley may have the following differences: (1) KNN-Shapley focuses on KNN classifiers. As a result, the applicability of KNN-Shapley scores as a proxy of data points' value with respect to other ML models may not be straightforward. However, it is noteworthy that KNN is asymptotically Bayes optimal, implying that KNN-Shapley scores can be justified as a proxy for the data's value relative to the best possible model, i.e., the Bayes classifier, under certain asymptotic conditions. (2) For high-dimensional data, such as images, KNN-Shapley requires a public model to first map the original data into data embeddings, and evaluates the value of these embeddings rather than the original data. While this is indeed a constraint in certain scenario, it is important to recognize that utilizing a publicly available foundation model to convert original data into embeddings, followed by the fine-tuning of the model's last layer, has become a common practice. Therefore, in many situations, it might be more desirable to evaluate the value of data embeddings instead of the original data.

Privacy and Data Valuation. Few studies in the literature consider the privacy risks of data valuation. [59] explores a scenario where a trusted server does not exist, and different data holders collaboratively compute each other's Shapley values without actually examining the data holder's data points, utilizing Multi-party Computation (MPC) techniques. The privacy risks addressed in [59] are orthogonal to those in our paper, and we can combine both techniques for end-to-end privacy protection. Another orthogonal line of works [45, 20, 10, 32] studies the scenario of a central platform collecting private data from privacy-aware agents and offering a differentially private statistic computed from the submitted data as a service in return. Agents consider the privacy costs and benefits of obtaining the statistic when deciding whether to participate and reveal their data truthfully. [46] proposes a privacy attack on the Shapley value for feature attribution, while in this work we study the privacy risks when using the Shapley value for data valuation.

The unpublished manuscript by [67] is the work most closely related to ours. [67] explores how to make the Shapley values of a data point to be differentially private against the rest of the dataset. However, instead of focusing on KNN-Shapley, they study the privatization of the less practical, retrain-based Data Shapley [23, 30]. Furthermore, their algorithm is highly restrictive in that the differential privacy guarantee relies on the "uniform stability" assumption, which is not verifiable for modern learning algorithms such as neural networks. Moreover, while [67] argues the "uniform stability" assumption holds for Logistic regression on bounded data domain, it is unclear whether the uniform stability assumption still holds when the Logistic regression is trained by SGD (which may involve training stochasticity and early stopping). In addition, [67] does not release the implementation. Hence, in our experiment, we do not compare with it.

Remark 6 (Brief background for the privacy risks in releasing aggregated statistics) . Since 1998, researchers have observed that a lot of seemingly benevolent aggregate statistics of a dataset can be used to reveal sensitive information about individuals [55]. A classic example is Netflix Prize fiasco, where the researchers show that an anonymized dataset can leak many sensitive information about individuals [49]. Dinur and Nissim [12] proved that 'revealing too many statistics too accurately leads to data privacy breach'. A great amount of discussion and practical realization of these privacy attacks on aggregated statistics can be found in [19]. In 2020, the US Census Bureau used these privacy attacks to justify its use of differential privacy.

KNN-Shapley score for an individual is one kind of aggregated statistic that depends on the rest of the dataset. Hence, KNN-Shapley score intrinsically reveals private information about the rest of the dataset (where we use membership inference attack as a concrete example in our paper). In addition, when users collude, their KNN-shapley values can be combined to make joint inferences about the rest of the dataset.

B Details about KNN-Shapley, its Privacy Risks & Challenges of Privatization

B.1 Full version of KNN-Shapley

KNN-Shapley was originally proposed in [29] and was later refined in [61]. Specifically, [61] considers the KNN's utility function formula (2) we present in the main text. Here is the main result of [61]:

Theorem 7 (KNN-Shapley [61]) . Consider the utility function in (2). Given a validation data point z (val) = ( x (val) , y (val) ) and a distance metric d ( · , · ) , if we sort the training set D = { z i = ( x i , y i ) } N i =1 according to d ( x i , x (val) ) in ascending order, then the Shapley value of each data point ϕ KNN z i corresponding to utility function v KNN z (val) can be computed recursively as follows:

where C denotes the number of classes for the classification task.

B.1.1 Older version of KNN-Shapley from [29]

Prior to the current formulation of KNN-Shapley introduced by [61], [29] proposed an older version of the algorithm. There is only a small distinction between the updated version by [61] and the older version from [29]. Specifically, the difference between the versions proposed by [61] and [29] lies in the utility function considered when computing the Shapley value. [61] use the utility function (2) as presented in the main text, while the utility function used in [29] is slightly less interpretable:

That is, (2) in the maintext divides the number of correct predictions by min( K, | S | ) , which can be interpreted as the likelihood of the soft-label KNN classifier predicting the correct label y (val) for x (val) . On the other hand, the function above (5) divides the number of correct predictions by K , which is less interpretable when | S | < K .

Nevertheless, the main result in [29] shows the following:

Theorem 8 (Older Version of KNN-Shapley from [29] 2 ) . Consider the utility function in (5). Given a validation data point z (val) = ( x (val) , y (val) ) and a distance metric d ( · , · ) , if we sort the training set D = { z i = ( x i , y i ) } N i =1 according to d ( x i , x (val) ) in ascending order, then the Shapley value of each data point ϕ KNN-OLD z i corresponding to utility function v KNN-OLD z (val) can be computed recursively as follows:

As we can see, both versions of KNN-Shapley have recursive forms. Since the utility functions they consider are very similar to each other except for the normalization term for small subsets, the two versions of KNN-Shapley have very close value scores in practice and as shown in the experiments of [61], the two versions of KNN-Shapley perform very similarly.

Remark 7 ( The use of older version of KNN-Shapley for DP-related experiments ) . In the experiments, we use the more advanced version of the KNN-Shapley from [61] except for the DPrelated experiments. This is because the global sensitivity of KNN-Shapley is difficult to derive, and we can only derive the global sensitivity of the older version of KNN-Shapley from [29] (as we will

show in Appendix B.3). Hence, we can only use the older version of KNN-Shapley from [29] as the baseline in DP-related experiments. It is important to note that for non-DP experiments, the performance of KNN-Shapley and its older variant is very close to each other, and which version of the KNN-Shapley we use for non-DP experiments does not affect the final conclusion .

B.2 Settings & Additional Experiments for Privacy Risks for KNN-Shapley

B.2.1 Experiment for the changes of data value after eliminating nearby points

We first investigate the impact of removing a data point on the KNN-Shapley score of another data point that is close to the removed one. Specifically, we calculate the KNN-Shapley score for a chosen data point in the dataset, and then repeat the process after eliminating one of its nearby points from the dataset.

Settings. Weuse commonly used datasets in the past literature, and the details for data preprocessing can be found in Appendix E.1. We calculate the KNN-Shapley score for a randomly selected data point in the dataset, and then repeat the computation of the KNN-Shapley score after we eliminate its nearest neighbor from the dataset.

Results. The results for a variety of other datasets are shown in Figure 3. We can see a significant difference in the KNN-Shapley score of the investigated data point depending on whether the nearest data point has been removed. We remark that how the data value change (increases or decreases) when the nearest data point is excluded depends on the label of the nearest data point as well as the validation data being used.

dataset:fraud

dataset:creditcard

dataset: apsfail

1.3

0.5

with thenearbydatapoint

KNN-Shapley

1.2

without the nearby data point

KNN-Shapley

KNN-Shapley

1.06

1.1

0.4

1.01

1.04

0.9

0.3

with the nearby data point

withthenearbydatapoint

1.02

0.8

without the nearby data point

without the nearby data point

5

10

15

5

10

15

5

10

15

K

K

K

dataset:phoneme

dataset:click

dataset:wind

1.4

with the nearby data point

0.40

0.70

without thenearby datapoint

KNN-Shapley

KNN-Shapley

0.35

0.65

1.0

0.30

0.60

0.25

with the nearby data point

0.8

with the nearby data point

withoutthenearbydatapoint

without thenearby datapoint

5

10

15

5

10

15

5

10

15

K

K

K

dataset: pol

dataset:2dplanes

3.0

with the nearby data point

KNN-Shapley

0.50

without the nearby data point

0.48

without thenearby datapoint

with the nearby data point

2.0

1.5

0.46

1.0

5

10

15

5

10

15

K

K

For completeness, we also plot the same figures but for the older version of KNN-Shapley from [29], which shows similar results.

dataset:fraud

dataset:creditcard

dataset: apsfail

1.3

0.5

with the nearby data point

KNN-Shapley

1.2

without the nearby data point

KNN-Shapley

KNN-Shapley

1.06

1.1

0.4

1.0

1.04

0.3

0.9

with thenearby datapoint

1.02

with the nearby data point

0.8

without the nearby data point

without the nearby data point

5

10

15

5

10

15

5

10

15

K

K

K

dataset:phoneme

dataset:click

dataset:wind

1.4

with the nearby data point

0.40

0.70

without thenearby datapoint

KNN-Shapley

KNN-Shapley

0.35

0.65

0.30

1.0

0.60

0.25

with the nearby data point

0.8

with the nearby data point

without the nearby datapoint

without the nearby datapoint

5

10

15

5

10

15

5

10

15

K

K

K

dataset:pol

dataset:2dplanes

3.0

with the nearby data point

0.50

without thenearby datapoint

KNN-Shapley

with the nearby data point

0.48

without the nearby datapoint

2.0

1.5

0.46

1.0

5

10

15

5

10

15

K

K

B.2.2 An Instantiation of Membership Inference Attack via Data Value Scores

The observation in Appendix B.2.1, demonstrates the potential for KNN-Shapley scores to leak membership information of other data points in the dataset. To further demonstrate such privacy risk, in this section we show an example of a privacy attack where an adversary could infer the presence/absence of certain data points in the dataset based on the variations in the KNN-Shapley scores, analogous to the classic membership inference attack on ML model [58]. The attack result further highlights the need for privacy protection when applying data valuation techniques.

Remark 8 (Motivation of Membership Inference Attack) . Membership inference attack, i.e., confirming one's membership in a database, could pose significant privacy risks. For example, if a medical database is known to contain data from patients with specific conditions, this could disclose members' health status. We choose MI attack as our main example of the privacy risk as it has long been recognized as a fundamental privacy challenge across various domains, and such attacks have been extended well into the field of machine learning.

Remark 9. We stress that our goal here is the proof-of-concept, where we demonstrate that the data value scores can indeed serve as another channel of privacy leakage, and can indeed lead to the design of membership inference attacks. We do not claim any optimality of the attack we construct here. Designing the best membership inference attack on data value scores can be the topic of a single paper on its own, and is an interesting future work.

Membership Inference Attack via KNN-Shapley Scores (Algorithm 1). Our membership inference attack technique leverages KNN-Shapley scores to detect the presence or absence of specific data points in a dataset. The design is analogous to the membership inference attack against ML models via the likelihood ratio test [8]. Threat model: The threat model we consider here is that the attacker can compute the value of any data point among any datasets (analogue to the setting of MIA against ML models where the attacker can train models on any datasets he/she constructs). Moreover,

Algorithm 1 - Membership Inference Attack via KNN-Shapley Scores. We collect data value scores of a copy of the target example on datasets with and without the target example, estimate mean and variance of the loss distributions, and compute a likelihood ratio test.

the attacker can craft the data point it owns, send it to the server and obtain the data value score of its own data point.

The attack goes as follows: firstly, we create a shadow dataset D shadow by randomly sampling from the data distribution D . D shadow serves as the function of D -z in the maintext. We repeat this sampling process T times, where T is a predefined number of iterations. For each iteration, we calculate the KNN-Shapley scores of a query example z = ( x, y ) both when it is included in the shadow dataset (IN) and when it is excluded (OUT). We thus collect sets of IN and OUT scores for the query example. Upon collecting all these scores, we calculate their respective mean and variance. Finally, we query the server about KNN-Shapley score of the query example z on the actual dataset D -z , which we refer to as the target data value (i.e., the server takes a copy the target data point as the data it holds, and send to the central server). We perform a likelihood ratio test using the distributions of the IN and OUT scores. This test involves comparing the probability of the observed target data value given the Normal distribution of the IN scores with the probability of the same given the Normal distribution of the OUT scores.

Intuition of the attack. The intuition of the attack is as follows: if the target data point is in the dataset, then if the attacker makes a copy of the target data point and send it to the server, the server effectively has two exactly the same data point, which will result in a lower value of each data point. On the other hand, if the target data point is a non-member, then the data value queried by the attacker will be higher.

Experiment Settings For each dataset we experiment on, we select 200 data points (members) as the private dataset held by the central server. We pick another 200 data points which serve as non-members. Moreover, we leverage another 400 data points where we can subsample 'shadow dataset' in Algorithm 1. We set the number of shadow datasets we sample as 32.

Results on KNN-Shapley and TKNN-Shapley. Table 1 in the main paper shows the AUROC score of the attack results on KNN-Shapley. For completeness, we additionally conducted the experiment of the proposed MIA against (non-private) TKNN-Shapley and the results are shown in Table 4. As we can see, our MIA attack can achieve a detection performance that is better than the random guess (0.5) for most of the settings. On some datasets, the attack performance can achieve > 0 . 7 AUROC. This demonstrates that privacy leakage in data value scores can indeed lead to non-trivial privacy attacks.

Results on DP-TKNN-Shapley. To directly demonstrate how DP-TKNN-Shapley can mitigate the privacy risk, we additionally conducted the experiment of evaluating the proposed MI attack on DP-TKNN-Shapley (see Table 5 and 6). Compared with the result on non-private TKNN-Shapley, we can see that the overall attack performance drops to around 0.5 (the performance of random guess).

The result shows that DP-TKNN-Shapley is indeed very effective against membership inference attacks.

We stress again that the point of this experiment is not claiming that the membership inference attack that we developed here is an optimal MI attack on data value scores; instead, this is a proof-of-concept for the claim that data value scores can leak private information about other data points in the dataset, and we instantiate a possible privacy attack that exploits this privacy risks. We believe it is interesting future work to improve the attack performance as well as explore other possible threat models.

B.3 Challenges in making KNN-Shapley being differentially private

In this section, we give more details of the inherent difficulties in making the KNN-Shapley ϕ KNN z i ( D -z i ) (i.e., Theorem 7) to be differentially private.

B.3.1 Large global sensitivity

We find it very challenging to tightly bound the global sensitivity of ϕ KNN z i ( D -z i ) . Moreover, we show that the global sensitivity of ϕ KNN z i ( D -z i ) can significantly exceed the magnitude of ϕ KNN z i by showing a lower bound. Specifically, we prove that the global sensitivity bound is at least around O (1) by constructing a specific pair of neighboring datasets.

Theorem 9 (Lower Bound for the global sensitivity of ϕ KNN z ( D -z ) ) . For a data point z = ( x, y ) and validation data point z (val) = ( x (val) , y (val) ) , denote the global sensitivity of ϕ KNN z ( D -z ; z (val) ) as

where C is the number of classes for the corresponding classification task.

Proof. The proof idea is to construct two neighboring datasets: D = { z 2 } , D ′ = { z 1 , z 2 } where z 1 = ( x 1 , y 1 ) , z 2 = ( x 2 , y 2 ) are two data points and we let y 1 = y (val) . Moreover, we let d ( z 1 , z (val) ) ≤ d ( z 2 , z (val) ) . From KNN-Shapley's formula in Theorem 7, we have and if K ≥ 2 , we have

, and if K = 1 , we have

Since ✶ [ y 1 = y (val) ] = 1 (our condition), we have

The above theorem tells us that the global sensitivity for KNN-Shapley is at the order of O (1) . On the other hand, we can see from the formula of KNN-Shapley in Theorem 7 that the magnitude ϕ KNN z for many of the data points z is at the order of O (1 /N ) . Hence, if we apply the Gaussian mechanism and add random noise proportional to the global sensitivity bound, the resulting privatized data value score could substantially deviate from its non-private counterpart, thereby compromising the utility of the privatized data value scores.

Tight Global Sensitivity for the older version of KNN-Shapley from [29]. As we said earlier, it is hard to bound the global sensitivity for ϕ KNN z . Moreover, even if we are able to bound the global sensitivity, the bound will highly likely be large compared with the magnitude of ϕ KNN z , as we can see from Theorem 9.

In order to find a reasonable baseline for comparing with DP-TKNN-Shapley, we consider the older version of the KNN-Shapley developed in [29], where we show that its global sensitivity can be tightly bounded (but still large).

Theorem 10 (Global sensitivity of ϕ KNN-OLD z ( D -z ) from [29]) . For a data point z = ( x, y ) and validation data point z (val) = ( x (val) , y (val) ) , denote the global sensitivity of ϕ KNN-OLD z ( D -z ; z (val) ) as ∆( ϕ KNN-OLD z ; z (val) ) := sup D -z ∼ D ′ -z ∣ ∣ ϕ KNN-OLD z ( D -z ; z (val) ) -ϕ KNN-OLD z ( D ′ -z ; z (val) ) ∣ ∣ . We have

Proof. For any dataset D = D -z ∪ { z } , we sort the data points according to the distance to x (val) , and we denote z j for the j th closest data point to z (val) . WLOG, suppose z i := z . We first write out the non-recursive expression for the older KNN-Shapley ϕ KNN-OLD z i :

If N ≤ K , then we have ϕ KNN-OLD z i = ✶ [ y i = y (val) ] K for all i (i.e., no privacy leakage in this case).

If N > K , there are two cases:

Case 1: i ≥ K , then we have

Hence, if we add/remove a data point z j = ( x j , y j ) s.t. d ( x j , x (val) ) ≥ d ( x i , x (val) ) , we have

If we add a data point z = ( x , y ) s.t. d ( x , x (val) ) < d ( x , x (val) ) , we have

When i ≥ K +1 , the sensitivity analysis for remove a data point z j = ( x j , y j ) s.t. d ( x j , x (val) ) < d ( x i , x (val) ) is similar to the analysis above where we also have

Case 2: i < K , then we have

By a similar analysis, we can also show that

The only remaining case that we haven't discussed yet is when i = K , and we remove a data point z j = ( x j , y j ) s.t. d ( x j , x (val) ) < d ( x i , x (val) ) . In this case, we have

We stress that the bound 1 K ( K +1) is tight. For example, for the case where i = K , then if we add another data point z ∗ = ( x ∗ , y ∗ ) s.t.

the change of the value ϕ KNN-OLD z i will be 1 K ( K +1) .

B.3.2 Difficulty (computational challenge) in incorporating subsampling technique

'Privacy amplification by subsampling' [4] is a technique where the subsampling of a dataset amplifies the privacy guarantees due to the reduced probability of an individual's data being included. Being able to incorporate such a technique is often important for achieving a decent privacy-utility tradeoff. However, the recursive nature of KNN-Shapley computation makes it hard to incorporate the subsampling techniques.

Specifically, recall that in practical data valuation scenarios, it is often desirable to compute the data value scores for all of z i ∈ D . To apply the subsampling technique, we first need to create a subsampled dataset and compute the KNN-Shapley for the target data point z i , i.e., we need to compute ϕ KNN z ( sample ( D -z )) . The subsampled dataset is usually constructed by sampling each data point independently with a probability q (this is usually referred to as Poisson subsampling [4]). If we view q as a constant, then the computation of ϕ KNN z ( sample ( D -z )) requires a runtime of ˜ O ( N ) for the computation of each ϕ KNN z i . This results in a final runtime of ˜ O ( N 2 ) for the computation of all ( ϕ KNN z i ) z i ∈ D , which is a significant increase in computational demand compared to the non-private KNN-Shapley. The recursive nature of KNN-Shapley computation significantly complicates the attempt of improving the computational efficiency. That is, it is not clear how to reuse the subsampled dataset to compute the KNN-Shapley score for those that are not sampled. Therefore, it seems that an ˜ O ( N 2 ) runtime is necessary if we want to incorporate the subsampling technique.

B.4 Baseline for experiments in Section 6.2.1

DP-KNN-Shapley. Given such an upper bound for the global sensitivity of ϕ KNN-OLD z , we can use Gaussian mechanism (Theorem 4) to privatize ϕ KNN-OLD z , i.e., we compute ̂ ϕ KNN-OLD z ( D -z ) := ϕ KNN-OLD z ( D -z )+ N ( 0 , 1 K ( K +1) ) . We note that such a bound is still not satisfactory as the magnitude ϕ KNN-OLD z for many of the data points z is at the order of O (1 /N ) . Nevertheless, this is a reasonable baseline (if it is not the only one) that we can use for comparison in DP-related experiments.

DP-KNN-Shapley with subsampling. Despite the high computational cost associated with the incorporation of the subsampling technique, we still compare our approach with this computationally intensive baseline for completeness. More specifically, we compute ̂ ϕ KNN-OLD z ( D -z ) := ϕ KNN-OLD z ( sample ( D -z )) + N ( 0 , ( 1 K ( K +1) ) 2 ) .

C Details for TKNN-Shapley

C.1 Consistency Result for TKNN

For a data point x and a threshold τ , we denote

the ball of radius τ centered at x . Recall that the prediction rule of TKNN when given a training set D is

Theorem 11. Suppose m is the target function that is Lipschitz on supp ( µ ) where µ is the probability measure of data distribution. As n →∞ , if τ n → 0 and nµ ( S x,τ ) →∞ for all x ∈ supp ( µ ) , then

Suppose m has Lipschitz constant L , i.e., for any pair of data points x, x ′ ∈ supp ( µ ) , we have | m ( x ) -m ( x ′ ) | ≤ L ∥ x -x ′ ∥

which → 0 as n →∞ .

Proof.

C.2 Proofs for TKNN-Shapley

Given a dataset S and a test data point ( x (val) , y (val) ) , let NB x (val) ,τ ( S ) := { ( x, y ) | ( x, y ) ∈ S, d ( x, x (val) ) ≤ τ } the nearest neighbor of x (val) that is within a pre-specified threshold τ . Recall that in this case, the utility function of soft-label TKNN classifier becomes

Semivalue. The class of data values that satisfy all the Shapley axioms except efficiency is called semivalues . It was originally studied in the field of economics and recently proposed to tackle the data valuation problem [39]. Unlike the Shapley value, semivalues are not unique. The following theorem by the seminal work of [13] shows that every semivalue of a player z (in our case the player is a data point) can be expressed as the weighted average of marginal contributions v ( S ∪{ z } ) -v ( S ) across different subsets S ⊆ N \ { z } .

Theorem 12 (Representation of Semivalue [13]) . A value function ϕ semi is a semivalue, if and only if, there exists a weight function w : [ N ] → R such that ∑ N k =1 ( N -1 k -1 ) w ( k ) = N and the value function ϕ can be expressed as follows:

Semivalues subsume both the Shapley value and the Banzhaf value with w shap ( k ) = ( N -1 k -1 ) -1 and w banz ( k ) = N 2 N -1 , respectively. In the following, we first discuss the general semivalue for soft-label TKNN classifier with a weight function w . We then plug in the specific weight function for the Shapley value.

Our goal is to derive ϕ z i ( D -z i ; v, w ) , the semivalue of z i = ( x i , y i ) with a weight function w when using soft-label TKNN classifier, i.e.,

Consider a validation data point ( x (val) , y (val) ) , and recall that v ( S ) := v ( S ; ( x (val) , y (val) )) . Denote

Case 2: if ∥ ∥ x i -x (val) ∥ ∥ ≤ τ , we have the following:

This means that the marginal contribution on S only depends on the subset NB x (val) ,τ ( S ) ⊆ S . Plugging in the above equation for v ( S ∪ z i ) -v ( S ) , we have

and

Hence

Hence,

C.2.1 The Shapley value for TKNN (TKNN-Shapley)

Theorem 13 (Full version of TKNN-Shapley) . Consider the utility function v TKNN z (val) in (3). Given a validation data point z (val) = ( x (val) , y (val) ) , the Shapley value ϕ TKNN z i ( v TKNN z (val) ) of each training point z i = ( x i , y i ) ∈ D can be calculated as follows:

and

and C is the number of classes for the classification task, and 1 /C is the random guess accuracy. Runtime: the computation of all ( ϕ TKNN z 1 , . . . , ϕ TKNN z N ) can be achieved in O ( N ) runtime in total.

The Shapley value when using full validation set: The Shapley value corresponding to the utility function v TKNN D (val) can be calculated as ϕ TKNN z i ( v TKNN D (val) ) = ∑ z (val) ∈ D (val) ϕ TKNN z i ( v TKNN z (val) ) .

Proof. For the Shapley value, we have w ( k ) = ( N -1 k -1 ) -1 .

C.2.2 Efficient Computation of TKNN-Shapley

As we can see, all of the quantities that are needed to compute TKNN-Shapley value ( c , c x (val) ,τ , c (+) z (val) ,τ ) are simply counting queries on D -z i , and hence C z (val) ( D -z i ) can be computed in O ( N ) runtime for any z i ∈ D . Since our goal is to release ϕ TKNN z i for all z i ∈ D , we need to compute C z (val) ( D -z i ) for all z i ∈ D . A more efficient way for this case is to first compute C z (val) ( D ) on the full dataset D , and then we can compute each of C z (val) ( D -z i ) by

✶ ✶ which can be computed in O (1) runtime. Hence, we can compute TKNN-Shapley ϕ TKNN z i ( v TKNN z (val) ) for all z i ∈ D within an overall computational cost of O ( N ) .

D Extra Details for DP-TKNN-Shapley

D.1 Privacy Guarantee for DP-TKNN-Shapley

Theorem 14 (Restatement of Theorem 6) . For any z (val) ∈ D (val) , releasing ̂ ϕ TKNN z i ( v TKNN z (val) ) := ϕ TKNN z i [ ̂ C z (val) ( D -z i ) ] to data owner i is ( ε, δ ) -DP with σ = √ 3 · √ log(1 . 25 /δ ) /ε .

Proof. The Gaussian mechanism is known to satisfy ( ε, δ ) -DP with σ = ∆ · √ 2 log(1 . 25 /δ ) /ε [16], where ∆ := sup D ∼ D ′ ∥ f ( D ) -f ( D ′ ) ∥ represents the global sensitivity of the underlying function f in ℓ 2 norm. In our case, f calculates the 3-dimensional vector [ c , c x (val) ,τ , c (+) z (val) ,τ ] . As each training data point may alter both c x (val) ,τ and c (+) z (val) ,τ by 1, the sensitivity ∆ is √ 3 . Since both ̂ c x (val) ,τ and ̂ c (+) z (val) ,τ are privatized using the Gaussian mechanism, the privacy guarantee of ̂ ϕ z i is ensured by the post-processing property of differential privacy.

D.2 Advantages of DP-TKNN-Shapley

D.2.1 Computational Efficiency via Reusing Privatized Statistics

Recall that in practical data valuation scenarios, it is often desirable to compute the data value scores for all of z i ∈ D . As we detailed in Section 4.2 and Appendix C.2.2, for TKNN-Shapley, such a computation only requires O ( N ) runtime in total if we first compute C z (val) ( D ) and subsequently C z (val) ( D -z i ) for each z i ∈ D . In a similar vein, we can efficiently compute the DP variant ̂ ϕ TKNN z i for all z i ∈ D . To do so, we first calculate ̂ C z (val) ( D ) and then, for each z i ∈ D , we compute ̂ C z (val) ( D -z i ) as follows:

✶ ✶ which can be computed in O (1) runtime. Hence, we can compute DP-TKNN-Shapley ̂ ϕ TKNN z i ( v TKNN z (val) ) for all z i ∈ D within an overall computational cost of O ( N ) . It is important to note that when releasing ̂ ϕ TKNN z i to individual i , the data point they hold, z i , is not private to the individual themselves. Therefore, as long as ̂ C z (val) ( D ) is privatized, ̂ C z (val) ( D -z i ) and hence ̂ ϕ TKNN z i also satisfy the same privacy guarantee due to DP's post-processing property.

D.2.2 Collusion Resistance via Reusing Privatized Statistics

In Section 5.1, we consider the single Shapley value ϕ TKNN z i ( D -z i ) as the function to privatize. However, when we consider the release of all Shapley values M ( D ) := ( ̂ ϕ TKNN z 1 ( D -z 1 ) , . . . , ̂ ϕ TKNN z N ( D -z N )) as a whole mechanism, one can show that such a mechanism satisfies joint differential privacy (JDP) [34] if we reuse the privatized statistic ̂ C z (val) ( D ) for the release of all ̂ ϕ TKNN z i . The consequence of satisfying JDP is that our mechanism is resilient against collusion among groups of individuals without any privacy degradation. That is, even if an arbitrary group of individuals in [ N ] \ i colludes (i.e., shares their respective ̂ ϕ TKNN z j values within the group), the privacy of individual i remains uncompromised. Our method also stands resilient in scenarios where a powerful adversary sends multiple data points and receives multiple value scores.

Definition 15 (Joint Differential Privacy [34]) . For ε, δ ≥ 0 , a randomized algorithm M : N X →Y N is ( ε, δ ) -joint differentially private if for every possible pair of z, z ′ ∈ X , for every i ∈ [ N ] , and for every subset of possible outputs E ⊆ Y N -1 , we have

where M -i denotes the output of M that excludes the i th dimension.

Theorem 16. The mechanism of releasing of all noisy KNN-Shapley values

satisfy ( ε, δ ) -JDP if σ = √ 3 · √ log(1 . 25 /δ ) /ε and each ϕ TKNN z i is computed via reusing the privatized C z (val) .

Proof. The proof relies on the 'post-processing' property of differential privacy.

First of all, we select σ = √ 3 · √ log(1 . 25 /δ ) /ε , ensuring that ̂ C z (val) ( D ) satisfies ( ε, δ ) -DP.

In the context of DP-TKNN-shapley, we calculate ̂ C z (val) ( D -z i ) for each z i as part of a postprocessing step on ̂ C z (val) ( D ) .

It is important to note that the released data value ̂ ϕ TKNN z i ( D -z i ) is a function of ̂ C z (val) ( D -z i ) , and ̂ C z (val) ( D -z i ) can be obtained as post-processing of ̂ C z (val) ( D ) given only the knowledge of z i . Hence, we can express the probability of M ( z i , D -z i ) -i belonging to a certain set E as follows:

where

We note that f D -z i is a function that does not depend on z i .

Then, for any z i , z ′ i and for any tuple of D -z i , denote D := z i ∪ D -z i and D ′ := z ′ i ∪ D -z i , we have

where the first inequality is due to the ( ε, δ ) -DP guarantee of ̂ C z (val) ( D ) .

Hence, the privacy guarantee for our DP-TKNN-Shapley remains the same even if multiple attackers collude and share the received noisy data value scores with each other. As a comparison, the naive DP-KNN-Shapley which independently adds noise for each released data value score does not enjoy such powerful collusion resistance property.

D.2.3 Privacy Amplification by Subsampling

In order to further boost the privacy guarantee, we incorporate the privacy amplification by subsampling technique [4]. Specifically, before any computation, we first sample each data point independently with a probability q (this is usually referred to as Poisson subsampling). The differentially private quantities based on the subsampled dataset, denoted as sample ( D ) , are calculated easily as follows:

The function round rounds a noisy value to the nearest integer. If the noisy value is out of range (e.g., < 0 ), then it is rounded to the nearest possible value within the valid range. By subsampling the data points, the privacy guarantee is amplified. This is because the subsampling process itself is a randomized operation that provides a level of privacy, making it harder to infer information about individual data points. When combined with the Gaussian mechanism, the overall privacy guarantee is improved.

The subsampling technique can be incorporated easily for DP-TKNN-Shapley as above, where the subsampling technique only improves the computational efficiency. As a comparison, incorporating subsampling technique for DP-KNN-Shapley will introduce a significantly higher computational cost, as we discussed in Appendix B.3.2.

D.3 Differentially private release of ϕ TKNN z i ( v TKNN D (val) ) (Privacy Accounting)

Recall that the TKNN-Shapley corresponding to the full validation set D (val) is ϕ TKNN z i ( v TKNN D (val) ) = ∑ z (val) ∈ D (val) ϕ TKNN z i ( D -z i ; z (val) ) . We can compute privatized ϕ TKNN z i ( v TKNN D (val) ) by simply releasing ϕ TKNN z i ( D -z i ; z (val) ) for all z (val) ∈ D (val) 3 . To better keep track of the overall privacy cost, we use the current state-of-the-art privacy accounting technique based on the notion of the Privacy Loss Random Variable (PRV) [17]. Here, we provide more details about the background of the composition of differential privacy and the privacy accounting techniques we use.

Background: Composition of Differential Privacy ('Privacy Accounting'). In practice, multiple differentially private mechanisms may be applied to the same dataset, denoted as M ( D ) = M 1 ◦ M 2 ( D ) := ( M 1 ( D ) , M 2 ( D )) . 4 Differential privacy offers strong composition guarantees, and the guarantees are derived by various composition theorems or privacy accounting techniques, including the basic composition theorem [14], advanced composition theorem [18], and Moments Accountant [1]. For example, the basic composition theorem states that if M 1 is ( ε 1 , δ 1 ) -DP and M 2 is ( ε 2 , δ 2 ) -DP, then the composition of M 1 and M 2 is ( ε 1 + ε 2 , δ 1 + δ 2 ) -DP. In our scenario, each individual mechanism is the release of ̂ ϕ TKNN z i ( D -z i ; z (val) ) for each of z (val) ∈ D (val) , and we would like to keep track of the privacy loss of releasing all of them.

Privacy Accounting Techniques based on Privacy Loss Random Variable (PRV). To better keep track of the privacy cost, we use the current state-of-the-art privacy accounting based on the notion of the Privacy Loss Random Variable (PRV) [17]. The PRV accountant was introduced in [37] and later refined in [36, 25]. For any DP algorithm, one can easily compute its ( ε, δ ) privacy guarantee based on the distribution of its PRV. The key property of PRVs is that, under (adaptive) composition, they simply add up; the PRV Y of the composition M = M 1 ◦ M 2 ◦ · · · ◦ M k is given by Y = ∑ k i =1 Y i , where Y i is the PRV of M i . Therefore, one can then find the distribution of Y by convolving the distributions of Y 1 , Y 2 , . . . , Y k . Prior works [36, 25] approximate the distribution of PRVs by truncating and discretizing them, then using the Fast Fourier Transform (FFT) to efficiently convolve the distributions. In our experiment, we use the FFT-based accountant from [25], the current state-of-the-art privacy accounting technique.

E Additional Experiment Settings & Results

E.1 Datasets

A comprehensive list of datasets and sources is summarized in Table 7. Similar to the existing data valuation literature [23, 39, 30, 60], we preprocess datasets for ease of training. For Fraud, Creditcard, and all datasets from OpenML, we subsample the dataset to balance positive and negative labels. For these datasets, if they have multi-class, we binarize the label by considering ✶ [ y = 1] . For the image dataset MNIST, CIFAR10, we apply a ResNet50 [27] that is pre-trained on the ImageNet dataset as the feature extractor. This feature extractor produces a 1024-dimensional vector for each image. We employ sentence embedding models [54] to extract features for the text classification dataset AGNews and DBPedia, and the extracted features are 1024-dimensional vectors for each text instance. We perform L2 normalization on the extracted features as part of the pre-processing step.

The size of each dataset we use is shown in Table 7. For some of the datasets, we use a subset of the full set. We stress that we are using a much larger size of the dataset compared with prior data valuation literature [30, 39, 60] (these works usually only pick a very small subset of the full dataset, e.g., 2000 data points for binarized CIFAR10). The validation data size we use is 10% of the training data size.

E.2 Settings & Additional Experiments for Runtime Experiments

For the runtime comparison experiment in Section 6.1, we follow similar experiment settings from prior study [40] and use a synthetic binary classification dataset. To generate the synthetic dataset, we sample data points from a d -dimensional standard Gaussian distribution. The labels are assigned based on the sign of the sum of the two features.

We choose a range of training data sizes N , and compare the runtime of both TKNN/KNN-Shapley at each N . We use 100 validation data points. For Figure 2 in the maintext, the data dimension d = 10 . Here, we show additional experiment results when d = 100 . As we can see from Figure 5, again TKNN-Shapley achieves better computational efficiency than KNN-Shapley across all training data sizes, and is around 30% faster than KNN-Shapley for large N .

E.3 Settings & Additional Experiments for Distinguishing Data Quality Experiments

E.3.1 Details for Mislabel/Noisy Data Detection Experiment

In the experiment of mislabeled (or noisy) data detection, we randomly choose 10% of the data points and flip their labels (or add strong noise to their features). For mislabel data detection, we flip 10% of the labels by picking an alternative label from the rest of the classes uniformly at random. For noisy data detection, we add zero-mean Gaussian noise to data features, where the standard deviation of the

Runtime Comparison (d = 100)

KNN-Shapley

seconds)

200

TKNN-Shapley

150

100

e

Runtime

50

R

0

0

50000100000150000200000

# Training Data Points

Gaussian noise added to each feature dimension is equal to the average absolute value of the feature dimension across the full dataset.

E.3.2 Additional Experiment Results for Private Setting

Table 8 provides a comprehensive comparison of DP-TKNN-Shapley and DP-KNN-Shapley. We conducted the experiments on DP-KNN-Shapley with subsampling on only half of the datasets due to the considerable computational cost (at least 30 × longer to execute compared to its non-subsampled counterpart). Additionally, because DP-KNN-Shapley injects Gaussian noise independently into the value score of each data holder, it does not have the property of being robust against collusion.

As evidenced by the results, DP-TKNN-Shapley consistently demonstrates a markedly superior privacy-utility tradeoff compared to DP-KNN-Shapley without subsampling across all datasets. Even when comparing with DP-KNN-Shapley with subsampling, DP-TKNN-Shapley still shows superior performance, and often significantly outshining this time-intensive and collusion-susceptible baseline. Notably, DP-TKNN-Shapley manages to maintain high AUROC values even when ε ≈ 0 . 1 . The low performance of DP-KNN-Shapley can be attributed to its relatively high global sensitivity, as we discussed in Appendix B.3.1.

E.3.3 Additional Experiment Results for Non-private Setting

Here, we show full experimental results for various data valuation techniques in mislabel/noisy data detection scenarios, summarized in Tables 9 and 10. In line with the main text findings, TKNNShapley exhibits performance largely on par with KNN-Shapley across the majority of datasets, demonstrating that TKNN-Shapley effectively mirrors KNN-Shapley's ability to discern data quality. In addition, both KNN-Shapley and TKNN-Shapley significantly outperform traditional retrain-based data valuation techniques (LOO, Data Shapley, and Data Banzhaf), corroborating observations made in existing studies [31, 52]. The inferior performance of Data Shapley can be attributed to sample inefficiency and the inherent randomness during the retraining process [60].

Settings for LOO, Data Shapley and Data Banzhaf. Following the experiment settings in [23, 39] we use logistic regression as the ML model for these techniques. For Data Shapley, we use permutation sampling [48] for estimation. For Data Banzhaf, we use the maximum-sample-reuse (MSR) algorithm [60] for estimation. For each dataset, we train models on 10,000 subsets (which takes more than 1000 × longer time compared with both TKNN/KNN-Shapley).

E.3.4 Ablation Study on the Choice of Threshold for TKNN-Shapley

KNN-Shapley is known for being effective on a wide range of choices of K [29]. In this experiment, we study the impact of different choices of τ on the performance of TKNN-Shapley on the tasks of mislabeled/noisy data detection. The results are shown in Table 11 and 12. As we can see, the

performance of TKNN-Shapley is quite stable on a wide range of choices of τ . For example, for 2dPlanes dataset, the performance of TKNN-Shapley only varies < 0 . 032 between τ ∈ [ -0 . 1 , -0 . 6] . Moreover, τ ≈ -0 . 5 is a relatively robust choice of the threshold value. Hence, hyperparameter tuning is not difficult for TKNN-Shapley. This is not too surprising; the formula of TKNN-Shapley (in Equation (7)) shows that regardless of the choice of τ , if a training data point is within the threshold but has a different label from the validation data, then its value is necessarily negative. At the same time, the points that share same label as validation data are guaranteed to have a positive value. The choice of τ only decides the magnitude but not the sign. As data valuation methods only rely on the rank of data values to perform data selection, the performance of TKNN-Shapley is relatively robust to the choice of τ in identifying mislabeled data points (who get negative values). A similar argument also applies to the task of noisy data detection: noisy data is typically far from the population; hence, as long as τ is not too large, the noisy data will receive a value score close to 0. On the other hand, clean data will receive a positive value score, and the choice of τ only affects the magnitude of the data value but not the sign.

Ideally, we would like to set a threshold such that for most of the queries, there are a certain amount of neighbors whose distance to the query is below the threshold. One possible way to do this is simply picking a τ that optimizes the final validation accuracy of the TKNN classifier (note that TKNN is very efficient in hyperparameter tuning).

For completeness, we also show the impact of different choices of K on the performance of KNNShapley in Table 13 and 14. As we can see, KNN-Shapley's performance is quite stable against the different choices of K .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an approach for data valuation that tries to be privacy-preserving by design. More precisely, the proposed method aims at computing the data valuation of a point through its Shapley value for the k-nearest neighbours classifier (knn) in a differentially-private manner.\n\nSTRENGTHS:\nThe paper is well-written and the authors have clearly summarized the main contributions of the paper. The background notions on Shapley values as well as knn Shapley values are also clearly explained. \n\nAn analysis of the privacy risks associated to the release of knn Shapley values has been conducted, which demonstrates the associated potential threats such as the possibility to perform a membership inference attack. Several arguments are also provided to explain why making a differentially-private version of the knn Shapley values is not a trivial task. \n\nA variant of knn Shapley, called threshold-based knn (tknn), is introduced that focuses on the neighbourhood around a point according to a specified threshold to compute the Shapley value of this point. The benefits of this approach is terms of robustness and computation efficiency are clearly highlighted and supported by the results of the experiments conducted.\n\nFinally another contribution is the design of a differentially-private variant of tknn. This approach is elegant in the sense that it exploits the structure of tknn to bound the sensitivity and thus the noise that needs to be added to achieve (\\epsilon,\\delta)-privacy. \n\nA thorough set of experiments has been performed on a variety of different datasets against various baselines corresponding to variants of knn Shapley. The results demonstrate clearly that both the non-private and differentially-private versions of tknn-Shapley outperform the knn-shapley method or offer similar utility but at a lower cost.\n\nWEAKNESSES:\nThe paper currently lacks a discussion on the limitations of the knn Shapley values compared to other methods for computing Shapley values in a more generic context. For instance, is there situations in which knn Shapley values are known not to be informative such as high-dimensional data?\n\nThe notions of collusion resistance and joint differential privacy should be defined in more depth. In particular, it would important to discuss how they differ from traditional differential privacy and whether or not this is a property that a lot of differential-privacy methods based on perturbations of counts could have.\n\nWith respect to analyzing the privacy risks associated to Shapley values, the paper is currently missing an important work from CCS last year that had conducted an in-depth study on the privacy risks related to the disclosure of Shapley values : \nLuo, X., Jiang, Y., & Xiao, X. (2022, November). Feature inference attack on shapley values. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (pp. 2233-2247).\nIt is important that the authors position their works with respect to this work. In particular, it would be great if the authors could comment on whether or not the attacks described in this paper could applied even against Shapley values computed with DP-tknn.\n\nQUESTIONS:\nSee weakness section for the main points raised.\n\nLIMITATIONS:\nOK.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-10"}

comments: SUMMARY:
The authors propose a relaxation of the KNN based Shapley valuation framework based on thresholding. By doing so, they are able to reduce the sensitivity associated with shapley valuation, and consequently apply DP to the mechanism.

STRENGTHS:
1. First formalization of TKNN based shapley valuation.

WEAKNESSES:
1. Unclear what the primary contribution is. I conjecture it is the TKNN relaxation, and its reduced run time + closed form solution.

QUESTIONS:
1. Could the authors kindly clarify how they validate the “goodness” of their TKNN method? Since the scores returned are “real numbers”, what does the AUROC measure actually capture? Similarly, what is the dependence of the threshold to determine the closeness of TKNN and KNN based shapley valuation?
2. The primary contribution of this work is demonstrating a relaxation from the KNN to the TKNN setting. The application of gaussian noise to ensure DP is a standard technique. However, the authors motivate the need for DP with an experiment that motivates how privacy is leaked through membership inference. Could the authors clarify the setting once more? The authors claim that there are multiple colluding entities that aim to identify if a particular point is present or absent in the data valuation pipeline. To this end, they need to know the data point under consideration, which seems impractical given the setup described for data valuation (e.g., where data is contributed by individual entities for medical purposes).
3. In algorithm 1, what is 	\phi_z (Dattack \cup z) formally defined as?

LIMITATIONS:
None discussed
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors suggest a modified version of KNN-Shapley that aims to enhance the privacy of the data valuation procedure. They introduce TKNN-Shapley, which focuses only on neighbors within a $\\tau$ vicinity of the utilized data point instead of a fixed number of neighbors (k), and DP-TKNN-Shapley to ensure differential privacy by incorporating the Gaussian mechanism. The paper also includes numerical experiments comparing the efficiency of TKNN-Shapley to KNN-Shapley and demonstrates the competitive performance of TKNN-Shapley in both private and non-private settings for detecting mislabeled and noisy data. The proposed approach consistently performs well across all presented results.\n\nSTRENGTHS:\nThe paper is generally clear with good coverage of related work (although I humbly listed some improvement points later in my review). \n\nI think that the main strength of the paper is the performance of the proposed method in particular in terms of data-valuation efficiency and detecting mislabeled/noisy data in the private setting. It also seems to be competitive in the non-private setting, as such thresholding operator itself for finding neighbors appears to be a safe choice. The theoretical analysis and claims also seem sound as far as my knowledge extends.\n\nWEAKNESSES:\nI have concerns regarding  a. the originality of the proposed method and b. how it concretely addresses the privacy risks. \n\na. The method is a combination of well-known techniques such as KNN-Shapley, threshold-KNN, and differential privacy, and most of the analyses are adapted from previous derivations (eg KNN-Shapley) for this very specific algorithm. As such, the theoretical contributions in the paper are mainly to be attributed to the existing result. I will gladly stand corrected by the authors if my understanding is limited. All said, as I indicated in my previous comment, its performance seems significant in the demonstrated tasks and it is valuable work. \n\nb. Despite having read the privacy risk section, I do not immediately see how the privacy risks are addressed, which is the main claim of the work. I am fairly less confident in this argument though, as I am not familiar with the privacy field. I wish to understand this aspect better.\n\nOther than that, in terms of clarity, I believe there is room for improvement:\n- I think the background is too long and the actual contribution starts on page 6. This also seems to be because the proposed method combines existing strategies, but I think the background can still rather be cut short and more room can be created for experimental results or even the related work in the appendix section. \n\nFurther notes are presented in the form of questions below.\n\nQUESTIONS:\nIt would be helpful if the authors could clarify the following confusions: \n- line 265: where does the computational complexity $\\mathcal{O}(N)$ of the TKNN-Shapley comes from? \n- Concretely what privacy risks can the proposed methods address? I was not able to understand the concrete risks (apart from line 156 which is outside the scope of the paper).\n- How does the noisy/mislabeled data detection in the numerical experiments tie to addressing privacy risks? I am not knowledgeable in the privacy field so I would appreciate the clarification even if it is an obvious explanation.\n- line 251: I am a bit confused about the raw summation. I thought it is rather averaged over the validation set\n- I think KNN-Shapley uses K in the denominator of Equation (2). Otherwise, I don't think their equation (69)* holds. It doesn't seem to affect anything with the proposed method though. Happy to be corrected.\n\n\n*https://arxiv.org/pdf/1908.08619.pdf\n\nLIMITATIONS:\nThe authors addressed several limitations in the submission. Additionally, I might add the sensitivity of DP-TKNN-Shapley's performance towards the choice of threshold $\\tau$ (though, probably not much further than what $k$ brings in the KNN-Shapley).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the privacy concerns surrounding KNN-Shapley, a prevalent method for data valuation. Authors introduce TKNN-Shapley, which employs the concept of differential privacy to protect privacy. Empirical tests demonstrate that TKNN-Shapley performs on par with KNN-Shapley in assessing data quality, while also providing a favorable balance between privacy.\n\nSTRENGTHS:\n1. The presentation is generally good.\n2. The methodology is solid. The proposed TKNN-Shapley method provides a guarantee of differential privacy, which enhances the privacy protection of the data valuation process.\n\nWEAKNESSES:\n1. The research problem is quite narrow. The paper focuses on a specific metric (KNN-Shapley) for a narrow ask (data valuation), which may not arise wide interest in the ML community. \n2. The significance of the privacy leakage of KNN-Shapley is not convincing.\n\nQUESTIONS:\nPlease see limitations\n\nLIMITATIONS:\n1. The privacy risk of KNN-Shapley is not convincingly demonstrated. The evidence presented in Figure 2 and the corresponding figures in Appendix C only suggests that modifying certain data points can lead to changes in the Shapley score, but it's unclear if these changes are unique and can lead to a membership inference attack (MIA). Additionally, KNN is not easily susceptible to overfitting, which makes MIA less likely. Furthermore, in Figure 1, the assumption of a \"trusted\" center for data upload is not representative of real-world scenarios, where untrusted centers are more common and raise additional privacy concerns.\n\n2. Data valuation is not the sole and universally accepted method for evaluating the importance of training data. Alternative dataset scoring schemes, such as core-sets or general dataset distillation approaches may be more prevalent in the field. As a recommendation, further research on these alternative methods could be beneficial.\n[1] Feldman, Dan. \"Introduction to core-sets: an updated survey.\" arXiv preprint arXiv:2011.09384 (2020).\n[2] Wang, Tongzhou, et al. \"Dataset distillation.\" arXiv preprint arXiv:1811.10959 (2018).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of evaluating and assigning a value to the quality of training data for KNN in a privacy preserving way. Therefore, it first introduces an adaptation of the KNN-Shapley algorithm, namely TKNN-Shapley which due to its non-recursive way of implementation facilitates integrating differential privacy. The paper then finally proposes how to do that and provides empirical evaluation.\n\nSTRENGTHS:\nI did really enjoy reading the paper due to its highly accessible flow, good motivation (in terms of demonstration of the privacy risks), the gentle introduction into the KNN-Shapley, the intuitive explanation why an integration of privacy is difficult, and then the solution in terms of an adapted algorithm and the integration of differential privacy.\n\nThe experimental evaluation seems sound and thorough, so does the supplementary material.\n\nThe figures (especially Figure 1) serve well the purpose of illustrating the approach described in the work.\n\nWEAKNESSES:\nMy main concern is the selection of the threshold in the TKNN. It seems crucial for the success of the algorithm, yet I could not find a good intuition and evaluation of the choice of this parameter. It would be helpful if the rebuttal could provide information about it.\nEspecially, I am curious about cases where there remain only few neighbors in the area below the threshold. \n\nAdditionally, I would enjoy reading more in the main paper around lines 310-312 about how subsampling can be efficiently integrated into the algorithm. Maybe, some intuition from Appendix E.2 could be migrated there?\n\nEven after consulting Appendix E.3, it remains unclear to me what the final privacy costs of releasing all Shapley-scores for the dataset would be and how they depend on the size of the test dataset.\n\nThe choice of the privacy parameter delta (10e-4) for, e.g. the 2DPlanes dataset seems relatively high. The rule of thumb would be to choose it 1/N with N being the number of private data points. From https://www.openml.org/search?type=data&sort=runs&id=215&status=active, I understand that the dataset has north of 40k data points, so choosing delta=10e-5 might be more appropriate.\n\nQUESTIONS:\nQ1: Can you specify the selection of the threshold in TKNN?\n\nQ2: Could you provide an explanation on the runtime of TKNN (O(N)) instead of the O(N log N)?\n\nQ3: Could you extend the privacy analysis of the release of all valuation scores of the full dataset - and show how it grows with the number of data points?\n\nQ4: Could you specify what the \"naively privatized KNN-Shapley\" baseline from line 324 refers to exactly? Without that understanding, it seems hard to follow on what are the improved privacy-utility trade-offs mentioned in the abstract.\n\nLIMITATIONS:\nThere seems to be no proper limitation section, if I haven't overlooked any. Adding a discussion on the strength of privacy protection, the ethical aspects of valuating private data, and the danger of wrong incentives for individuals should be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new variant of KNN-Shapley named TKNN-Shapley from which they derive a privacy-preserving data value approach DP-TKNN-Shapley. They conduct experiments to showcase that their approaches obtain better privacy-utility tradeoff and comparable performance to baselines.\n\nSTRENGTHS:\n- Data valuation has limited work on privacy-preserving variants and incorporating privacy in these methods (especially KNN-Shapley) is a timely problem.\n- The empirical results demonstrate the usefulness of the TKNN-Shapley and DP-TKNN-Shapley approaches.\n- As a standalone variant, TKNN-Shapley has linear time complexity which is a logarithmic factor improvement over KNN-Shapley while incurring minimal drop in performance. The non-recursive nature of the proposed solution is beneficial for privacy amplification via subsampling as well.\n\nWEAKNESSES:\n- My main issue with the work is regarding motivation and presentation. Currently, the need for a DP variant to standard data valuation approaches is motivated in Section 3. However, a number of key experiments (such as with regards to the membership inference attack) are deferred to the appendix and are even then, fairly limited. In my opinion, the need for a DP variant only exists if there is a risk for membership inference, that then leads to the research question-- _is it possible that data valuation approaches leak membership information?_ If the answer is yes, then the paper is very well-motivated and DP-TKNN-Shapley is a significant contribution _if it is resilient to such adversarial scenarios_. \n- However, even though the membership inference attack proposed provides an affirmative answer to the first research question (as the results in the appendix indicate) it is not covered in the main paper in sufficient detail. I believe the authors need to make the following changes: 1) provide a simple version of the membership inference attack (MIA) algorithm in the main paper (Section 3) and present basic attack results (Table 3 in appendix) to motivate the need for a privacy-preserving data valuation algorithm; 2) evaluate DP-TKNN-Shapley for the MIA paradigm to demonstrate its effectiveness under such adversarial influence. Here the latter is highly significant. Otherwise I feel that the problem is not well-motivated and some necessary results are missing from the work. To reiterate, without an extant threat to inferring membership, there should be no need for a private version of KNN-Shapley. \n- While the approach itself is quite simplistic, I am inclined to agree with the authors' statement that simplicity is better when private solutions need to be implemented.\n- Is there a reason why the authors have not considered the \"learning with weighted samples\" or \"point addition/removal\" application experiments for TKNN-Shapley evaluation as in previous work (refer to [1-2]) on data valuation?\n- The authors should also go through the manuscript and fix any minor formatting errors and typos-- for example, Table 10 and 11 captions in the appendix should read $K$ instead of $\\tau$.\n\nReferences:\n- [1] Kwon, Yongchan, and James Zou. \"Beta shapley: a unified and noise-reduced data valuation framework for machine learning.\" arXiv preprint arXiv:2110.14049 (2021).\n- [2] Wang, Tianhao, and Ruoxi Jia. \"Data banzhaf: A data valuation framework with maximal robustness to learning stochasticity.\" arXiv preprint arXiv:2205.15466 (2022).\n\nQUESTIONS:\nThese are provided in the Weaknesses section above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-29"}