Title: DU-Shapley: A Shapley Value Proxy for Efficient Dataset Valuation
Venue: NeurIPS 2024
Accepted: yes
Keywords: data valuation

shapley value

approximation
Abstract: We consider the dataset valuation problem, that is the problem of quantifying the incremental gain, to some relevant pre-defined utility of a machine learning task, of aggregating an individual dataset to others.
The Shapley value is a natural tool to perform dataset valuation due to its formal axiomatic justification, which can be combined with Monte Carlo integration to overcome the computational tractability challenges. Such generic approximation methods, however, remain expensive in some cases. In this paper, we exploit the knowledge about the structure of the dataset valuation problem to devise more efficient Shapley value estimators. We propose a novel approximation, referred to as discrete uniform Shapley, which is expressed as an expectation under a discrete uniform distribution with support of reasonable size. We justify the relevancy of the proposed framework via asymptotic and non-asymptotic theoretical guarantees and illustrate its benefits via an extensive set of numerical experiments.

DU-Shapley: A Shapley Value Proxy for Efficient Dataset Valuation

Abstract

We consider the dataset valuation problem , that is, the problem of quantifying the incremental gain, to some relevant pre-defined utility of a machine learning task, of aggregating an individual dataset to others. The Shapley value is a natural tool to perform dataset valuation due to its formal axiomatic justification, which can be combined with Monte Carlo integration to overcome the computational tractability challenges. Such generic approximation methods, however, remain expensive in some cases. In this paper, we exploit the knowledge about the structure of the dataset valuation problem to devise more efficient Shapley value estimators. We propose a novel approximation, referred to as discrete uniform Shapley, which is expressed as an expectation under a discrete uniform distribution with support of reasonable size. We justify the relevancy of the proposed framework via asymptotic and non-asymptotic theoretical guarantees and illustrate its benefits via an extensive set of numerical experiments.

1 Introduction

One of the main challenges for training machine learning (ML) models with enough generalization capabilities is to access a sufficiently large set of labeled training data. These data often exist but are commonly spread across many parties, impairing their usage in a direct and simple way. Real world examples range from the advertising industry, where different retailers hold sets of observations with either similar or complementary features from consented data about browsing and shopping habits of individual users; to the medical sector where hospitals may improve their diagnostics accuracy by sharing their data. By collaborating with each other and pooling their individual datasets together, these dataset owners could learn better ML models for their applications. Naturally, many questions raise from such collaborations. Federated learning [8, 9], for example, addresses the issues related to the practical ways that dataset owners can share their data. We consider a complementary problem to the one in federated learning: measuring the additional value each party would obtain by participating in the joint ML effort. In order to compute or estimate compensating rewards allowing to incentivize parties to share their data, a first stage that is commonly considered in the literature is to perform so-called dataset valuation [1, 42, 44].

Motivated by natural properties expected for fair valuation, different solution concepts from cooperative game theory [3] have been considered, the Shapley value [40] being arguably the most broadly

Benjamin Heymann*

* Equal contribution

ENSAE, FairPlay joint team

studied valuation scheme in ML due to its axiomatic justification. Agarwal et al. [1] designed a data marketplace and used the Shapley value to allocate the data among buyers. Tay et al. [44] considered a cooperative environment where agents can jointly train a generative model, from which synthetic data are drawn and distributed to the parties according to their Shapley values. Sim et al. [42] rewarded parties based on the Shapley value and information gain on model parameters. The critical challenge when using the Shapley value is its well-known computational intractability. To cope with it, [1, 44] considered Monte Carlo (MC) approximations, while [42] worked with a small set of three players. This approximation methods, however, remain expensive whenever computing the marginal contributions involve retraining. Moreover, they are generic and do not use the specific structure of the dataset valuation problem at stake, leaving open the possibility to find more adapted approximations for that problem.

The Shapley value was also used in the related problem of data valuation . Data valuation measures the contribution of a single data point within a dataset in the training of a given prediction model. Several solution concepts based on the Shapley value have been proposed for the data valuation problem including Data Shapley [10, 16], DShapley [11, 24], Beta Shapley [23] or CS-Shapley [38], together with different MC variants to cope with the computational intractability issue. For the data valuation problem, the structure was exploited to give easier-to-compute solutions in certain cases, in particular for the k -nearest neighbor problem [12, 15, 25, 26, 35, 41, 46]. Unlike data valuation, however, dataset valuation aims at quantifying the marginal contribution of a whole dataset to a given ML task with respect to (w.r.t.) the datasets brought by other dataset owners. Although data and dataset valuation are related problems, they are different and the techniques developed for data valuation cannot be used for the dataset valuation problem that we study (we further develop this point in Section 2.3).

Contributions. We consider the dataset valuation problem. Following the ML literature, we model it as a cooperative game whose value function relates to the considered ML task, and aim at estimating the Shapley value to measure the dataset owners contribution. We propose a new way to address the computational intractability issue of the Shapley value. Instead of relying on generic MC approximation schemes, our approximation method leverages the structure of the dataset valuation problem as well as a convergence result for a key random variable of the problem. Our approximation behaves well in many cases, both theoretically and empirically. More specifically, our main contributions can be summarized as follows:

We propose DU-Shapley (Definition 3), a novel Shapley value approximation that exponentially reduces the number of utility function valuations required for the computation. This is the first dataset valuation approach leveraging the specific structure of the utility function.

Based on three different use-cases, we establish asymptotic and non-asymptotic theoretical guarantees for DU-Shapley , showing notably that it converges almost surely to the Shapley value as the number of dataset owners grows.

We assess the benefits of the proposed methodology using extensive numerical experiments on both Shapley value approximation and dataset valuation use-cases. We show, in particular, that DU-Shapley outperforms all considered MC approximations of the Shapley value.

Additional Related Work. Cooperative game theory has been applied to solve multi-agents ML problems beyond data and dataset valuation [6, 18, 29, 47]. In particular, the Shapley value has been used to solve several problems including variable selection [5], feature importance [7, 27, 28], or model interpretation [4]. In these problems, similarly to the data and dataset valuation problems, the computational intractability issue of the Shapley value is usually addressed via MC [2, 31, 32].

2 Problem Formulation and Main Concepts Involved

This section presents the dataset valuation problem we aim to solve, along with preliminaries including the definition and classical approximations of the Shapley value. For n ∈ N and A , we denote [ n ] := { 1 , .., n } and U( A ) the uniform distribution with support on A .

2.1 Generic Model

We consider a collaborative ML setting involving a set I of I = |I| ∈ N ∗ dataset owners, also referred to as players in the sequel, who are willing to cooperate in order to solve a common ML problem. Each player i ∈ I is assumed to possess an individual dataset D i = { ( x ( j ) i , y ( j ) i ) } j ∈ [ n i ]

where x ( j ) i ∈ X ⊂ R d stands for a feature vector, y ( j ) i ∈ Y is a label, n i = | D i | refers to the number of data points in D i , and samples are drawn independently from a player-dependent distribution p i , i.e., ( x ( j ) i , y ( j ) i ) ∼ p i , for all j ∈ [ n i ] and i ∈ I .

Our basic motivation is to quantify the incremental contribution that a given player i ∈ I brings by sharing her dataset D i with other players towards solving some ML task. Hence, we are interested in scenarios in which, even though the data distribution might differ across players, they face a similar ML task, for instance the minimization of the expectation (with respect to p i ) of some loss function ℓ ( ˆ Y , Y ) , where ˆ Y denotes a prediction of Y . In such cases, players can usually learn from others' datasets, in the sense that given some X , the optimal prediction ˆ Y that minimizes E [ ℓ ( ˆ Y , Y ) | X ] is the same for all player. This holds, e.g. , if the conditional distributions (or, in many cases, simply the conditional expectation) of y ( j ) given x ( j ) are the same but the marginal distributions of x ( j ) differ.

To model this problem with full generality, we assume that the players i ∈ I collaborate in solving an ML task whose success is measured through some abstract metric u that maps any dataset to a real number (say, the prediction accuracy in a classification problem). With a slight abuse of notation, for any coalition of players S ⊆ I , we define u ( S ) = u (D S ) , where D S := ∪ i ∈S D i . Hence, u : 2 I → R can be seen as a game-theoretical utility function that quantifies how well coalitions of players can solve the considered ML task based on the union of their datasets.

The following subsections provide three theoretical use-cases that instantiate the generic model and give specific utility functions u to illustrate the dataset valuation problem. Using different tools and techniques, Section 3 provides theoretical guarantees in each of them. These theoretical results are then complemented in Section 4 by numerical evidence of our proposed approach in more intricate practical problems on real data.

2.1.1 Theoretical use-case 1: Non-parametric Regression

The first use case we shall investigate is quite generic and consists in non-parametric regression. We assume the existence of a function f ∗ such that y ( j ) i = f ∗ ( x ( j ) i ) + η ( j ) i with η ( j ) i i.i.d., and a quadratic loss function. Without regularity assumption on f ∗ ( · ) , learning can be arbitrarily slow; hence it is usually assumed that this mapping is Lipschitz (or at least β -Hölder [13, 45]).

The standard estimation method of f ∗ we shall consider is called the regressogram or binning (also applied in [13] to study local differential privacy within regression) and consists in learning optimal piece-wise constant functions. More precisely, given some parameter B ∈ N -chosen exogeneously as a function of the function regularity β , the ambient dimension d and the total number n of datapoints, typically B ≃ n d / ( d +2 β ) -, the feature space X is partitioned into B cubic bins. The excess risk of learning f ∗ can then be decomposed into

where ˆ f is the estimator of f ∗ , ¯ f ( x ) := ∑ b ∈ [ B ] ¯ f b ✶ { x ∈ b } , and ¯ f b is any value that f ∗ can take on the bin b . The second term in (1) being agnostic to the players' datasets, the problem of measuring the contributions of the players to estimating f ∗ can be decomposed into measuring their contributions to estimating each ¯ f b . In particular, the utility u ( S ) of a coalition S can be defined, and split into the sum of B sub-utilities u b ( S ) functions, as follows

where ˆ f S is the estimator of ¯ f when using the datasets of all players in S and ˆ f S ,b is the estimator ¯ f b when using, for all players in S , the datasets of points in the bin b . Interestingly, after this reduction, the problem is decomposed into B independent sub-problems-one per bin-, where the utility is a sole function of the number of data points used to estimate ¯ f b , i.e., we can write u b ( S ) = w b ( ∑ i ∈S n i,b ) for some function w b : N → R , where n i,b is the number of data points that player i has in the bin b . This last property motivates our second theoretical use-case.

2.1.2 Theoretical use-case 2: Homogeneous case

The second theoretical setting considers a general learning problem (not necessarily restricted to regression) and supposes that all players have the same sampling distribution, i.e., it takes p i = p

for all i ∈ I . This homogeneity on the players allows to reduce the problem of measuring the contribution of the players to just counting the number of data points contributed by each of them. Formally, and similarly to the previous use-case, we suppose the existence of a function w : N → R such that u ( S ) = w ( ∑ i ∈S n i ) .

2.1.3 Theoretical use-case 3: Heterogeneous Linear Regression - Local Differential Privacy

The third theoretical setting we consider is linear regression with random design and different variance of the features and labels per player. Although the setting is more general, one of the motivations behind it is standard linear regression with homogeneous data between players, but where players can purposely add noise when sharing their dataset (in order to provide Local Differential Privacy, for instance). Formally, for any i ∈ I , we consider the following linear model that generates the dataset D i of size n i :

with θ ∈ R d a ground-truth parameter, σ i positive and known, and ε i the differential privacy level chosen by player i . Under the linear regression framework defined in (2), and following [8], the utility function of a set S ⊆ I of players is defined by the negative expected mean square error over a hold-out dataset, i.e.,

where the expectation is taken over the distribution p test of a hold-out testing datum x ∈ R d , the sampling distributions N(0 , σ 2 i I d ) for all i ∈ S , and the linear regression error distributions N(0 , ε 2 i ) , ∀ i ∈ S , j ∈ [ n i ] , and ˆ θ S stands for the generalized least square estimator defined by ˆ θ S = ( X ⊤ S Σ -1 S X S ) -1 X ⊤ S Σ -1 S Y S , where Σ S = diag(( ε 2 i ) i ∈S ) ∈ R |S|×|S| . The notations X S and Y S refer to the concatenation of { X i } i ∈S and { Y i } i ∈S , respectively, and X i ∈ R n i × d is defined by X i = ([ x (1) i ] ⊤ , . . . , [ x ( n i ) i ] ⊤ ) ⊤ while Y i ∈ R n i is defined by Y i = ( y (1) i , . . . , y ( n i ) i ) ⊤ .

The following result provides a close-form expression for the utility function in this case:

Proposition 1. Let S be a coalition of players and consider the value function as above. It follows,

In particular, considering p test = N(0 , I d ) , we get u ( S ) = d d +1 -q ( S ) .

Proposition 1 shows that, in this use-case, the utility function can be written as a function w ( q ( S )) of a scalar quantity q ( S ) that captures the datasets heterogeneity. Notice that in this use-case, if we add the homogeneity assumption that σ i /ε i = σ/ε , for all i ∈ I , then the term q ( S ) becomes ∑ i ∈S n i and, as a consequence, we get

Recall that, in the non-parametric regression use-case, it holds u ( S ) = ∑ b ∈ [ B ] P ( x ∈ b ) w b ( q b ( S )) where q b ( S ) = ∑ i ∈S n i,b . Therefore, in our three uses-cases, the utility of a coalition can be summarized as the function of some scalar quantity of interest. This observation will be useful to state later our theoretical results.

2.2 Shapley Value

The Shapley value [40] is a classical solution concept in cooperative game theory to fairly allocate the total gains generated by a coalition of players. Given a utility function u , the Shapley value of a player i is defined as the average marginal contribution of her dataset D i to all possible subsets of { D j } j ∈I\{ i } , built by aggregating the datasets of the other players. Formally, the Shapley value φ i of player i writes

where Π( I ) refers to the set of permutations over I and P π i to the set of predecessors of player i ∈ I in permutation π ∈ Π( I ) . The Shapley value of player i is equivalently expressed as

The Shapley value has been commonly used in ML and cooperative game theory as it uniquely satisfies the following set of desirable properties.

Efficiency. ∑ I i =1 φ i ( u ) = u ( I ) , i.e, the sum of all Shapley values is equal to the value of I .

Symmetry. If, for any S ⊆ I \ { i 1 , i 2 } , u ( S ∪ { i 1 } ) = u ( S ∪ { i 2 } ) , then φ i 1 ( u ) = φ i 2 ( u ) , i.e., whenever two players have the same marginal contributions, their Shapley values coincide.

Dummy. If, for any S ⊆ I \ { i } , u ( S ∪ { i } ) = u ( S ) , then φ i ( u ) = 0 , i.e., whenever a player has null marginal contributions, her Shapley value is zero.

Linearity. φ i ( u 1 + u 2 ) = φ i ( u 1 ) + φ i ( u 2 ) , i.e., the Shapley value of sums of games is the sum of the Shapley values of the respective games.

MCapproximation of the Shapley Value. Evaluating the Shapley value is unfortunately computationally expensive in general. As a consequence, many MC approximations have been considered by sampling with replacement T terms from the sum of either (4) or (5). Regarding (4), this boils down to considering the estimator

2.3 Data valuation vs Dataset valuation

A tentative, but naive, approach to solve the dataset valuation problem could be to run an auxiliary data-valuation algorithm on all the data and to assign to each dataset the sum of the values of its datapoints. We highlight the cons of this idea on a very simple, yet insightful example. Consider two datapoints x 1 and x 2 , three datasets D 1 = { x 1 } , D 2 = { x 2 } , D 3 = { x 2 , x 2 } , and the following toy utility function u (D) = ✶ { x 1 , x 2 ∈ D } . In data valuation, any point x 2 shall have the same value, as they are identical. In particular, a naive summation would value D 3 twice the value of D 2 . In dataset valuation, and for this toy problem at hand, it is quite clear that both datasets should have the same value. Moreover, the Shapley values are 1 / 6 for D 2 and D 3 versus 2 / 3 for D 1 .

The message here is twofold. Data valuation and dataset valuation are two fundamentally different concepts and one cannot directly reduce the latter to the former. This is actually true, and this is the second message, because the utility function u is highly non-linear (even for the regression task).

3 Discrete Uniform Shapley Value

This section introduces and studies our approximation scheme for the Shapley value. Section 3.1 shows an asymptotic property that gives the general intuition behind our approximation. The result holds for the three use-cases of Sections 2.1.1 to 2.1.3. Section 3.2 presents a general approximation methodology for dataset valuation and shows its almost surely convergence as the number of players grows for our three uses-cases. Section 3.3 studies the rate of convergence, first for the homogeneous setting (Section 2.1.2), and then leverages this result to obtain a similar one for the non-parametric regression setting (Section 2.1.1). All proofs are postponed to the supplementary material.

3.1 Insights behind DU-Shapley

The Shapley value, by re-arranging the coalitions S ⊆ I \ { i } by their cardinality in the sum in (5), can be equivalently expressed as

where 2 I\{ i } K denotes the subsets of I \ { i } of cardinality K . In our three uses-cases, it follows that

where w : R + → R is such that u ( S ) = w ( q ( S )) for any S ⊆ I , and q ( S ) is the scalar quantity of interest identified in Sections 2.1.1 to 2.1.3 for each use-case:

and for the first use-case, q b ( S ) is analogously defined at every bin, with γ b i = 1 for all players and all bins. We remark that the definition of q ( S ) in the first and second use-cases is not restricted to linear regression. Equation (8) explicitly reveals a key random variable, namely q ( S ) . Interestingly, Figure 1 suggests that q ( S ) converges in distribution to a uniform random variable as the number of players increases (with i.i.d. datasets sizes). Theorem 2 proves this result formally for any ( γ i ) i ∈I .

Theorem 2. Let { n i , γ i } i ∈ [ I ] be two sequences of positive numbers such that the following limits

all exist, for some constants µ A , µ B , σ A , σ B > 0 . Let K ∼ U( { 0 , . . . , I } ) , S K ∼ U([2 I K ]) . Then, almost surely, q ( S K ) q ( I ) I →∞ -- - → U([0 , 1]) .

0.12

0.12

0.12

q(S)/q(I)

q(S)/q(I)

q(S)/q(Z)

F 0T°0

U~U(0,1)

F 0I°0

U ~U(0,1)

- 0I°0

U~U(0,1)

0.08

0.08

0.08

- 90°0

0.06

0.06

0.04

F 10°0

F 10°0

0.02 -

0.02

0.02

0.00

0.00

0.00

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

3.2 Discrete Uniform Shapley value

The Shapley value re-arrangement in (7) exposes the main tool behind our approximation: it is enough to approximate the distribution of the random variable D S that takes values on the subsets of D -i := ∪ j ∈I\{ i } D j (recall that u ( S ) = u (D S ) ). Theorem 2, taking the example of the second use-case for intuition, indicates that these datasets have uniformly distributed numbers of points in the limit. Generalizing this intuition, we propose to approximate D S by taking I samples of increasing size from the pool D -i by sampling data points uniformly. This leads to the following definition of DU-Shapley for our generic model:

Definition 3. [ DU-Shapley ] For any i ∈ I , the discrete uniform Shapley value ( DU-Shapley ) of the i -th player, denoted by ψ i , is given by

where D ( k ) is a set of data points uniformly sampled without replacement from D -i of size kµ -i , with µ -i = 1 ( I -1) | D -i | .

Compared to the Shapley value defined in (5), which involves 2 I terms to compute, note that DU-Shapley only involves I terms and hence it presents an exponential reduction of the number of utility function evaluations. Of course, these computational savings come at the cost of some bias. The latter is precisely quantified in Section 3.3 for our first two use-cases.

By definition, DU-Shapley is a random variable which depends on the sampled data points. However, whenever u ( S ) = w ( q ( S )) , with q ( S ) some scalar quantify of interest, as in our use-cases, we

can get rid of the stochastic nature of DU-Shapley by considering I real values from well chosen intervals. In particular, in our uses-cases, DU-Shapley boils down to:

where

We remark the notation abuse as we should write ψ ( w ◦ q ) . For simplicity, we omit the composition and only write ψ ( w ) . Equation (10) coincides exactly with Definition 3 in the first two use-cases, i.e., when γ j = γ for all j ∈ I . Indeed, as the random datasets D ( k ) have a fixed size and the value function only looks at the number of data points within the coalition, we obtain,

For the third use-case, Equation (10) is an approximation that comes from assuming that, for any j ∈ I \ { i } , | D j ∩ D ( k ) | = k · n j I -1 , which holds with high probability for large values of I , since

Theorem 2 implies the following result.

Corollary 4. Let φ i and ψ i be, respectively, the Shapley value (5) and the DU-Shapley (10) of player i . Then, in our three uses-cases, it holds, lim I →∞ | φ i -ψ i | = 0 almost surely.

While our theoretical results are based on Equation (10) for the cases where u ( S ) = w ( q ( S )) , we will see through numerical experiments that Definition 3 gives good results in more general cases.

3.3 Non-Asymptotic Theoretical Guarantees

Corollary 4 states asymptotic guarantees for DU-Shapley . In this section, we show non-asymptotic results that give the convergence rate for the first two uses-cases. 1 Recall that in non-parametric estimation, the utility writes as u ( S ) = ∑ b ∈ [ B ] u b ( S ) P ( x ∈ b ) , and therefore, by the linearity axiom of the Shapley value, for any i ∈ I , φ i ( u ) = ∑ b ∈ [ B ] φ i ( u b ) P ( x ∈ b ) . As a consequence, in order to estimate φ i ( u ) , it is enough to compute each φ i ( u b ) . In particular, the Shapley value approximation error over the whole feature space becomes a simple aggregation of the Shapley value approximation errors over the bins. We focus firstly on bounding the bias of our method in the homogeneous use-case to then extend it to the non-parametric regression case.

As in the homogeneous use-case the utility function writes as u ( S ) = w ( ∑ i ∈I n i ) , we consider the following regularity assumptions on w .

H1. The function w : R + → R is increasing, twice continuously differentiable, and such that lim n →∞ n 2 | w (2) ( n ) | < ∞ (where w (2) represents the second derivative).

Monotonicity is a natural assumption in our framework as, the more data, the more precise the ML prediction is expected to be. The condition over the limit aims at controlling the growth behavior of the utility function and it is automatically satisfied whenever w is bounded and w (2) is monotone, by the mean value theorem. Theorem 5 bounds the bias of DU-Shapley for the homogeneous use-case.

Theorem 5. Under Assumption H 1, there exists a constant κ > 0 , such that, for any i ∈ I , it holds,

where φ i and ψ i are respectively the Shapley value and the DU-Shapley of player i , µ -i = 1 ( I -1) | D -i | is the average dataset size of all players but i , σ 2 -i = 1 I -1 ∑ j ∈I\{ i } ( n j -µ -i ) 2 their empirical variance, and ζ -i measures the variability of the dataset sizes across players. Formally, it is defined as ζ -i := R 2 -i τ 2 -i / 4 n max -i where R -i := max j ∈I\{ i } | n j -µ -i | , n max -i := max j ∈I\{ i } n j , and τ -i := n max -i / min j ∈I\{ i } n j .

The full proof of Theorem 5 is included in Appendix C.3 and it relies on controlling the absolute value of E [ w ( µ -i K ) -w ( ∑ j ∈S n j )] , where K ∼ U( { 0 , ..., I -1 } ) and S is the random variable in (7). Using a second order Taylor expansion, the problem is reduced to controlling the term related to the second derivative of w by using the regularity assumptions in H 1.

As advertised before, Theorem 5 can be directly generalized to the non-parametric use-case, since,

Corollary 6. Under Assumption H 1 for all functions w b , there exist constants κ b > 0 , such that, for any i ∈ I , it holds that

where φ i and ψ i are respectively the Shapley value and the DU-Shapley of player i , and all terms are equivalently defined to Theorem 5 at each bin b ∈ [ B ] .

The upper bound in (11) depends on natural quantities related to the dataset valuation problem described in Section 2.1 at each bin, such as the first two moments µ -i,b and σ -i,b of the datasets' size distribution. More precisely, the error increases when there are some outlier players with a very small or large dataset size. This behavior is expected since, in this particular setting, the random variable inside of the Shapley value differs from a uniform random variable. As showcased in Theorem 2, the error vanishes when the number of players I tends towards infinity.

4 Numerical Experiments

We illustrate the benefits of DU-Shapley by measuring numerically three properties: (1) how well DU-Shapley approximates the Shapley value in real data, (2) how many (theoretical) iterations need other methods to achieve the same accuracy level than DU-Shapley , and (3) how well DU-Shapley performs in classical dataset valuation tasks with real data. Appendix A.1 complements the results by a complexity comparison between our method and SVARM [22] and Appendix A.2 by experiments on synthetic data. The experiments strongly suggest that DU-Shapley performs well in all tasks.

4.1 Approximating the Shapley Value in Real-World Data

We consider the real-world datasets in Mitchell et al. [32], whose details are provided in Table 3 in the appendix. To tackle these problems we consider logistic regression models and gradient-boosted decision trees (GBDT). For classification tasks, the utility function has been taken as the expected accuracy of the trained logistic regression model over a hold-out testing set while for regression tasks, the utility function corresponds to the averaged MSE over a hold-out testing set. In both cases we took a hold-out testing set with 10% of the size of the training dataset. For each dataset, we considered two worst-case scenarios for our method, namely I = 10 players and I = 20 players.

Starting from the datasets in Table 3, we heterogeneously allocate datasets to the players. We compare ourselves with two approaches, referred to as MC-Shapley , for the standard MC approximation defined in (6), and MC-anti-Shapley that considers, in addition, antithetic sampling [32]. We compute the averaged MSE across all players between the true Shapley value and each estimator.

Since computing the marginal contributions in this experiment requires re-training, which is clearly not feasible for a large number of epochs, we chose to restrict ourselves to 20 steps of stochastic gradient descent for logistic regression and 20 boosting iterations for GBDTs. For MC-based approaches, we considered I samples to compare those approximations with the proposed methodology on a fair basis, i.e., associated to the same computational budget.

Table 1 depicts the results. We clearly see that, even in the worst-case scenario where the number of players is small and far from the theoretical assumptions from Section 3.3, DU-Shapley competes favorably with the MC-based methods.

4.2 Complexity of Computing the Shapley Values of all Players

We have looked at the number of iterations that DataShapley and the Improved Group Testing-Based method [46] ( IGTB ) require to achieve DU-Shapley 's accumulated bias, formally given by

To do so, we have replaced ε = DUbias( I ) , respectively, in the formula in Section 4.1 in [16] and Equation 5 in [46], with a value function motivated from our third use-case under the homogeneity assumption σ i /ε i = σ/ε for all i ∈ I . The results are illustrated in Figure 2. Remark DU-Shapley requires I 2 iterations to compute all Shapley values. We observe that in all tested instances, both methods require a higher number of iterations to achieve the same error than DU-Shapley.

1011

1011

1

1

IGTB

DataShapley

IGTB

DataShapley

1010

1

1

DataShapley

IGTB

DU-Shapley

109

DU-Shapley

109

1

DU-Shapley

Iterations

109

108

107

107

107

106

105

10°

105

104

102

103

104

102

103

104

102

103

104

Number of Players

Number of Players

Number of Players

Iterations

1011

109

1

IGTB

DataShapley

DU-Shapley

1010

109

108

IGTB

DataShapley

DU-Shapley

1010

109

108

1

IGTB

DataShapley

DU-Shapley

107

107

107

106

106

10

105

105

104

104

102

103

104

102

201

104

102

103

104

Number of Players

Number of Players

Number of Players

4.3 Applying DU-Shapley to dataset valuation problems

We considered non-tabular datasets used in [17], namely bbc-embedding, IMDB-embedding, both text datasets, and CIFAR10-embedding, an image dataset. Feature embedding have been generated

using pretrained DistilBERT and ResNet50 models, respectively. In addition we have adapted three baselines from data valuation to our setting: Leave-One-Out (LOO), DataShapley, and KNN-Shapley. Appendix B.2 gives the implementations details. For these datasets associated to classification problems, we used a multi-layer perceptron classifier as prediction model.

We have considered three dataset valuation problems, none of them needing the real Shapley values, which allows us to increase the number of players w.r.t. the experiments in Section 4.1. We investigated noisy label detection (NLD), dataset removal (DR), and dataset addition (DA) [17]. For NLD, we used as a metric the F1-score (the larger the better). For DR, we used the testing accuracy (the lesser the better). For DA, we used the testing accuracy (the lesser the better).

We considered splitting the dataset across I = 100 players. The results are summarized in Table 2. We observe that DU-Shapley has competitive results compared to classical baselines despite of the fact that none of the considered cases verifies the structural assumptions from Section 3.3. In addition, we can see that DU-Shapley tends to have similar and even better results as Data Shapley (which is a MC based method). This is in line with our theory as, for larger number of players, DU-Shapley tends to better estimate the true Shapley value.

5 Conclusion

We model the dataset valuation problem as a cooperative game and design a Shapley value approximation, named DU-Shapley , that exploits the underlying structure of the utility function and exponentially reduces the number of functions valuations required for the computation. In three different uses-cases, DU-Shapley is proved to almost surely converge to the real Shapley value as the number of players grows. Moreover, we find the rate of convergence, which depends only on natural parameters of dataset valuation. Numerical experiments showcase that DU-Shapley performs well in approximating the Shapley value and performing dataset valuation tasks, even when the assumptions needed for the theoretical guarantees do not hold, and it has a good complexity when computing the Shapley values of all players.

Limitations of our method . Our non-asymptotic bound for the non-parametric regression setting in Corollary 6 indicates that DU-Shapley works better when agents' datasets are regular in the sense that they have similar sizes. Hence, a limitation of our approximation is that it may work poorly in settings where some players have large datasets compared to others, as the distribution of the random variable within the Shapley value drives apart from being uniform. Moreover, our convergence result in Theorem 2 (for all use-cases) assume the existence of limits, which roughly requires that heterogeneity between players-in terms of both dataset size and variance-can be bounded. This also indicates that convergence may be not be guaranteed if the heterogeneity is arbitrarily high.

Acknowledgments

This research was supported in part by the French National Research Agency (ANR) in the framework of the PEPR IA FOUNDRY project (ANR-23-PEIA-0003) and through the grant DOOM ANR23-CE23-0002. It was also funded by the European Union (ERC, Ocean, 101071601). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.

References

Anish Agarwal, Munther Dahleh, and Tuhin Sarkar. A Marketplace for Data: An Algorithmic Solution. In Proceedings of the ACM Conference on Economics and Computation , page 701-726, 2019.

Javier Castro, Daniel Gómez, and Juan Tejada. Polynomial calculation of the Shapley value based on sampling. Computers & Operations Research , 36(5):1726-1730, 2009.

Georgios Chalkiadakis, Edith Elkind, and Michael J. Wooldridge. Computational Aspects of Cooperative Game Theory . Morgan & Claypool Publishers, 2011.

Jianbo Chen, Le Song, Martin J. Wainwright, and Michael I. Jordan. L-Shapley and C-Shapley: Efficient Model Interpretation for Structured Data. In International Conference on Learning Representations , 2019.

Shay Cohen, Eytan Ruppin, and Gideon Dror. Feature Selection Based on the Shapley Value. In International Joint Conference on Artificial Intelligence , page 665-670, 2005.

Mingshu Cong, Han Yu, Xi Weng, and Siu Ming Yiu. A game-theoretic framework for incentive mechanism design in federated learning. Federated Learning: Privacy and Incentive , pages 205-222, 2020.

Ian C. Covert, Scott Lundberg, and Su-In Lee. Understanding Global Feature Contributions with Additive Importance Measures. In Advances in Neural Information Processing Systems , 2020.

Kate Donahue and Jon Kleinberg. Model-sharing games: Analyzing federated learning under voluntary participation. Proceedings of the AAAI Conference on Artificial Intelligence , 35(6): 5303-5311, May 2021. URL https://ojs.aaai.org/index.php/AAAI/article/view/ 16669 .

Kate Donahue and Jon Kleinberg. Optimality and Stability in Federated Learning: A Gametheoretic Approach. In Advances in Neural Information Processing Systems , volume 34, 2021.

Amirata Ghorbani and James Zou. Data Shapley: Equitable Valuation of Data for Machine Learning. In International Conference on Machine Learning , 2019.

Amirata Ghorbani, Michael Kim, and James Zou. A Distributional Framework For Data Valuation. In International Conference on Machine Learning , 2020.

Amirata Ghorbani, James Zou, and Andre Esteva. Data shapley valuation for efficient batch active learning. In 2022 56th Asilomar Conference on Signals, Systems, and Computers , pages 1456-1462. IEEE, 2022.

László Györfi and Martin Kroll. On rate optimal private regression under local differential privacy. arXiv preprint arXiv:2206.00114 , 2022.

Wassily Hoeffding. Probability Inequalities for Sums of Bounded Random Variables. Journal of the American Statistical Association , 58(301):13-30, 1963.

Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas Spanos, and Dawn Song. Efficient Task-Specific Data Valuation for Nearest Neighbor Algorithms. Proc. VLDB Endow. , 12(11):1610-1623, 2019.

Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J. Spanos. Towards Efficient Data Valuation Based on the Shapley Value. In International Conference on Artificial Intelligence and Statistics , 2019.

Kevin Jiang, Weixin Liang, James Y Zou, and Yongchan Kwon. Opendataval: a unified benchmark for data valuation. Advances in Neural Information Processing Systems , 36, 2023.

Jiawen Kang, Zehui Xiong, Dusit Niyato, Shengli Xie, and Junshan Zhang. Incentive mechanism for reliable federated learning: A joint optimization approach to combining reputation and contract theory. IEEE Internet of Things Journal , 6(6):10700-10714, 2019.

R. Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters , 33(3):291-297, 1997.

Andre I Khuri, Thomas Mathew, and Daan G Nel. A test to determine closeness of multivariate satterthwaite's approximation. Journal of Multivariate Analysis , 51(1):201-209, 1994.

Ron Kohavi. Scaling up the Accuracy of Naive-Bayes Classifiers: A Decision-Tree Hybrid. In International Conference on Knowledge Discovery and Data Mining , 1996.

Patrick Kolpaczki, Viktor Bengs, Maximilian Muschalik, and Eyke Hüllermeier. Approximating the Shapley value without marginal contributions. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 13246-13255, 2024.

Yongchan Kwon and James Zou. Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learning . In International Conference on Artificial Intelligence and Statistics , pages 8780-8802, 2022.

Yongchan Kwon, Manuel A. Rivas, and James Zou. Efficient Computation and Analysis of Distributional Shapley Values . In International Conference on Artificial Intelligence and Statistics , pages 793-801, 2021.

Weixin Liang, James Zou, and Zhou Yu. Beyond user self-reported likert scale ratings: A comparison model for automatic dialog evaluation. arXiv preprint arXiv:2005.10716 , 2020.

Weixin Liang, Kai-Hui Liang, and Zhou Yu. Herald: an annotation efficient method to detect user disengagement in social conversations. arXiv preprint arXiv:2106.00162 , 2021.

Scott M. Lundberg and Su-In Lee. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems , page 4768-4777, 2017.

Scott M. Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M. Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. From local explanations to global understanding with explainable AI for trees. Nature Machine Intelligence , 2(1):56-67, 2020.

Lingjuan Lyu, Xinyi Xu, Qian Wang, and Han Yu. Collaborative fairness in federated learning. Federated Learning: Privacy and Incentive , pages 189-204, 2020.

Olvi L. Mangasarian, W. Nick Street, and William H. Wolberg. Breast Cancer Diagnosis and Prognosis Via Linear Programming. Operations Research , 43(4):570-577, 1995.

Irwin Mann and Lloyd S. Shapley. Values of Large Games, IV: Evaluating the Electoral College by Montecarlo Techniques . RAND Corporation, Santa Monica, CA, 1960.

Rory Mitchell, Joshua Cooper, Eibe Frank, and Geoffrey Holmes. Sampling Permutations for Shapley Value Estimation. Journal of Machine Learning Research , 23(43):1-46, 2022.

Sérgio Moro, Paulo Cortez, and Paulo Rita. A data-driven approach to predict the success of bank telemarketing. Decision Support Systems , 62:22-31, 2014.

Guillermo Owen. Multilinear Extensions of Games. Management Science , 18(5):64-79, 1972.

Konstantin D Pandl, Fabian Feiland, Scott Thiebes, and Ali Sunyaev. Trustworthy machine learning for health care: scalable data valuation with the shapley value. In Proceedings of the Conference on Health, Inference, and Learning , pages 47-57, 2021.

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research , 12(85):2825-2830, 2011.

Gabriel Fernando Pivaro, Santosh Kumar, Gustavo Fraidenraich, and Claudio Ferreira Dias. On the exact and approximate eigenvalue distribution for sum of wishart matrices. IEEE Transactions on Vehicular Technology , 66(11):10537-10541, 2017.

Stephanie Schoch, Haifeng Xu, and Yangfeng Ji. CS-shapley: Class-wise shapley values for data valuation in classification. In Advances in Neural Information Processing Systems , 2022.

Robert J Serfling. Probability inequalities for the sum in sampling without replacement. The Annals of Statistics , pages 39-48, 1974.

Lloyd S. Shapley. A Value for N-Person Games . RAND Corporation, Santa Monica, CA, 1952.

Dongsub Shim, Zheda Mai, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang. Online class-incremental continual learning with adversarial shapley value. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 9630-9638, 2021.

Rachael Hwee Ling Sim, Yehong Zhang, Mun Choon Chan, and Bryan Kian Hsiang Low. Collaborative machine learning with incentive-aware model rewards. In Proceedings of the 37th International Conference on Machine Learning , ICML'20. JMLR.org, 2020.

WYTan and RP Gupta. On approximating a linear combination of central wishart matrices with positive coefficients. Communications in Statistics-Theory and Methods , 12(22):2589-2600, 1983.

Sebastian Shenghong Tay, Xinyi Xu, Chuan-Sheng Foo, and Bryan Kian Hsiang Low. Incentivizing collaboration in machine learning via synthetic data rewards. In AAAI Conference on Artificial Intelligence , 2021.

Alexandre B. Tsybakov. Introduction to Nonparametric Estimation . Springer Publishing Company, Incorporated, 1st edition, 2008. ISBN 0387790519.

Jiachen T Wang, Yuqing Zhu, Yu-Xiang Wang, Ruoxi Jia, and Prateek Mittal. Threshold knn-shapley: A linear-time and privacy-friendly approach to data valuation. arXiv preprint arXiv:2308.15709 , 2023.

Han Yu, Zelei Liu, Yang Liu, Tianjian Chen, Mingshu Cong, Xi Weng, Dusit Niyato, and Qiang Yang. A fairness-aware incentive scheme for federated learning. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , pages 393-399, 2020.

A Complementary Numerical results

All experiments were executed on a laptop running macOS 13.3.1 and equipped with Apple M1 chip with 16GB of RAM. The minimum amount of compute was roughly 5 minutes while the maximum one roughly 10 hours.

A.1 DU-Shapley vs SVARM

We have looked at the probability at which SVARM (Theorem 4 [22]) can ensure, after I 2 iterations (without considering the warm up as part of the budget), an error equal to DU-Shapley's accumulated bias. We have considered the same value function than in Section 4.2 with n max ∈ { 2 · 10 3 , 3 · 10 3 , 5 · 10 3 , 10 4 } and 100 simulations of sets of players at each time. Figure 3 shows the results. We observe how SVARM cannot ensure, with high enough probability, an approximation error equal to the one of DU-Shapley .

n_max = 2000

n_max =3000

1.2

1.0

1.0

8'0

8'0

0.6

0.6

Probability

0.4

0.4

0.2

0.2

0.0

0.0

0.2

0.2

102

103

102

103

Number of Players

Number of Players

n_max = 5000

n_max= 10000

0.8

8'0

0.6

0.6

0.4

0

0.2

0.2

0.0

0.0

0.2

0.2

102

103

102

103

Number of Players

Number of Players

A.2 Approximating the Shapley value in Synthetic Data

We consider a toy dataset valuation problem associated to our heterogeneous linear regression with local differential privacy use-case (Section 2.1.3) and we measure the value of a coalition S with the utility function in close-form from Proposition 1. We consider d = 10 .

In order to benchmark the performances of DU-Shapley , we consider four competitive approaches, relying on Monte Carlo (MC) approximation strategies [32]. The first one, referred to as MC-Shapley is the standard MC approximation defined in (6). The second one, coined MC-anti-Shapley is a variance-reduced version of MC-Shapley that considers antithetic sampling. The third one coined Owen-Shapley stands for the multilinear extension of Owen [34] which represents the Shapley value as two nested expectations (further explained in Appendix B.3). Finally, the fourth approach, coined Orthogonal-Shapley , relies on efficient permutation sampling techniques on the hypersphere to draw permutations in (4) in a dependent way. To assess the performance of the aforementioned Shapley value estimators, we used the mean square error (MSE) averaged over all players. DU-Shapley is computed exactly by using (10) while, for each MC-based estimator, we performed 25 Shapley value estimations to compute the MSE, and did it 10 times to obtain confidence intervals for the MSE.

Figure 4 compares DU-Shapley (the horizontal line which does not depend on the sampling budget as we compute it exactly) and the MC-based methods, which are computed at several different budgets. The x-axis corresponds to the sampling budget allowed to the MC-bases methods w.r.t. DU-Shapley , i.e., 10 -1 means a budget equal to 10 %the one of DU-Shapley, 10 0 means same budget (indicated by the black vertical line), and 10 1 means 10 times the DU-Shapley budget. Remark that, even when the MC-methods use 10 times the budget of DU-Shapley , our method keeps approximating better the Shapley value.

DU-Shapley

DU-Shapley

10-1

MC-Shapley

MC-Shapley

MC-Anti-Shapley

10-2

MC-Anti-Shapley

Owen-Shapley

Owen-Shapley

Ortho-Shapley

Ortho-Shapley

10-3

E10-2

MSE

10-4

10-3

10-5

10-1

100

101

10-1

100

101

Sampling budget w.r.t.DU-Shapley

Sampling budget w.r.t.DU-Shapley

DU-Shapley

DU-Shapley

10-1

MC-Anti-Shapley

MC-Shapley

10-2

MC-Anti-Shapley

MC-Shapley

Owen-Shapley

Owen-Shapley

Ortho-Shapley

Ortho-Shapley

10-3

MSE

10-2

MSE

10-4

10-3

10-5

10-6

10-1

100

101

10-1

100

101

Sampling budget w.r.t. DU-Shapley

Sampling budget w.r.t. DU-Shapley

B Further details about numerical implementations

B.1 Datasets considered in Section 4.1.

Table 3 summarizes the real-world datasets considered in Section 4.1.

B.2 OpenDataVal implementations

In this section we describe more in detail the implementations of DataShapley, Leave-One-Out (LOO), and KNN-Shapley for our numerical results in Section 4.3.

DataShapley, applied to the dataset valuation problem, simply corresponds to the method coined MC in Section 4.1. Therefore, we sample datasets and output the averaged marginal contribution.

Regarding LOO, notice that it corresponds to compute just one marginal contribution, usually computed on the big coalition, i.e.,

As players' marginal contributions to large datasets tend to be small, we have preferred to sample one dataset D from D -i and to output

Finally, regarding KNN-Shapley, we refer the reader to [15], Section E.3 of the appendix who explain how to adapt the method to dataset valuation.

B.3 Owen's Shapley value approximation

In Section A.2, we considered the Shapley value approximation referred to as Owen-Shapley as a state-of-the-art competitor to DU-Shapley . We provide in the following additional details regarding Owen-Shapley . For the other competitors, we directly refer the interested reader to Mitchell et al. [32]. Owen [34] studied the multilinear extension of a cooperative game and an alternative way to express the Shapley value. Formally, a cooperative game G = ( I , u ) consists on a set of I players I = { 1 , 2 , ..., I } and a value function u : 2 I → R such that, for any S ⊆ I , u ( S ) corresponds to the value generated by the coalition S . The multilinear extension of G , denoted ¯ G = ( I , ¯ u ) , is obtained when considering the value function ¯ u : [0 , 1] I → R given by,

Intuitively, ¯ u ( x 1 , x 2 , ..., x I ) corresponds to the expected value of a coalition when each player i ∈ I joins the coalition with probability x i . Theorem 5 in [34] gives an alternative way to compute the Shapley value φ i ( u ) of player i in game G , namely,

where E i ( τ ) is a random subset of I \ { i } , such that, ∀ j ∈ I \ { i } , j is included in E i ( τ ) with probability τ . In words, the Shapley value of player i corresponds to her expected marginal contribution to the random set E i ( τ ) , when τ is uniformly distributed on [0 , 1] . This brings an alternative way to use Monte Carlo to approximate the Shapley value φ i ( u ) , coined Owen-Shapley, as,

where for each t ∈ { 1 , ..., T } , we draw τ t independently and uniformly in [0 , 1] and then, create a random set E i ( τ t ) by adding each player j ∈ I \ { i } to it with probability τ t .

C Missing proofs

C.1 Proof of Proposition 1

Proposition 1 . Let S ⊆ I be a coalition of players and consider the value function u as in (3) . It follows,

In particular, considering p test = N(0 , I d ) , we get,

Proof. Let S ⊆ I be a coalition of players and X S , Y S be the concatenation of their datasets. The linear model can be rewritten in matrix form as

where η S is the concatenation of η ( j ) i for all i ∈ S and j ∈ [ n i ] . Take ˆ θ S = ( X ⊤ S Σ -1 S X S ) -1 X ⊤ S Σ -1 S Y S where Σ S = diag(( ε 2 i ) i ∈S ) , and let x ∼ p test be a hold-out testing datum in R d . It follows,

We take expectation with respect to the different stochastic terms. Since η ( k ) i ∼ N(0 , ε 2 i ) for any i ∈ S , k ∈ [ n i ] , it holds,

Since players distributions differ on their variances, ∑ i ∈S X ⊤ i ε -2 i X i corresponds to a semicorrelated Wishart random variable where each 1 ε i X i ∼ N(0 , ( σ i ε i ) 2 I d ) . In particular, the semicorrelated Wishart random variable can be approximated by a central Wishart distribution [37, 43], whose precision depends on the homogeneity of the coefficients σ i /ε i over all i ∈ I , as showed in [20]. It follows,

where

With all this in mind, it follows,

In particular, considering p test = N(0 , I d ) , we get,

C.2 Proof of Theorem 2

Theorem 2. Let { n i , γ i } i ∈ [ I ] be two sequences of positive numbers such that the following limits

all exist, for some constants µ A , µ B , σ A , σ B > 0 . Let K ∼ U( { 0 , . . . , I } ) , S K ∼ U([2 I K ]) , and define q ( S K ) as in (9) for the third use-case. Then, almost surely, q ( S K ) / q ( I ) I →∞ -- - → U([0 , 1]) .

Proof. Introduce, for any t, t 0 ∈ (0 , 1) and any s > 0 ,

By construction, Y A ( t, I ) and Y B ( t, I ) are sums of sampling without replacement of ⌊ It ⌋ elements. Therefore, by Corollary 1.3 in [39], for s fixed, there exists I A 0 , I B 0 ∈ N such that,

In other words, for any s > 0 and I large enough, almost surely, it holds,

It follows,

which is arbitrarily small as µ A ( I ) , µ B ( I ) → µ A , µ B < ∞ . Therefore, almost surely,

The proof concludes noticing that

and that K = ⌊ IU ⌋ with U ∼ U([0 , 1]) .

C.3 Proof of Theorem 5

To prove Theorem 5, we need two preliminary results: Lemma 3, which itself needs two supplementary results (Lemmas 1 and 2), and Lemma 4, which is directly proved.

C.3.1 Technical lemmata

Lemma 1. Consider a set of I values N = { n 1 , . . . , n I } . Let X 1 , . . . , X k and Y 1 , . . . , Y k denote, respectively, k random samples with and without replacement from N . For any continuous and convex function f , it follows,

Proof. The proof follows from [14].

Lemma 2. Let I ∈ N , N := { n 1 , . . . , n I } ∈ R I + , µ = 1 I ∑ I i =1 n i be their mean value and σ 2 = 1 I ∑ I i =1 ( n i -µ ) 2 be their variance. For k ∈ { 0 , . . . , n } , let S k ∼ U( { S k ⊆ [ I ] : | S k | = k } ) be a uniform random variable on the subsets of { 1 , . . . , I } of size k , and n S k = ∑ i ∈S k n i be the random variable defined by the sum of the elements of S k . Let K ∼ U( { 0 , . . . , I } ) and define Y = n S K . Then,

Proof. We prove (12) directly.

Thus, (12) follows as E [ µ K | K = k ] = µk . To prove (13), let ( X i ) k i =1 be k independent samples from the set N . From Lemma 1 it holds,

Therefore,

̸

̸

̸

The steps come from rearranging the terms, using the independence of X i with respect to K , the independence of X i , X j for i = j , and finally that E [ X i ] = µ and Var ( X i ) = σ 2 .

Lemma 3. Let I ∈ N , N := { n 1 , . . . , n I } ∈ R I + , and define,

Consider S K , n S K , K , and Y as in Lemma 2. Let w : R + → R be a function in C 2 , increasing, and suppose there exists κ ∈ R + , such that,

where w ( k ) is the k-th derivative of w . Then, it holds,

Proof. The proof considers a second-order Taylor extension of w at µk to recover the expected value of E [ w ( µ K ) -w ( Y )] . Noticing that the first derivative has a null expected value, the upper bound stated on the Lemma comes from bounding the expected value of the second derivative.

The Taylor-Lagrange Theorem on w at µk > 0 provides,

for some τ between y and µk . Therefore, there exists a random variable T , almost surely between µ K + and Y , such that,

where K + corresponds to K conditioned to be positive. To avoid overcharging the notation, we drop the index from K + . We observe that,

by Lemma 2, Equation (12). Therefore,

Setting I := { | µ K -Y | ≤ 1 2 ( µ K + Y ) } , the previous expected value can be expressed as,

We deal with each term separately. Notice that, as T is almost surely between Y and µ K ,

Thus,

Regarding the second term, denote n := max i ∈I n i and n := min i ∈I n i . As K n ≤ min { µ K , Y } ≤ T , we have,

as the integral corresponds to the cumulative distribution function of an exponential random variable of parameter λ = µ 2 / 2 n . The upper bound on the theorem's statement is obtained when gathering all together.

Lemma 4. Let w : R + → R + be a smooth and increasing function such that

Then, there exists κ> 0 such that n 2 | w (2) ( n ) | ≤ κ .

Proof. Notice that the assumptions imply, in particular, that | w (2) ( n ) | is bounded. We argue by contradiction. Suppose that for any m> 0 , there exists n m such that

Suppose the sequence ( n m ) m converges to a point n ∗ . Then,

which is a contradiction with | w (2) ( n ) | being bounded. Therefore, necessarily ( n m ) m has to diverge. However, this implies,

obtaining again a contradiction.

C.3.2 Proof of Theorem 5

We are ready to prove Theorem 5.

Theorem 5. Under Assumption H 1, there exists a constant κ > 0 , such that, for any i ∈ I , it holds,

where φ i and ψ i are respectively the Shapley value and the DU-Shapley of player i , µ -i = 1 I -1 ∑ j ∈I\{ i } n j is the average dataset size of other players, σ 2 -i = 1 I -1 ∑ j ∈I\{ i } ( n j -µ -i ) 2 its empirical variance, and ζ -i measures the variability of the dataset sizes across players. Formally, it is defined as

where R -i := max j ∈I\{ i } | n j -µ -i | , n max -i := max j ∈I\{ i } n j , and τ -i := n max -i min j ∈I\{ i } n j .

Proof. Under Assumption H 1, Lemma 4 implies the existence of κ> 0 such that the value function w satisfies all assumptions from Lemma 3. Theorem 5 comes from (a) noticing that

where K ∼ U([ I -1]) and Y -i = n S ( i ) K with S ( i ) K taking values on the subsets of I \ { i } of size K , (b) writing

and (c) applying Lemma 3 to each of the expected values, as the function n → w ( n + n i ) also satisfies H 1.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Both abstract and introduction states exactly what we proved in our article.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: A limitations section is included in Section 5. Moreover, all mathematical assumptions are clearly stated on the main article and discussed.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Assumptions are stated in each theoretical result and all proofs are included in the supplementary material. We have carefully checked the accuracy of all mathematical steps.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: At the beginning of each numerical results section we explain the used methodology and the sources of the used data. Besides, Appendix B complements these descriptions.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The codes are included and the implementations are detailed in the appendix.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All details are explained in each numerical results section.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The experiments in Section 4.2 and Appendices A.1 and A.2 present error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All details are included at the beginning of Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted conformes with all points in the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss potential applications in Section 1.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All creators are properly credited by citing their works.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of dataset valuation, a variant of data valuation, where the contributions of multiple datasets, instead of single data points, to a metric such as performance of a machine learning prediction model are quantified. The authors analyze the computation of the Shapley value, where each player represents a single dataset. In this context, it is argued that the utility function (or game) is essentially determined by the size of each dataset, which is theoretically supported by three general synthetic use-cases involving piece-wise constant functions, homogeneous data distributions, and linear regression with noisy labels, related to local differential privacy. The author present \"Discrete Uniform\" Shapley values (DU-Shapley), which approximates the Shapley value under this assumption efficiently. If the assumption on the utility function is satisfied, then the method can be further simplified by constructing suitable intervals instead of sampling, where the authors show for all use-cases asymptotic convergence (for infinite number of datasets) and non-asymptotic theoretical guarantees for the first and second use-case. The authors then evaluate their approach empirically, on multiple benchmark datasets against permutation-based Monte Carlo approaches, and on OpenDataVal benchmark tasks.\n\nSTRENGTHS:\n- Dataset valuation is an important variant of data valuation and analyzing the structural properties of the involved utility function is an important research question. \n- The paper is well-structured and the mathematical notation is precise and understandable\n- The paper presents interesting theoretical results for synthetic use-cases, which have direct consequences for the computatability\n- The empirical performance on OpenDataVal tasks is promising\n\nWEAKNESSES:\n- The baselines in the experimental evaluation of the Shapley value approximation are limited. The authors only consider permutation sampling (with antithetic sampling as a variant) as the main competitor. However, there exist a variety of other methods, cf. evaluations in [1], that have been applied in data valuation and outperform this approach\n- There is no evaluation, if the assumptions / structures of DU-Shapley specific to dataset valuation are apparent in real-world applications. Especially, it remains unclear if the synthetic use-cases are related to real-world applications.\n- Section 3.2 is hard to follow, especially the derivation of (10) from Definition 3. In particular, it remains unclear to me, how Definition 3 and (10) are the same estimator, and what is the intuition behind Definition 3, which is used in real-world applications.\n\n**Minor Comments**\n\n- In Table 1, it is unclear which values are bold (top-2 or best)? It seems inconsistent.\n- Related work could be improved, there are several algorithms for Shapley values, cf. the methods in [1,2,3].\n- Instead of saying \"exponentially reduce the complexity\", e.g. line 62,  it would be better to state the complexity\n\n\n[1] Li, W., & Yu, Y. (2023, October). Faster Approximation of Probabilistic and Distributional Values via Least Squares. In The Twelfth International Conference on Learning Representations.\n\n[2] Chen, H., Covert, I. C., Lundberg, S. M., & Lee, S. I. (2023). Algorithms to estimate Shapley value feature attributions. Nature Machine Intelligence, 5(6), 590-601.\n\n[3] Kolpaczki, P., Bengs, V., Muschalik, M., & Hüllermeier, E. (2024, March). Approximating the shapley value without marginal contributions. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 12, pp. 13246-13255).\n\nQUESTIONS:\n- Could you explain a bit more the intuition behind the DU-Shapley approach and Definition 3? In particular, the definition of $D^{(k)}$ is unclear to me, \"is a set of data points uniformly sampled without replacement\", it seems to me that it is the union of $k$ datasets rather than $k$ data points? If not, then $u$ is not defined for this setting? Maybe a pseudo-code algorithm would help.\n- Could you elaborate on section 3.2? I did not understand the derivation of (10) from Definition 3 using the assumption. In particular, how does (10) not depend on the sampled data points anymore from Definition 3? These seem to be two different estimators.\n- It seems that DU-Shapley has to be computed for every individual dataset separately? Could you comment on this limitation, and is there a way to circumvent it? I think this should be included in the limitations sections. Furthermore, the method then has quadratic complexity?\n\nLIMITATIONS:\nThe author discuss limitations of their work, which centers around the applicability of the approach when dataset sizes and variances are strongly heterogeneous.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel proxy for the Shapley value (SV) which is called discrete uniform (DU) Shapley. Unlike the SV, this proxy does not suffer from the exponential complexity problem of the SV. Of course the DU Shapley, therein does not capture all information entailed in the SVs for a cooperative game. The DU Shapley are motivated and derived from the setting of dataset valuation. Dataset valuation is a similar setting to data valuation, with the difference being that not individual data points are making up the player set but collection of data points. The work studies this dataset valuation setting and the DU Shapley theoretically and performs empirical experiments illustrating the effectiveness.\n\nSTRENGTHS:\n- **Theoretical Results:** The work studies the setting of dataset valuation in a theoretical and well-substantiated manner. While I think details of 2.1.1, 2.1.2, and 2.1.3 may be deferred to the appendix to expand on 2.2 and 2.3, the theoretical analysis of the dataset valuation problem is a good contribution. Further, the theoretical results regarding DU Shapley and the mechanisms behind it (for Example Figure 1 and Figure 2) support the work well. \n- **Empirical Evaluation For Data Valuation:** While DU Shapley is not proposed for the data valuation task but rather dataset valuation, the work also contains evaluations with OpenDataVal highlighting that DU Shapley can also be applied to this setting. This is particularly interesting since these values are quite cheap to compute offering a good baseline for future work or an initial insight before applying more costly approaches.\n- **Empirical Evaluation in real-world scenarios:** The paper shows how DU Shapley seems to be a viable approach to use as a proxy stand-in compared to other sampling-based baseline methods in traditional machine learning settings. This shows that even when the assumptions made on the value function are not fulfilled the estimates are still better than estimates from alternative methods. I still think better comparisons could have been made (c.f. Weakness Section).\n- **Reproducible:** Working on a related problem with the same problem formulation, I observe the same results this paper hypothesizes and describes in its formulation. With increasing dataset size the worth of a dataset increases (though not monotonically but still increases). The proxy approach of DU Shapley seems to work very well in this setting. I suspect this also to hold for different data modalities like text or images, which could be a fruitful future research direction.\n\nWEAKNESSES:\n## Major Weaknesses\n- **Weak Baselines** The empirical evaluation (comparison against sampling) compares the effectiveness of DU Shapley in with subpar and not interesting baselines. The evaluation does not include the KernelSHAP [1] method for computing SVs. KernelSHAP in many areas outperforms the sampling based estimators by a large margin. Further, as the work hypothesizes and arguments strongly that the size of a dataset (and thus also the coalition size) is important to determine the worth of it, methods like SVARM [2] that are derived for this setting are also missing from the empirical comparison. Both of these methods are not compared, which might strongly alleviate the performance increase of DU Shapley compared to the current baselines.\n- **Weak Sampling Budget** It seems like the empirical evaluation comparing DU Shapley with the sampling-based black-box estimators (Section 4.1) uses a very low computational budget for the estimators. While it is true that when DU Shapley requires a very low budget, a fair comparison would want to compare the current state of the art in the same settings. However, DU Shapley is a proxy for the SV. There it is definitely interesting to also see when the state-of-the-art catches up with the proxy. While it is very obvious that the proxy DU Shapley value is much better in these very low dataset valuation settings. However, the other methods estimate the SV and might get better in a reasonable time (in terms of computational budget). This should be evaluated well with proper error/evaluation curves for different estimation budgets.\n\n## Smaller Weaknesses\n- **Too little Focus on Data Valuation vs. Dataset Valuation:** I think this work may benefit from a better high-level motivation, and looses itself in too much details (2.1.1, 2.1.2, and 2.1.3 make up 1.5 pages). These motivating examples could be shorter and placed in the appendix to make more room for a broader theoretical background delineating this works contribution from the classical data valuation setting better than the rather short intermission with Section 2.3. I would also think that actually plotting/visualizing these Dataset Valuation attribution values for a ground truth setting would help. It is not tangible how these values actually look for real-world scenarios from Section 4.2.\n\n\n## References\n- [1] https://proceedings.neurips.cc/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf (KernelSHAP in my opinion is better explained here: https://proceedings.mlr.press/v130/covert21a.html)\n- [2] https://ojs.aaai.org/index.php/AAAI/article/view/29225\n\nQUESTIONS:\n1. How does DU Shapley compare against stronger approximations like Kernel-based estimators (aka. KernelSHAP) or stratified methods like SVARM?\n2. I am a bit confused by your statement in line 305. Did you run the MC estimations on 10 or 20 subset evaluations as the computational budget? The axis description in Figure 3 also does not help.\n\n## Suggestions\nI have run a similar setup for the cal. housing experiment as you do in Section 4.1. I compute the ground-truth SVs for a 10 player dataset valuation setting. The SVs are as follows (empty-set is manually set to 0):\n- $D_1$: 0.05790964954064733\n- $D_2$: 0.06817158889611828\n- $D_3$: 0.07737767986447398\n- $D_4$: 0.08361640115201878\n- $D_5$: 0.08403403608007484\n- $D_6$: 0.08978074706995469\n- $D_7$: 0.08771940889382171\n- $D_8$: 0.08772011962276798\n- $D_9$: 0.09842460151765661\n- $D_{10}$: 0.11083578157985912\n\nThe dataset sizes are monotonically increasing from $1$ to $10$, which is inline with the papers claims in regards to the behavior of the SVs. The SVs also increasing (though not so monotonically but still increase) mainly depending on the size of the datasets. Note that for this a gradient boosted tree (as implemented in xgboost) is fitted without any HPO. I suggest to add something similar to this in an illustrative/abstract way to show how the value function behaves and make this clear to a potentially unfamiliar reader too motivate this solution better.\n\nLIMITATIONS:\nThe work contains a separate limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies dataset evaluation with Shapley value. They assume that each dataset contains i.i.d. data points from a distribution and exploit the knowledge about the structure of the underlying data distribution to design more efficient Shapley value estimators. Based on asymptotic analysis of Shapley values of datasets, they propose an approximation of Shapley values of datasets. They prove estimation error bounds for non-asymptotic settings. The estimation accuracy is tested via numerical experiments in certain settings in comparison to other approaches. They also test the effectiveness of the approach for noisy label detection, dataset removal, and dataset addition.\n\nSTRENGTHS:\nThe paper explores a novel setting in which Shapley values are computed for datasets comprising i.i.d. data points. Leveraging the distributional structures of datasets, they propose an approximation method that reduces running time. Their theoretical analysis is novel and strong, offering valuable insights into the distribution of Shapley values for datasets under some assumptions. They show by experimental results that this approximation method outperforms other MC-based approaches with computational power is limited.\n\nWEAKNESSES:\nIn the context of dataset evaluation, previous data point Shapley approximation methods can still be applied by treating the dataset as a single entity. The main advantage of this new approach appears to be its improved computational efficiency compared to earlier Data Shapley methods. However, the computational efficiency improvement is not thoroughly discussed or demonstrated through experiments, which primarily focus on estimation accuracy.\n\nThe theoretical analysis relies on certain distributional assumptions, potentially limiting the approach's applicability. For instance, the accuracy of the estimation seems dependent on having a large number of data providers. In contrast, previous Data Shapley approximation methods, such as those proposed by Jia et al. (2019), do not appear to require these assumptions. Furthermore, the difference in the number of retraining iterations, O(n log n) versus O(n), is not substantial.\n\nThe use cases for dataset valuation are not well articulated. Common tasks like noisy label detection, data removal, and data addition are typically considered at the data point level. It is unclear why the data needs to be removed as a set rather than individually.\n\nQUESTIONS:\n1. What’s the reason for choosing 20 SGD steps and 20 boosting iterations? Will the experiment results significantly change when the numbers increase?\n2. Could you explain the improvement in computational efficiency compared to Jia et al. 2019?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a general dataset valuation method which only requires N number of utility function evaluations, where N is the number of players. The approach is motivated by a homogeneous setting for linear regression, but experiments also show the effectiveness of such an approach in general settings.\n\nSTRENGTHS:\nThe paper is easy to read and has a very clear structure. The topic is very important as existing data valuation approaches mostly focus on data point-level valuation, while this paper focus on dataset-level valuation.\n\nWEAKNESSES:\nI have reviewed this paper at ICML. The paper has addressed most of the issues in this revision and has a huge improvement. My remaining concerns are the following: \n\n**Computational complexity.** \nWhile DU-Shapley only requires $I$ utility function evaluations for a single data owner, if we want to evaluate all data owners it still requires $I^2$ function evaluations. This is worse than the group-testing estimator introduced in [1] (a refined version can be found in [2]), and I believe [3] has already reduced the complexity to $O(I \\log I)$. \n\nI did not verify the correctness of the results in Section 3.3, but the expressions seem to be reasonable. \n\n[1] Jia, Ruoxi, et al. \"Towards efficient data valuation based on the shapley value.\" The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.\n\n[2] Wang, Jiachen T., and Ruoxi Jia. \"A Note on\" Towards Efficient Data Valuation Based on the Shapley Value''.\" arXiv preprint arXiv:2302.11431 (2023).\n\n[3] Li, Weida, and Yaoliang Yu. \"Faster Approximation of Probabilistic and Distributional Values via Least Squares.\" The Twelfth International Conference on Learning Representations.\n\nQUESTIONS:\nWhat's the specific MC estimator being used for computing Data Shapley in the experiment?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-25"}