Title: Robust Data Valuation with Weighted Banzhaf Values
Venue: NeurIPS 2023
Accepted: yes
Keywords: data valuation

robustness

weighted Banzhaf values
Abstract: Data valuation, a principled way to rank the importance of each training datum, has become increasingly important. However, existing value-based approaches (e.g., Shapley) are known to suffer from the stochasticity inherent in utility functions that render consistent and reliable ranking difficult. Recently, Wang and Jia (2023) proposed the noise-structure-agnostic framework to advocate the Banzhaf value for its robustness against such stochasticity as it achieves the largest safe margin among many alternatives. Surprisingly, our empirical study shows that the Banzhaf value is not always the most robust when compared with a broader family: weighted Banzhaf values. To analyze this scenario, we introduce the concept of Kronecker noise to parameterize stochasticity, through which we prove that the uniquely robust semi-value, which can be analytically derived from the underlying Kronecker noise, lies in the family of weighted Banzhaf values while minimizing the worst-case entropy. In addition, we adopt the maximum sample reuse principle to design an estimator to efficiently approximate weighted Banzhaf values, and show that it enjoys the best time complexity in terms of achieving an $(\epsilon, \delta)$-approximation. Our theory is verified under both synthetic and authentic noises. For the latter, we fit a Kronecker noise to the inherent stochasticity, which is then plugged in to generate the predicted most robust semi-value. Our study suggests that weighted Banzhaf values are promising when facing undue noises in data valuation.

Robust Data Valuation with Weighted Banzhaf Values

Abstract

Data valuation, a principled way to rank the importance of each training datum, has become increasingly important. However, existing value-based approaches (e.g., Shapley) are known to suffer from the stochasticity inherent in utility functions that render consistent and reliable ranking difficult. Recently, Wang and Jia (2023) proposed the noise-structure-agnostic framework to advocate the Banzhaf value for its robustness against such stochasticity as it achieves the largest safe margin among many alternatives. Surprisingly, our empirical study shows that the Banzhaf value is not always the most robust when compared with a broader family: weighted Banzhaf values. To analyze this scenario, we introduce the concept of Kronecker noise to parameterize stochasticity, through which we prove that the uniquely robust semi-value, which can be analytically derived from the underlying Kronecker noise, lies in the family of weighted Banzhaf values while minimizing the worst-case entropy. In addition, we adopt the maximum sample reuse principle to design an estimator to efficiently approximate weighted Banzhaf values, and show that it enjoys the best time complexity in terms of achieving an ( ϵ, δ ) -approximation. Our theory is verified under both synthetic and authentic noises. For the latter, we fit a Kronecker noise to the inherent stochasticity, which is then plugged in to generate the predicted most robust semi-value. Our study suggests that weighted Banzhaf values are promising when facing undue noises in data valuation.

1 Introduction

In machine learning, data curation plays a crucial role in training powerful models, and one big difficulty facing practitioners is how to quantify the extent of usefulness each datum possesses (under a specific application). In practice, large datasets are typically noisy and often contain mislabeled data, which could harm the downstream performance of models trained on top. Sources that affect data quality include mechanical or human labelling errors (Frénay and Verleysen 2013), and poisoning attacks from malicious adversaries (Steinhardt et al. 2017). Therefore, it would be extremely useful if the potential importance of each datum can be efficiently and reliably determined. Data valuation aims to lay out a principled way to filter out noisy data and hunt down data that are valuable to retain.

In the emerging field of data valuation, the concept of value , originally developed in cooperative game theory as an axiomatic and principled way to assign contributions to all players in a game, has become increasingly popular in machine learning (e.g., Ghorbani and Zou 2019; Jia et al. 2019a,b; Kwon and Zou 2022; Lin et al. 2022; Wang and Jia 2023). Typically, the procedures of training and testing are treated as games that constitute utility functions while each datum serves as a player. The contribution imputed to each player is then equivalent to the importance determined for each datum.

Recently, Wang and Jia (2023) pointed out that the inherent stochasticity in utility functions, which stems from the procedure of training models, is substantial enough to incur unexpected inconsistency in ranking data. To solve this issue, they introduced the noise-structure-agnostic framework to

demonstrate that the Banzhaf value (Banzhaf 1965) is the most robust against such stochasticity compared with other previously employed values. Surprisingly, our extensive experiments on various datasets, see Figure 1 for some examples, reveal that the Banzhaf value is not always the most robust compared with weighted Banzhaf values, to which the Banzhaf value belongs.

FMNIST (

10

classes)

MNIST (

10

classes)

gas (

6

classes)

Spearman's rank correlation coefficient

Spearman's rank correlation coefficient

1.0

0.8

0.6

0.4

0.2

-0.2

—0.2

0.00

0.25

0E:0

0

1.00

0.00

0.05

0

0

0

0

0.95

1.00

88

0

15.

0.75

！

semi-value

semi-value

semi-value

0.7

0.8

0.5

0.6

0.7

0.6

0.4

0.5

0.3

0.2

0.2

0.2

0.1

0.1

0.1

0.0000

3

0

4

.0

0.0

0000

0

0.0

0.00

S0:0

0.10

0.15

0

0

0

semi-value

semi-value

semi-value

In this work, our goal is to shed further light on the above-mentioned phenomenon both theoretically and empirically. As one might expect, our extensive experiments confirm that overall there is no single universally the best value, in terms of either noisy label detection or ranking consistency, across a variety of experiment settings. Nevertheless, we show that it is possible to adaptively estimate a weighted Banzhaf value that achieves competitive robustness compared with the existing baselines. We summarize our contributions as follows:

First, we introduce the concept of Kronecker noise to parameterize the inherent stochasticity in practice, and prove that the uniquely most robust semi-value lies in the family of weighted Banzhaf values in terms of minimizing the worst-case entropy. Specifically, the parameter for the most robust semi-value can be directly derived from the underlying Kronecker noise.

Second, it is known that evaluating each semi-value is exponentially expensive w.r.t. the number of data being valuated, and thus we adopt the maximum sample reuse principle to design an efficient estimator for approximating weighted Banzhaf values. Particularly, we show that it enjoys the best time complexity in terms of achieving an ( ϵ, δ ) -approximation.

Third, both synthetic and authentic noises are employed in our experiments to verify the robustness assertion in Theorem 1. For the latter, we fit a Kronecker noise to the inherent stochasticity by minimizing the Kullback-Leibler divergence between Gaussian distributions.

Last, our empirical study shows that it is often that some member of weighted Banzhaf values achieves the best performance in noisy label detection as well as ranking consistency when compared with the existing baselines. The empirical evidence highlights that there is no single universally the best value across various experiment settings.

Our code is available at https://github.com/watml/weighted-Banzhaf .

2 Background

2.1 Notations

Let D tr and D val be training and validation datasets, respectively. We write n = |D tr | , and identify D tr with [ n ] = { 1 , 2 , . . . , n } . The definition of utility function v : 2 [ n ] → R is pivotal for employing valued-based data valuation methods. Typically, for each subset S ⊆ [ n ] , v ( S ) is set to be the performance of a chosen model trained on S . For example, v ( S ) is usually the accuracy measured on D val for classification tasks. Specifically, v ( ∅ ) is set to be the performance of randomly initialized models in this work. Let G n be the set that contains all possible utility functions with domain 2 [ n ] , and define G = ⋃ n ≥ 1 G n . Particularly, the subscript of v n ∈ G is to indicate the number of data being valuated. For convenience, we write S ∪ i and S \ i instead of S ∪ { i } and S \{ i } , and use s = | S | when S represents a set.

2.2 Axioms for Values

Value-based data valuation methods take advantage of the axiomatic approach developed in cooperative game theory (Dubey et al. 1981; Shapley 1953; Weber 1977). A value is a function ϕ that maps each utility function v n ∈ G to a contribution vector c ∈ R n . In other words, ϕ i ( v ) = c i is the contribution of the i -th datum in D tr for the model performance achieved by training on D tr . There are four axioms we expect a value to have, which are

Linearity : ϕ ( a · u n + v n ) = a · ϕ ( u n ) + ϕ ( v n ) for every u n , v n ∈ G and every a ∈ R ;

Dummy : ϕ i ( v n ) = C if v n ( S ∪ i ) = v n ( S ) + C for every S ⊆ [ n ] \ i ;

Monotonicity : ϕ i ( v n ) ≥ 0 if v n ( S ∪ i ) ≥ v n ( S ) for every S ⊆ [ n ] \ i ;

Symmetry : 1 ϕ ( v n ◦ π ) = ϕ ( v n ) ◦ π for every permutation π of [ n ] .

As proved by Weber (1977, Theorems 5 and 10), 2 a value ϕ satisfies the above four axioms if and only if there exists a list of non-negative vectors Ξ = ( p n ∈ R n ) n ≥ 1 such that, for each n ≥ 1 , ∑ n s =1 ( n -1 s -1 ) p n s = 1 and

All such values are called probabilistic values.

There is one more regularization implicitly used by Dubey et al. (1981) to filter out some undesired probabilistic values. The eventual regularized probabilistic values are called semi-values. 34 We explicitly rephrase the regularization as an extra axiom in this paper. Let z be an extra datum labelled by n +1 , and extend each utility function v n ∈ G into v n +1 defined by v n +1 ( S ) = v n ( S ∩ [ n ]) for every S ⊆ [ n +1] . The extra axiom reads as follows:

The axiom of consistency states that a value should not change the previously assigned values of importance if the added data do not contribute anything. Note that the proposed axiom of consistency is the discrete version of the dummy consistency proposed by Friedman (2003, Definition 7) in cost-sharing problems. We have the following characterization for semi-values:

Proposition 1 (Dubey et al. 1981, Theorem 1(a)) . A probabilistic value ϕ is a semi-value if and only if there exists a probability distribution P on the interval [0 , 1] such that

Moreover, the map from all such probability distributions to semi-values is one-to-one.

The axiom of consistency basically imposes that

which is the key for connecting all components in Ξ . All in all, combing Eqs. (1) and (2) yields the formula for each semi-value,

for every v n ∈ G and every i ∈ [ n ] . It suggests that each semi-value is just the integral of weighted Banzhaf values w.r.t. the underlying probability distribution P . Particularly, substituting the uniform distribution for P recovers the formula for the Shapley value (Shapley 1953) derived from the multilinear extension of utility functions (Owen 1972, Theorem 5).

2.3 Semi-values for Data Valuation

As a pioneering work, Ghorbani and Zou (2019) proposed Data Shapley that includes G-Shapley and TMC-Shapley for data valuation, and demonstrated its effectiveness in applications. Later, Beta Shapley was developed to improve the overall performance of value-based data valuation (Kwon and Zou 2022). Recently, Wang and Jia (2023) set up a noise-structure-agnostic framework to demonstrate that their proposed Data Banzhaf is the most robust compared with the previous works. Basically, the distinction between these methods is the use of semi-values. The underlying cumulative density functions (CDFs) of P in Eq. (2) for these value-based data valuation methods are listed in Table 1. Specifically, the TMC-Shapley (if the indices for truncation are the same for all sampled permutations of [ n ] ) is some probabilistic value up to some scalar.

In cooperative game theory references, weighted Banzhaf values (which are semi-values) were discovered by Ding et al. (2010) and Marichal and Mathonet (2011, Theorem 10) based on the approximation of utility functions (Hammer and Holzman 1992). 5 To obey the axiom of symmetry, the family of symmetric weighted Banzhaf values is taken herein, which can be parameterized by w ∈ [0 , 1] . Precisely, for w -weighted Banzhaf value, the corresponding coefficients in Eq. (1) are

Equivalently, its underlying CDF is χ [ w, 1] , while its underlying probability distribution is the Dirac delta distribution δ w defined by

Particularly, 0 . 5 -weighted Banzhaf value is exactly the Banzhaf value employed in Data Banzhaf, while 1 . 0 -weighted Banzhaf value is just the leave-one-out method, which is

3 Main Work

In this section, we first introduce how to use the maximum sample reuse (MSR) principle to design an efficient estimator for weighted Banzhaf values, and also present its time complexity in terms of ( ϵ, δ ) -approximation. Then, we introduce the concept of Kronecker noise to parameterize the underlying stochasticity in utility functions, through which we show that the uniquely most roust semi-value always lies in the family of weighted Banzhaf values while minimizing the worst-case entropy.

3.1 Efficient Estimator for Weighted Banzhaf Values

To exactly compute each probabilistic value shown in Eq. (1), we have to evaluate the provided utility function 2 n times in total, which is intractable when n is large, not to mention the expensive cost of training models. Therefore, in general, each probabilistic value can only be approximated using estimators. Specifically, the sampling lift estimator can be used to approximate every probabilistic value (Moehle et al. 2022), and its weighted version was employed by Kwon and Zou (2022) for approximating their proposed Beta Shapley values. Meanwhile, there are efficient estimators designed specifically for the Shapley value (e.g., Castro et al. 2009; Covert and Lee 2021; Jia et al. 2019b; Lundberg and Lee 2017). Recently, Wang and Jia (2023) demonstrated how to use the MSR principle to design an efficient estimator for the Banzhaf value, but they also presented that such a principle can not extend to many other probabilistic values (e.g., the Beta Shapley values). Nevertheless, we show that MSR estimator exists for weighted Banzhaf values.

For approximating w -weighted Banzhaf value based on some v n ∈ G , we first independently sample a sequence of Bernoulli vectors { b 1 , b 2 , . . . , b m } where the entries of each b j ∈ R n are drawn from n independent Bernoulli distribution X with P ( X = 1) = w ; then, the sequence of Bernoulli vectors are transformed into a sequence of subsets { S 1 , S 2 , . . . , S m } of [ n ] by defining i ∈ S j if and only if the i -th entry of b j is 1 ; finally, the approximate value is computed by

where S ∋ i = { S j | i ∈ S j } and S ̸∋ i = { S j | i ̸∈ S j } .

As shown in Eq. (4), each utility evaluation is employed for estimating the importance of every datum in [ n ] , while for the (reweighted) sampling lift it is two utility evaluations for only one datum; see Appendix D.2 for more details. Therefore, it could be expected that the MSR estimators are much more efficient, which is certainly confirmed by Figure 2. Meanwhile, such an MSR estimator enjoys the currently best time complexity bound demonstrated by Wang and Jia (2023, Theorem 4.9).

Proposition 2. Assume ∥ v n ∥ ∞ ≤ r for every v n ∈ G . The estimator based on Eq. (4) requires O ( n ϵ 2 log n δ ) utility evaluations to achieve P ( ∥ ˆ ϕ ( v n ; δ w ) -ϕ ( v n ; δ w ) ∥ 2 ≥ ϵ ) ≤ δ for every utility function v n ∈ G .

3.2 Robustness under Kronecker Noises

If 2 [ n ] is ordered, each stochastic utility function v n ∈ G can be treated as a random vector in R 2 n , and thus its randomness can be summarized by its covariance matrix D ∈ R 2 n × 2 n . However, there are Θ(2 n ) parameters to characterize. To make the analysis tractable, our approach is to parameterize D as simply as possible so that the most robust semi-value can be analyzed accordingly. Before proceeding, we introduce the binary ordering for 2 [ n ] , so that we can legally have the covariance matrix D .

Ordering of 2 [ n ] Define a binary sequence [0] 2 , [1] 2 , . . . , [2 n -1] 2 by letting [ k ] 2 be the binary representation of k in n -digit format. We order { S } S ⊆ [ n ] along this binary sequence by assigning to each binary representation [ k ] 2 a subset S k ⊆ [ n ] defined by i ∈ S k if and only if the ( n +1 -i ) -th digit of [ k ] 2 is 1 . For example, if n = 3 , the binary sequence is 000 , 001 , 010 , 011 , 100 , 101 , 110 , 111 , and the corresponding ordering for the subsets of [3] is ∅ , { 1 } , { 2 } , { 1 , 2 } , { 3 } , { 1 , 3 } , { 2 , 3 } , { 1 , 2 , 3 } . We refer to this ordering as binary ordering. In the rest of the paper, we treat each stochastic utility function v n ∈ G as a random vector in R 2 n w.r.t. the binary ordering. The advantage of this ordering

is that it aligns well with the Kronecker product so that we can parameterize the covariance matrix of each stochastic utility function using just one simple covariance matrix Σ ∈ R 2 × 2 .

Assumption 1 (Kronecker noises) . Let 2 [ n ] be ordered w.r.t. the binary ordering. For each random vector v n ∈ R 2 n (equivalently, a stochastic utility function in G ), we assume v n -E [ v n ] = ϵ n ⊗ ϵ n -1 ⊗··· ⊗ ϵ 1 where ( ϵ i ) 1 ≤ i ≤ n are n independent continuous random vectors in R 2 that have the same positive-definite covariance matrix Σ .

Proposition 3. Under Assumption 1, there is

Remark 1. According to Proposition 3, Assumption 1 provides a simple parameterization for the covariance matrix of the stochasticity inherent in v n . Though such a simple modeling is yet ideal, it is sufficient for arguing theoretically that there is no single universally the most robust semi-value across different stochastic utility functions. Notably, as shown in Table 2 , weighted Banzhaf values tend to be the most consistent in ranking data, and the most empirically robust one varies across different experiment settings. Overall, our theory emphasizes that a noise-structure-specific framework is able to provide a finer conclusion for robustness compared with the previous noise-structure-agnostic framework (Wang and Jia 2023).

Next, we propose to use the entropy of continuous random vectors to define the worst-case uncertainty brought by the parameterized covariance matrix, which is

where P is a probability distribution that defines a semi-value Eq. (3), h is the entropy that measures the uncertainty of continuous random vectors, and Σ [ n ] = Σ ⊗ Σ ⊗···⊗ Σ ( n repetitions). Intuitively, ϕ ( v n ; P ) tends to be deterministic as h ( ϕ ( v n ; P )) decreases to zero. Since Eq. (5) defines the least upper bound on h ( ϕ ( v n ; P )) , the lower Eq. (5) is, the more robust (equivalently, more deterministic) the corresponding semi-value is. Therefore, the most robust semi-value is defined to be the one that minimizes

where P is the set that contains all (Borel-measurable) probability distributions on the interval [0 , 1] . In other words, we try to determine a semi-value that can best tolerate the largest uncertainty brought by any noises having the covariance matrix Σ [ n ] . As one may expect, the most robust semi-value depends on the choice of Σ ∈ R 2 × 2 .

Since ϕ ( v n ; P ) is linear in v n ∈ G n , the result of Cov( ϕ ( v n ; P )) subject to Cov( v n ) = Σ [ n ] only depends on Σ [ n ] and P . Consequently, sup v n ∈G n : Cov( v n )= Σ [ n ] h ( ϕ ( v n ; P )) = sup Y : Cov( Y )= Ψ h ( Y ) where Ψ = Cov( ϕ ( v n ; P )) . It is known that sup Y : Cov( Y )= Ψ h ( Y ) = n 2 (1+log 2 π )+ 1 2 log det Ψ , and the maximum is achieved if Y is a Gaussian random vector. Besides, Y is Gaussian if v n is Gaussian because ϕ ( v n ; P ) is linear in v n ∈ G n . All in all, the optimization problem (6) equivalently reduces to be

Theorem 1. Under Assumption 1, and further assume σ 12 < min( σ 11 , σ 22 ) , the Dirac delta distribution δ w ∗ with

is optimal to the problem (7) for every n ≥ 1 . In addition, the optimal solution is unique if n > 3 .

Remark 2. Theorem 1 suggests that 0 . 5 -weighted Banzhaf value (i.e., the Banzhaf value) is the most robust provided that σ 11 = σ 22 , which coincides with (Wang and Jia 2023, Theorem 4.6). Specifically,

̸

they simulated some isotropic Gaussian noises added to deterministic utility functions to verify that the Banzhaf value is the most robust in ranking data, which is also supported by Theorem 1. However, the first column of Figure 3 shows that the Banzhaf value is not always the most robust if the added noises follow Kronecker noises with σ 11 = σ 22 . In practice, Table 2 also refutes that the Banzhaf value is universally the most robust.

4 Experiments

Our experiments contain three aspects: 1) approximation efficiency of the MSR estimator for weighted Banzhaf values; 2) verifying Theorem 1 using synthetic and authentic noises; 3) empirical study to show that the most robust/effective semi-values tend to be some weighted Banzhaf values.

All datasets used are from open sources, and are classification tasks. Except for MNIST and FMNIST, each D tr or D val is balanced between different classes. Without explicitly stated, we set |D val | = 200 . All utility functions are set to be the accuracy reported on D val with logistic regression models being trained on D tr , except that we implement LeNet (LeCun et al. 1998) for MNIST and FMNIST. To have the merit of efficiency, we adopt one-epoch onemini-batch learning for training models in all types of experiments (Ghorbani and Zou 2019). More experiment details and results are included in Appendices D and E.

4.1 Approximation Efficiency

The MSR estimator is compared with reweighted sampling lift and sampling lift estimators. To calculate the exact values, we set |D tr | = 16 . Besides, the learning rate is set to be 1 . 0 . To have fair comparison, we plot relative difference ∥ ˆ ϕ -ϕ ∥ 2 / ∥ ϕ ∥ 2 along the number of average utility evaluations, i.e. #utility evaluations n . Figure 2 is plotted with 10 different random seeds used by estimators. It demonstrates that the MSR estimator is significantly the most efficient.

4.2 Verification for the Theorem 1

Experiments based on synthetic and authentic noises are designed to verify Theorem 1. We set |D tr | = 10 so that the exact values can be cheaply computed.

For synthetic experiments, we simulate Kronecker noises with each ϵ i independently and identically following a Gaussian distribution N (0 , Σ ) , which are employed as added noises to deterministic utility functions. Besides, the learning rate is set to be 0 . 05 . Specifically, we fix σ 11 and σ 22 while varying σ 12 along the horizontal axis. For each σ 12 , the corresponding added noises are simulated using 10 different seeds. The Spearman's rank correlation coefficient is used to measure the similarity between the rankings of the ground-truth value and any noisy one. The results are reported in the first column of Figure 3, where the one tagged as 'robust semi-value' refers to the corresponding weighted Banzhaf values predicted by Eq. (8). It is clear that Theorem 1 predicts the most robust semi-value well, and that the Banzhaf value is not universally the most robust under these synthetic noises.

Next, we examine Theorem 1 by fitting a Kronecker noise to the stochasticity inherent in the given utility function, and then compute the predicted most robust semi-value using Eq. (8). To produce simple utility functions, we set |D val | = 10 . Besides, the learning rate for utility functions is set to be 1 . 0 . The underlying covariance matrix is empirically approximated by ˆ D , and then we fit a Kronecker

1.75

sampling lift

1.50

weightedsamplinglift

maximumsamplereuse

21.25

erer

diffe

1.00

@0.75

ative

0.50

e

0.25

0.00

0

250

500

750

1000

1250

1500

number of utility evaluations per datum

spambase (synthetic)

iris (authentic)

phoneme (authentic)

Spearman's rank correlation coefficient

1.00

Spearman's rank correlation coefficient

1.0

0.75

0.50

0.6

0.25

Shapley

0.85

Banzhaf

0.00

0.2

Beta(16,1)

0.25

0.0

Beta(4,1)

Beta(1, 4)

-0.50

0.2

Beta(1, 16)

robust semi-value

0.4

0.2

0.0

012

0.2

0.4

semi-value

semi-value

0.200

0.4

0.175

0.150

0.3

0.9

Shapley

Banzhaf

0.8

Beta(16,1)

Beta(4,1)

0.075

Beta(1, 4)

Beta(1, 16)

0.1

0.050

robustsemi-value

0.025

0.0

-0.4

0.2

012

0.0

0.2

0.4

semi-value

e

semi-value

noise by optimizing the Kullback-Leibler divergence between Gaussian distributions, which is

The final results averaged over 10 random seeds are presented in the last two columns of Figure 3. Each parameter in { 0 . 00 , 0 . 05 , 0 . 10 , . . . , 0 . 95 , 1 . 00 } defines a specific weighted Banzhaf value, while { (16 , 1) , (4 , 1) , (1 , 4) , (1 , 16) } are parameters for the Beta Shapley values. The one tagged as 'robust' is generated from the approximate Kronecker noise using Eq. (8). As demonstrated, the predicted 'robust' semi-values do achieve the best in both ranking consistency and variance.

4.3 Empirical Study

Following the previous study (Wang and Jia 2023), we implement an exploratory study on 23 datasets, including two types of experiment: 1) noisy label detection and 2) ranking consistency.

We fix |D tr | = 1 , 000 for all datasets except that it is |D tr | = 2 , 000 for MNIST and FMNIST, which means the induced stochasticity inherent in utility functions is more complicated than Assumption 1. For each dataset, the chosen model is individually fine-tuned for the learning rate. The randomness in each utility function stems from the underlying random seed that determines the order of feeding training data as well as the initialization of models. All results are reported using mean and standard deviation over 10 random seeds. For each estimator, the total number of utility evaluations is set to be 400 , 000 . Permutation estimator (Castro et al. 2009) is used for Shapley value, reweighted sampling lift estimator for Beta Shapley, and MSR estimator for weighted Banzhaf values. Specifically, 0 -weighted Banzhaf value and 1 -weighted Banzhaf value require only n + 1 utility evaluations to compute exactly, and thus we do not use any estimators for them.

Ranking Consistency In data valuation, it could be the ranking of data that matters the most, but the randomness of utility functions may produce inconsistent rankings. For each semi-value, the averaged Spearman's rank correlation coefficients over all possible pairs, which is ( 10 2 ) = 45 in total, are calculated to measure ranking consistency. The results are summarized in Table 2, while figures with more details are included in Appendix E. As presented, it is mostly some member of weighted Banzhaf values, not always the Banzhaf value, that achieves the highest ranking consistency compared with the baselines.

Noisy Label Detection Notice that a more robust semi-value does not necessarily produce a higher quality of data valuation. This experiment is to present how well weighted Banzhaf values would achieve in noisy label detection. For each dataset, we randomly flip the labels of 20 percent of data in D tr to be any of the rest in a uniform manner. Those flipped data are treated as having noisy/incorrect labels. Particularly, such a flipped D tr is also used for reporting ranking consistency in Table 2. Then, each estimated semi-value is used to detect noisy labels by marking data assigned with values of importance less than or equal to 20 percent percentile as mislabeled. The results are summarized in Table 3, and figures that contain more details are included in Appendix E. As observed, the best performance is mostly achieved by weighted Banzhaf values, and most of them are not the Banzhaf value.

In both types of experiments, it manifests itself that the semi-value that achieves the best in ranking consistency does not coincide with the one that achieves the highest F1-score on each dataset. Nevertheless, the detailed figures in Appendix E show that weighted Banzhaf values tend to be more robust than the Beta Shapley values, which include the Shapley value. Moreover, we can conclude that there is no single universally the best semi-value across various experiment settings in terms of either ranking consistency or noisy label detection. For each utility function, it would be useful if the best semi-value can be effectively determined in advance.

5 Conclusion

We introduce a noise-structure-specific framework to theoretically assert that it is always a member of weighted Banzhaf values that is deemed the most robust in terms of minimizing the worst-case entropy. Theorem 1 shows that the most robust semi-value would vary across different experiment settings. Our extensive empirical study demonstrates that weighted Banzhaf values are promising when dealing with inevitable noises inherent in utility functions. Particularly, all experiments suggest that there is no universally the best or the most robust semi-value for all experiment settings. Therefore, future investigation is to set up principled approaches that can pin down the most robust or the most effective value in advance.

6 Limitations

The concept of Kronecker noise is still ideal for fitting the stochasticity in practice. The verification of Theorem 1 using fitted Kronecker noises can only be done based on simple utility functions, as one may expect. Nevertheless, the theoretical and empirical evidence is sufficient to argue that there is no single universally the most robust or the most effective semi-value across various experiment settings, and that weighted Banzhaf values are promising in practice.

Acknowledgments and Disclosure of Funding

We thank the reviewers and the area chair for thoughtful comments that have improved our final draft. YY gratefully acknowledges NSERC and CIFAR for funding support.

References

Banzhaf III, J. F. (1965). 'Weighted Voting Doesn't Work: A Mathematical Analysis'. Rutgers Law Review , vol. 19, pp. 317-343.

Castro, J., D. Gómez, and J. Tejada (2009). 'Polynomial Calculation of the Shapley Value Based on Sampling'. Computers & Operations Research , vol. 36, no. 5, pp. 1726-1730.

Covert, I. and S.-I. Lee (2021). 'Improving KernelSHAP: Practical Shapley Value Estimation via Linear Regression'. In: International Conference on Artificial Intelligence and Statistics . PMLR, pp. 3457-3465.

Ding, G., R. F. Lax, J. Chen, P. P. Chen, and B. D. Marx (2010). 'Transforms of Pseudo-Boolean Random Variables'. Discrete Applied Mathematics , vol. 158, no. 1, pp. 13-24.

Domenech, M., J. M. Giménez, and M. A. Puente (2016). 'Some Properties for Probabilistic and Multinomial (Probabilistic) Values on Cooperative Games'. Optimization , vol. 65, no. 7, pp. 1377-1395.

Dragan, I. (2006). 'On the Semivalues and the Least Square Values Average Per Capita Formulas and Relationships'. Acta Mathematica Sinica , vol. 22, pp. 1539-1548.

Dubey, P., A. Neyman, and R. J. Weber (1981). 'Value Theory without Efficiency'. Mathematics of Operations Research , vol. 6, no. 1, pp. 122-128.

Frénay, B. and M. Verleysen (2013). 'Classification in the Presence of Label Noise: A Survey'. IEEE Transactions on Neural Networks and Learning Systems , vol. 25, no. 5, pp. 845-869.

Friedman, E. J. (2003). 'Paths and Consistency in Additive Cost Sharing'. International Journal of Game Theory , vol. 32, no. 4, pp. 501-518.

Ghorbani, A. and J. Zou (2019). 'Data Shapley: Equitable Valuation of Data for Machine Learning'. In: International Conference on Machine Learning . PMLR, pp. 2242-2251.

Hammer, P. L. and R. Holzman (1992). 'Approximations of Pseudo-Boolean Functions; Applications to Game Theory'. Zeitschrift für Operations Research , vol. 36, no. 1, pp. 3-21.

Jia, R., D. Dao, B. Wang, F. A. Hubis, N. M. Gurel, B. L. C. Zhang, and C. S. D. Song (2019a). 'Efficient Task-Specific Data Valuation for Nearest Neighbor Algorithms'. Proceedings of the VLDB Endowment , vol. 12, no. 11.

Jia, R. et al. (2019b). 'Towards Efficient Data Valuation Based on the Shapley Value'. In: The 22nd International Conference on Artificial Intelligence and Statistics . PMLR, pp. 1167-1176.

Kwon, Y. and J. Zou (2022). 'Beta Shapley: A Unified and Noise-reduced Data Valuation Framework for Machine Learning'. In: International Conference on Artificial Intelligence and Statistics . PMLR, pp. 87808802.

LeCun, Y., L. Bottou, Y. Bengio, and P. Haffner (1998). 'Gradient-Based Learning Applied to Document Recognition'. Proceedings of the IEEE , vol. 86, no. 11, pp. 2278-2324.

Lin, J., A. Zhang, M. Lécuyer, J. Li, A. Panda, and S. Sen (2022). 'Measuring the Effect of Training Data on Deep Learning Predictions via Randomized Experiments'. In: International Conference on Machine Learning . PMLR, pp. 13468-13504.

Lundberg, S. M. and S.-I. Lee (2017). 'A Unified Approach to Interpreting Model Predictions'. Advances in Neural Information Processing Systems , vol. 30.

Marichal, J.-L. and P. Mathonet (2011). 'Weighted Banzhaf Power and Interaction Indexes Through Weighted Approximations of Games'. European Journal of Operational Research , vol. 211, no. 2, pp. 352-358.

Moehle, N., S. Boyd, and A. Ang (2022). 'Portfolio Performance Attribution via Shapley Value'. Journal of Investment Management , vol. 20, no. 3, pp. 33-52.

Owen, G. (1972). 'Multilinear Extensions of Games'. Management Science , vol. 18, no. 5-part-2, pp. 64-79.

Shapley, L. S. (1953). 'A Value for N-Person Games'. Annals of Mathematics Studies , vol. 28, pp. 307-317.

Steinhardt, J., P. W. Koh, and P. Liang (2017). 'Certified Defenses for Data Poisoning Attacks'. In: Advances in Neural Information Processing Systems . Vol. 30.

Wang, J. T. and R. Jia (2023). 'Data Banzhaf: A Robust Data Valuation Framework for Machine Learning'. In: International Conference on Artificial Intelligence and Statistics . PMLR, pp. 6388-6421.

Weber, R. J. (1977). 'Probabilistic Values for Games'. The Shapley Value. Essays in Honor of Lloyd S. Shapley , pp. 101-119.

Table of Contents

A The Axiom of Equal Treatment

In the existing references, the axiom of symmetry is sometimes used to refer to the axiom of equal treatment, but they are not always interchangeable.

The axiom of equal treatment is not always equal to the axiom of symmetry. As a counterexample, pick n = 2 , and consider a value ϕ defined by ϕ 1 ( v ) = 1 2 v (1) + 1 2 v (2) and ϕ 2 ( v ) = 1 3 v (1) + 2 3 v (2) . The reader can verify that it satisfies the axiom of equal treatment but violates the axiom of symmetry (e.g., take v (1) = 3 and v (2) = 6 ).

Nevertheless, they are equivalent in some contexts. Recall that a weaker version of the axiom of dummy is

Proposition 4. Suppose a value ϕ verifies the axioms of linearity and null, then ϕ verifies the axiom of symmetry if and only if it verifies the axiom of equal treatment.

Proof. The only if part is trivial as the axiom of symmetry always implies the axiom of equal treatment, and thus we only prove the if part. Let n ≥ 1 be fixed, since ϕ verifies the axioms of linearity and null, by (Weber 1977, Theorem 2), 6 we have

̸

For every S ⊆ N , we define u S ∈ G n by letting u S ( T ) = 1 if T = S and 0 otherwise. For every i = j and S ⊆ [ n ] \ ij , by the axiom of equal treatment, there is

On the other hand, for every S with s > 1 , and every i, j ∈ S , there is

Combing Eqs. (9) and (10), we can get p i S = p s for every i ∈ [ n ] and every S ⊆ [ n ] \ i . To demonstrate, take p 1 { 2 , 3 } and p 4 { 5 , 6 } as example, there is

Therefore, the equality p i S = p s for every i ∈ [ n ] and every S ⊆ [ n ] \ i implies the axiom of symmetry.

B Proofs for the Propositions

Proposition 2. Assume ∥ v n ∥ ∞ ≤ r for every v n ∈ G . The estimator based on Eq. (4) requires O ( n ϵ 2 log n δ ) utility evaluations to achieve P ( ∥ ˆ ϕ ( v n ; δ w ) -ϕ ( v n ; δ w ) ∥ 2 ≥ ϵ ) ≤ δ for every utility function v n ∈ G .

Proof. Note that it only requires n + 1 utility evaluations for exactly yielding 0 and 1 -weighted Banzhaf values. Therefore, the MSR estimator is designed for 0 < w < 1 . Our proof is adapted from (Wang and Jia 2023, Theorem 4.9).

Let v n ∈ G and i ∈ [ n ] be fixed, and by assumption ∥ v n ∥ ∞ ≤ r . According to the underlying sampling for { S j } , there is |S ∋ i | ∼ binomial( w ) . Let a bridge estimator defined by

For simplicity, we write ϕ i , ˆ ϕ i and ϕ i instead of ϕ i ( v n ; δ w ) , ˆ ϕ i ( v n ; δ w ) and ϕ i ( v n ; δ w ) . Plus, we also have s ∋ i = |S ∋ i | and s ̸∋ i = |S ̸∋ i | .

First of all, we prove that E [ ϕ i ] = ϕ i . Suffice it to prove that E [ c i ( S ) v n ( S )] = ϕ i where c i ( S ) = 1 w if i ∈ S and -1 (1 -w ) otherwise. For every S ∈ [ n ] \ i ,

On the other hand, ϕ i = ∑ S ⊆ [ n ] \ i w s (1 -w ) n -1 -s ( v n ( S ∪ i ) -v n ( S )) .

Next, we prove the convergence for the MSR estimator. Since E [ ϕ i ] = ϕ i , by the Hoeffding's inequality,

where T = max( 1 w , 1 1 -w ) . Assume s ∋ i > 0 and s ̸∋ i > 0 , note that s ∋ i + s ̸∋ i = m , there is

Note that the first inequality in the above still holds if s ∋ i = 0 or s ̸∋ i = 0 . Since s ∋ i ∼ binomial( w ) , by the Hoeffding's inequality, there is

Therefore, | ˆ ϕ i -ϕ i | < r ∆ w (1 -w ) m provided that | s ∋ i -wm | < ∆ . Then,

where the last inequality is due to that 1 -2 exp ( -2∆ 2 m ) ≥ 2 3 if m is sufficiently large. Solving the equation m ( ϵ -r ∆ w (1 -w ) m ) 2 2 r 2 T 2 = 2∆ 2 m yields ∆ = w (1 -w ) mϵ r (2 Tw -2 Tw 2 +1) . Specifically, 2 Tw -2 Tw 2 +1 = 2 | w -0 . 5 | +2 , and thus its range is [2 , 3) provided that w ∈ (0 , 1) , which indicates ϵ -r ∆ w (1 -w ) m ≥ ϵ 2 . So, we have 2∆ 2 m = Cmϵ 2 where C = 2 w 2 (1 -w ) 2 r 2 (2 | w -0 . 5 | +2) 2 . Eventually,

provided that 1 -2 exp ( -2∆ 2 m ) ≥ 2 3 , or equivalently m ≥ log 6 Cϵ 2 . Then,

Solving 5 n exp ( -Cmϵ 2 n ) ≤ δ leads to m ≥ n Cϵ 2 log 5 n δ . Eventually, we conclude that the MSR estimator requires

samples (equivalently, utility evaluations) to achieve P ( ∥ ˆ ϕ -ϕ ∥ 2 ≥ ϵ ) ≤ δ .

Proposition 3. Under Assumption 1, there is

Proof. By Assumption 1, v n -E [ v n ] = ϵ n ⊗ ϵ n -1 ⊗··· ⊗ ϵ 1 such that ( ϵ i ) 1 ≤ i ≤ n are independent continuous random vectors in R 2 that have the same covariance matrix Σ . For simplicity, we write ϵ [ n ] = ϵ n ⊗ ϵ n -1 ⊗··· ⊗ ϵ 1 .

Since ( A ⊗ B )( C ⊗ D ) = ( AC ) ⊗ ( BD ) , there is

By the independence, we have

Next, we prove the rest by induction. It is clear that it holds for n = 1 . Suppose it holds for n = k > 0 , i.e., ,

and we proceed for n = k +1 . Suppose the binary ordering for { S } S ⊆ [ k ] is

then, the binary ordering for { S } S ⊆ [ k +1] is

which aligns well with ϵ k +1 ⊗ ( ϵ k ⊗ ϵ k -1 ⊗··· ⊗ ϵ 1 ) . Thus, we have v k +1 ( S ) = E [ v k +1 ( S )] + ∏ i ∈ S ϵ i, 2 ∏ i ̸∈ S ϵ i, 1 for every S ⊆ [ k +1] . On the other hand, write C [ k ] = Cov( v k ) , by Eq. (11), we have

Let S, T ⊆ [ k +1] be given, there are four cases in total: i) S, T ⊆ [ k ] ; ii) S, T ̸⊆ [ k ] ; iii) S ⊆ [ k ] , T ̸⊆ [ k ] ; and iv) S ̸⊆ [ k ] , T ⊆ [ k ] . For the first case, combing Eqs. (13), (14) and (12), there is

The rest of the cases can be done similarly.

C Prerequisites and Proof for Theorem 1

To prove Theorem 1, our first step is to lift each ϕ ( v n ; P ) into R 2 n w.r.t. the binary ordering. Precisely, we extend each semi-value ϕ P , a shorthand for ϕ ( · ; P ) , to be a semi-index I P that maps each v ∈ G n (for every n ≥ 1 ) into another utility function in G n , whose matrix representations on all G n are inductive. Then, we prove a theorem of robustness for semi-indices, which is then used to prove Theorem 1.

C.1 Extend Semi-values to Be Semi-indices

Index is another concept developed in cooperative game theory that serves as an extension for value (Fujimoto et al. 2006; Grabisch and Roubens 1999), which has also been employed in the community of machine learning (Sundararajan et al. 2020; Zhang et al. 2021). Specifically, an index I is a function that maps each utility function v ∈ G n (for every n ≥ 1 ) into another utility function in G n . Note that each utility function v ∈ G n can be viewed as a vector indexed by all subsets of [ n ] . In other words, I is to impute contributions to all possible subsets of [ n ] , not just to each datum in [ n ] , so as to account for the contributions of interactions among all data [ n ] . For convenience, we write I ( v n , S ) for the value of I ( v n ) indexed at S .

Notably, the axiom of recursion was proposed by Grabisch and Roubens (1999) to extend the Banzhaf and Shapley values into the Banzhaf and Shapley indices, respectively. To state the axiom of recursion, we need further definitions. Suppose v ∈ G n and S ⊆ [ n ] are given. The restriction of v on S , denoted by v S , is defined by

In other words, the domain 2 [ n ] of v is restricted to be 2 S .

Plus, let j ∈ [ n ] \ S be given, the restriction of v on S in the presence of j , denoted by v ∪ j S , is defined by

Remark 3. The definition of v ∪ j S is not the same as the one defined by Grabisch and Roubens (1999), which is instead v ∪ j S ( T ) = v ( T ∪ j ) -v ( j ) , ∀ T ⊆ S . The term -v ( j ) is only to make v ∪ j S ( ∅ ) = 0 , and we comment that it is not necessary for all the results therein. Moreover, this extra term will make Eq. (17) false.

, there is

The axiom of recursion dictates that the contribution imputed to the subset S should be the difference of the contributions imputed to S \ j in the presence and absence of any player j ∈ S . So, starting from I P ( v n , i ) = ϕ i ( v n ; P ) for every v n ∈ G and i ∈ [ n ] , the axiom of recursion can be used to define I P ( v n , S ) recursively for every non-empty subsets S ⊆ [ n ] . Particularly, the defined value of I ( v n , S ) is independent of the choice of j ∈ S .

One thing that remains is to determine the contribution for the empty subset. For each semi-value ϕ P , we argue that it should be what is stated as the axiom of empty.

Empty : To extend a semi-value ϕ P to be a semi-index I P , there should be I P ( v n , ∅ ) = ∑ S ⊆ [ n ] p n +1 s +1 · v ( S ) for every v n ∈ G . Recall that p n +1 s +1 = ∫ 1 0 t s (1 -t ) n -s d P ( t ) for every 0 ≤ s ≤ n .

Observe that ∑ S ⊆ [ n ] p n +1 s +1 = ∑ n s =0 ( n s ) p n +1 s +1 = 1 , the imposed value for I P ( v n , ∅ ) is just the expected value of v n w.r.t. the probability distribution ( p n +1 s +1 ) S ⊆ [ n ] induced by the underlying P ∈ P . One can verify the axiom of empty consistently extends the axiom of recursion, i.e.,

Definition 1 (Semi-indices) . A semi-index I P is extended from a semi-value ϕ P using the axioms of recursion and empty.

Remark 4. We can therefore extend each w -weighted Banzhaf value to be w -weighted Banzhaf index. Particularly, 0 -weighted Banzhaf index is exactly the Möbius transform Eq. (18) , while 1 -weighted Banzhaf index is instead the co-Möbius transform (Grabisch et al. 2000, Eq. (12)). In addition, extending the Shapley value and the Banzhaf value in this way will yield the Shapley interaction index and the Banzhaf interaction index presented in (Grabisch and Roubens 1999, Theorem 3).

Proposition 5 (Formulas for Semi-indices) . For each semi-index I P , there is

Proof. For simplicity, we write v instead of v n . It is clear that it holds when s = 0 and s = 1 . Assume it holds for all S such that | S | = k . Let S ⊆ [ n ] that satisfies | S | = k +1 , and some i ∈ S be given, there is

Note the above reasoning is independent of the choice of i ∈ S .

The reader can also verify that each semi-index I P satisfies the following axioms, the first three of which are often laid out for how indices should be in the fields of cooperative game theory and machine learning (Fujimoto et al. 2006; Grabisch and Roubens 1999; Sundararajan et al. 2020).

Linearity : I P ( a · u n + v n , S ) = a · I P ( u n , S ) + I P ( v n , S ) for every v n , u n ∈ G , a ∈ R and S ⊆ [ n ] .

Dummy Interaction : If S ⊆ [ n ] with s > 1 contains a datum i such that for some constant C , v n ( T ∪ i ) = v n ( T ) + C for all T ⊆ [ n ] \ i , then I P ( v n , S ) = 0 .

Symmetry : I P ( v n ◦ π, S ) = I P ( v n , π ( S )) for every utility function v n ∈ G , every permutation π of [ n ] and every S ⊆ [ n ] .

Consistency : For each utility function v n ∈ G , its extension v n +1 is defined by v n +1 ( S ) = v n ( S ∩ [ n ]) for every S ⊆ [ n +1] . Then, I P ( v n +1 , S ) = I P ( v n , S ) for every S ⊆ [ n ] .

C.2 Interaction Transform

The concept of interaction transform is crucial to establish well-structured matrix representations for each semi-index I P . Specifically, Definitions 3 and 4 are inspired by (Denneberg and Grabisch 1999).

Definition 2 (Möbius Transform) . The Möbius transform µ is a function that maps each utility function v ∈ G n (for every n ≥ 1 ) into another utility function in G n . For each v n ∈ G , we write µ ( v n , S ) for the value of µ ( v n ) indexed at S ⊆ [ n ] . The definition is

Definition 3 (Convolution) . The convolution ι ⋆ u between each ι ∈ R N and each u ∈ G n is defined by

In other words, each ι ∈ R N may be seen as a function that maps each utility function v ∈ G n (for every n ≥ 1 ) into another utility function in G n , and thus we may write ι ( u ) instead of ι ⋆ u .

Definition 4 (Interaction Transform) . For each semi-index I P , its interaction transform is defined to be a vector ι P ∈ R N that satisfies

Proposition 6 (Interaction Transforms for Semi-indices) . For each semi-index I P , its interaction transform ι P ∈ R N is

Here, ι P ( i ) refers to the i -th entry of ι P .

Proof. Let v ∈ G n be given, and for simplicity, we write µ instead for µ ( v ) . For every S ⊆ [ n ] , there is

We refer the reader to (Grabisch et al. 2000, Eq. (10)) for the equality ∆ S v ( T ) = ∑ R ⊆ T µ ( S ∪ R ) used in the second equality.

Since p m k = ∫ 1 0 t k -1 (1 -t ) m -k for 1 ≤ k ≤ m , there is

Therefore, we have

C.3 Matrix Representation for Semi-indices

From now on, let 2 [ n ] be ordered w.r.t. the binary ordering. Then, v n ∈ G is a vector in R 2 n , and so is I P ( v n ) . Since I P is linear in v n ∈ G n , I P has a matrix representation on each G n .

Proposition 7 (Matrix Representations for Semi-indices) . Let 2 [ n ] be ordered using the binary ordering, the matrix representation of semi-index I P on G n w.r.t. the standard basis is

where

Here, ⊗ is the Kronecker product.

Proof. Since I P = ι P ◦ µ , suffice it to find the matrix representations for both µ and ι P w.r.t. the standard basis in G n .

As provided by Grabisch et al. (2000, Eq. (45)), which can be verified by induction using Eq. (13), the matrix representation for the Möbius transform µ on G n w.r.t. the standard basis is

We first prove by induction that the matrix representation of ι P on G n w.r.t. the standard basis is

where

It is clear that it holds for n = 1 . Assume it holds for some k > 0 , i.e., for every u [ k ] ∈ G k

and we proceed with k +1 . Let u [ k +1] ∈ G k +1 and S ⊆ [ k +1] be given, there are two cases: i) S ⊆ [ k ] and ii) S ̸⊆ [ k ] . If S ⊆ [ k ] , by Eq. (19),

The remaining case can be done in a similar fashion. After all, the matrix representation of ι P on G n w.r.t. the standard basis is

where

C.4 A Theorem of Robustness for Semi-indices

Definition 5 (Truncated Semi-indices) . Given a semi-index I P and a non-empty collection of subsets S ⊆ 2 [ n ] , the truncated semi-index I P, S is obtained by only retaining the values of I P indexed by S .

Theorem 2. Under Assumption 1, and further assume σ 12 < min( σ 11 , σ 22 ) , for every non-empty collection of subsets S ⊆ 2 [ n ] , the Dirac delta distribution δ w ∗ with

is optimal to the optimization problem

Proof. The matrix representation of I P on G n w.r.t. the standard basis is M [ n ] P = T [ n ] P × W [ n ] , as presented in Proposition 7. For simplicity, we write V [ n ] ,P = Cov( I P ( v n )) . Since I P is linear in v n , there is

Note that det( T [ n ] P ) = det( W [ n ] ) = 1 as they are triangular matrices with all diagonal entries being one, and thus we have

Observe that Cov( I P, S ( v n )) is just a sub-matrix of V [ n ] ,P . To represent such sub-matrices, for every non-empty α, β ⊆ [2 n ] , V α,β [ n ] ,P is defined to be the sub-matrix of V [ n ] ,P whose rows and columns are indexed by α and β , respectively. Plus, we write V α [ n ] ,P and V : ,α [ n ] ,P instead for V α,α [ n ] ,P and V [2 n ] ,α [ n ] ,P , respectively. Therefore, for each I P, S ( v n ) , there exists a non-empty α ⊆ [2 n ] such that V α [ n ] ,P = Cov( I P, S ( v n )) .

We prove by induction that for every n > 0 , it holds that for every non-empty α ⊆ [2 n ] and for every P ∈ P , det( V α [ n ] ,P ) ≥ det( V α [ n ] ,δ w ∗ ) .

For n = 1 , let s = ∫ 1 0 t d P ( t ) , there is

by which one can verify the statement by noticing that the function s ↦→ σ 11 (1 -s ) 2 + σ 22 s 2 -2 σ 12 s (1 -s ) achieves its minimum at point s = w ∗ . Suppose the statement holds for some k > 0 , and we proceed for k +1 .

Before proceeding with the induction, we have to explore the Kronecker structure between V [ k +1] ,P and V [ k ] ,P . By Eq. (23), and Proposition 7,

Denote L [ k ] P = ∫ 1 0 t B [ k ] ( t )d P ( t ) , Eq. (26) can be rewritten as

where

Our goal is to simplify the expression

Note that E and M [ k ] P are invertible by Eqs. (24) and (25) and that Σ is assumed to be positive definite (and thus det( C [ n ] ) > 0 ). Observe that σ 11 + σ 22 -2 σ 12 > 0 by assumption, using Eqs. (23) and (31), Eq. (29) is equal to

which indicates that

The last equality is obtained by using Eqs. (23) and (31) again. Write R = M [ k ] P -L [ k ] P . From Eqs. (28) and (30), there is

Notice that

Combining Eqs. (32) and (23), we eventually get the simplified formula for S ,

Particularly, σ 11 σ 22 -σ 2 12 > 0 as Σ is assumed to be positive definite. So far so good, we are ready to complete the induction by cases while looking into Eq. (27).

Case 1: If

[2

k

+1

]

\

[2

k

]

Note that we can write

⊆

α

⊆

[2

k

+1

]

, let

β

=

α

\

([2

k

+1

]

\

[2

k

])

, which could be empty. Then, there is

where D β is the corresponding column-selecting matrix. It also gives that

Using Eqs. (33), (34) and (35), according to Lemma 2, there is

where L = det ( ( σ 11 + σ 22 -2 σ 12 ) V [ k ] ,P ) = ( σ 11 + σ 22 -2 σ 12 ) 2 k det( C [ k ] ) by Eq. (25). Therefore, by inductive assumption, for every P ∈ P ,

̸

Case 2: If [2 k +1 ] \ [2 k ] ̸⊆ α ⊆ [2 k +1 ] and α = ∅ , let β = [2 k +1 ] \ [2 k ] and γ = α ∩ β . Note that γ could be empty. By Lemma 5, there is

̸

which implies, for every P ∈ P ,

The second inequality comes from the inductive assumption as well as case 1, while the last equality is due to that V [ k +1] ,δ w ∗ is diagonal according to Lemma 7

C.5 Proof for Theorem 1

Theorem 1. Under Assumption 1, and further assume σ 12 < min( σ 11 , σ 22 ) , the Dirac delta distribution δ w ∗ with

is optimal to the problem (7) for every n ≥ 1 . In addition, the optimal solution is unique if n > 3 .

Proof. Let S = {{ 1 } , { 2 } , . . . , { n }} , then

Therefore, by Theorem 2, δ w ∗ is an optimal solution. For the uniqueness, it does not hold if n ≤ 2 . For example, let

we point out that every P ∈ O is optimal if n = 1 or n = 2 . To see |O| > 1 , for example, if σ 11 = σ 22 and σ 12 = 0 , the probability distribution for the Shapley value lies in O . Nevertheless, we prove the uniqueness as follows: 1) first show that if n = k possesses such a uniqueness, so does n = k +1 , and 2) then demonstrate that it holds for n = 4 .

̸

For the inductive part, it can be completed by reusing the induction in the proof for Theorem 2. Specifically, for case 1, let α = ([2 k +1 ] \ [2 k ]) ∪ β where β contains all indices pointing to subsets { 1 } , { 2 } , . . . , and { k } , then the inequality in Eq. (36) is strict if P = δ w ∗ . On the other hand, for case 2, let α contain all indices pointing to subsets { 1 } , { 2 } , . . . , and { k +1 } , the second inequality in Eq. (37) is also strict if P = δ w ∗ by using the aforementioned strict inequality in case 1.

̸

To show the uniqueness for n = 4 , it can also be done by looking into the proof for Theorem 2. Specifically, in the induction, we focus on case 2 for k = 3 and α = { 2 , 3 , 5 , 9 } that points to subsets { 1 } , { 2 } , { 3 } and { 4 } . Note that β = [2 4 ] \ [2 3 ] = { 9 , 10 , . . . , 16 } in case 2. By Lemma 5, a necessary condition for the first inequality in Eq. (37) to be equal is that each column of V { 2 , 3 , 5 } , { 10 , 11 ,..., 16 } [4] ,P lies in span( V { 2 , 3 , 5 } , { 9 } [4] ,P ) . We prove that this condition can only be met by δ w ∗ , and thus the first inequality in Eq. (36) is strict if P = δ w ∗ .

Regarding Eq. (23), using the fact that

one can verify that V { 2 , 3 , 5 } , { 9 } [4] ,P = c P 1 for some c P ∈ R . Therefore, to meet the necessary condition, all entries in each column of V { 2 , 3 , 5 } , { 10 , 11 ,..., 16 } [4] ,P should be the same. Particularly, there is

where p ( w ) = ( σ 11 + σ 22 -2 σ 12 ) w + σ 12 -σ 11

The equation V { 3 } , { 12 } [4] ,P -V { 5 } , { 12 } [4] ,P = 0 leads to

Set g ( w ) = ( σ 11 σ 22 -σ 2 12 ) p ( w ) 2 , since g ( w ) attains its minimum 0 only at the point w ∗ , by Lemma 6, the necessary condition is met if and only if P = δ w ∗ .

For the ease of the reader to verify the calculation in Theorem 1, we include below a Matlab code to help compute.

C.6 Lemmas

In this section, we provide all lemmas necessary to prove Theorems 2 and 1.

Lemma 1. Suppose A is positive definite and B is positive semi-definite, there is

where the equality holds if and only if B = 0 .

Proof. Since A + B = A 1 2 ( I + A -1 2 BA -1 2 ) A 1 2 , there is det( A + B ) = det( A ) det( I + A -1 2 BA -1 2 ) . By spectral decomposition, A -1 2 BA -1 2 = QDQ ⊤ where QQ ⊤ = I and D is diagonal with all entries being non-negative. Then, det( I + A -1 2 BA -1 2 ) = det( I + D ) ≥ 1 , and thus det( A + B ) ≥ det( A ) .

For the claim on the equality, suppose det( I + A -1 2 BA -1 2 ) = 1 , it leads to D = 0 , and thus B = 0 .

Lemma 2. Given a positive definite matrix M ∈ S ++ that is partitioned as

there is det( M ) = det( A -BC -1 B ⊤ ) det( C ) .

Proof.

Lemma 3. Let M be the matrix given in Lemma 2, then

Moreover, the equality holds if and only if B = 0 .

Proof. By Lemmas 2 and 1, det( M ) = det( A -BC -1 B ⊤ ) det( C ) ≤ det( A ) det( C ) . Notice that the proof for Lemma 2 ensures that A -BC -1 B ⊤ is positive definite.

By Lemma 1, the equality holds if and only if BC -1 B ⊤ = 0 . Note that for a positive definite matrix C -1 , x ⊤ C -1 x = 0 if and only if x = 0 . Therefore, BC -1 B ⊤ = 0 if and only if B = 0 .

Notation For a square matrix A ∈ R m × m , and α, β ⊆ { 1 , 2 , . . . , m } , A [ α, β ] is defined to be the sub-matrix of A by only keeping the rows and columns indexed by α and β , respectively. Plus, A [ α ] is abused for A [ α, α ] .

Lemma 4. Suppose a positive definite matrix M ∈ R m × m is given, for every α ⊆ { 1 , 2 , . . . , m } , there is

The convention is det( M [ ∅ ]) = 1 .

Proof. Without loss of generality, we write M = ( A B B ⊤ C ) such that M [ α ] = C and M [ α c ] = A . By Lemma 2, det( M ) = det( S ) det( C ) where S = A -BC -1 B ⊤ . On the other hand,

Therefore, we have

Lemma 5. For every positive definite matrix M m × m ∈ S ++ , and every α, β ⊆ { 1 , 2 , . . . m } ,

̸

The convention here is det( M [ ∅ ]) = 1 . Moreover, provided that ( α ∪ β ) \ α = ∅ and ( α ∪ β ) \ β = ∅ , and w.l.o.g., write

̸

such that M [ α ] = ( A B B ⊤ D ) and M [ β ] = ( D E E ⊤ F ) , A necessary condition to have the equality is that each column of C lies in span( B ) .

Proof. Our proof is adapted from (Horn and Johnson 2012, Theorem 7.8.9), by which we add a necessary condition for the equality.

By Lemmas 4 and 3, there is

̸

Suppose ( α ∪ β ) \ α = ∅ and ( α ∪ β ) \ β = ∅ , by Lemma 3, the equality holds if and only if M -1 [( α ∪ β ) \ β, ( α ∪ β ) \ α ] = 0 . Using Eq. (38), one can verify that

̸

and thus the equality holds if and only if C = BG H .

Lemma 6. Given a continuous non-negative function g : [0 , 1] → R + that achieves its minimum 0 only at one point w ∗ ∈ [0 , 1] , δ w ∗ is the unique probability measure on [0 , 1] that minimizes

Proof. It is clear that L ( δ w ∗ ) = 0 = inf P ∈P L ( P ) . Suffice it to prove that L ( P ) = 0 implies P = δ w ∗ .

̸

Suppose P ∈ P is given such that L ( P ) = 0 . A point w ∈ [0 , 1] is said to be irrelevant w.r.t. P if there exists ϵ > 0 , P (( w -ϵ, w + ϵ ) ∩ [0 , 1]) = 0 . For the sake of contradiction, suppose a distinct point w ∈ [0 , 1] ( w = w ∗ ) is relevant. Since g ( w ) > g ( w ∗ ) = 0 and g is continuous, there exists ϵ > 0 such that g ( s ) > 0 for all s ∈ [ w -ϵ, w + ϵ ] ∩ [0 , 1] . Since m = inf s ∈ [ w -ϵ,w + ϵ ] ∩ [0 , 1] g ( s ) > 0 and w is relevant, we have

a contradiction. So the only possible relevant point is w ∗ .

Next, we prove that P = δ w ∗ . For each δ > 0 , construct S δ = [0 , w ∗ -δ ] ∪ [ w ∗ + δ, 1] . For each w ∈ S δ , since w is irrelevant, there exists ϵ ( w ) > 0 such that P ( O w ) = 0 where O w = ( w -ϵ ( w ) , w + ϵ ( w )) ∩ [0 , 1] . Then, { O w : w ∈ S δ } forms an open (w.r.t. [0 , 1] ) cover of S δ . Since S δ is compact, there is a finite subcover S δ ⊆ ⋃ 1 ≤ i ≤ k O w i . Therefore, P ( S δ ) ≤ ∑ k i =1 P ( O w i ) = 0 , which implies P ( S δ ) = 0 . By letting δ → 0 , P ( S δ ) → P ([0 , 1] \{ w ∗ } ) = 0 . Using the π -λ theorem, e.g., (Durrett 2019, Theorem A.1.4), one can get P = δ w ∗ .

Lemma 7. Suppose V [ n ] ,P = M [ n ] P C [ n ] ( M [ n ] P ) ⊤ where M [ n ] P is defined in Proposition 7 and C [ n ] = Σ ⊗ Σ ⊗···⊗ Σ ( n repetitions), and assume σ 12 < min( σ 11 , σ 22 ) . Then, the Dirac delta distribution

δ w ∗ where w ∗ = σ 11 -σ 12 σ 11 + σ 22 -2 σ 12 makes the matrix V [ n ] ,P diagonal. In addition, δ w ∗ is the unique probability distribution on [0 , 1] that makes V [ n ] ,P diagonal if n > 1 .

Proof. Since ( A ⊗ B )( C ⊗ D ) = AC ⊗ BD , according to Proposition 7, for each w ∈ [0 , 1] ,

Using Eq. (13) (note 2 [ n ] is ordered w.r.t. the binary ordering), by induction, one can show the ( A,B ) -th entry of V [ n ] ,δ w is p 1 ( w ) | [ n ] -( A ∪ B ) | p 2 ( w ) | ( A \ B ) ∪ ( B \ A ) | C | A ∩ B | for every A,B ⊆ [ n ] . Therefore, for every off-diagonal entry of V [ n ] ,δ w , it must contain the term p 2 ( w ) k where k is some positive integer. Notice that p 2 ( w ∗ ) = 0 , and thus V [ n ] ,δ w ∗ is diagonal.

Conversely, suppose that a probability distribution P on [0 , 1] is given such that V [ n ] ,P is diagonal. Then,

Observe that V [ n ] ,P equals to ∫ 1 0 B [ n ] ( t ) C [ n ] ( B [ n ] ( t )) ⊤ d P ( t ) along the last column as we can prove by induction that the last row of B n ( t ) contains only constants. Since B [ n ] ( t ) C [ n ] ( B [ n ] ( t )) ⊤ can be computed recursively, one can show by induction that V [ n ] ,P ( ∅ , [ n ]) = ∫ 1 0 p 2 ( t ) n d P ( t ) and V n,P ( { 1 } , [ n ]) = ∫ 1 0 Cp 2 ( t ) n -1 d P ( t ) .

Set g ( w ) = p 2 ( w ) n when n is even and g ( w ) = Cp 2 ( w ) n -1 otherwise. Note that g ( w ) ≥ 0 and attains its minimum 0 only at the point w ∗ . By Lemma 6, it implies P = δ w ∗ .

D Experiment Settings

The datasets we use in the main paper are summarized in Table 4. The logistic model used is

In the experiment of noisy label detection, the F1-score is computed by using

which is the harmonic mean of precision and recall.

D.1 Fitting Kronecker Noises

We independently sample 128 sequence ∅ = S 0 ⊊ S 1 ⊊ · · · ⊊ S n = [ n ] in a way that | S i +1 \ S i | = 1 for every 0 ≤ i < n . For each subset S i ( 0 ≤ i ≤ n ) in a sampled sequence, v ( S i ) is computed using 128 random seeds. All these utility evaluations are then employed to generate an estimated covariance matrix ˆ D ∈ R ( n +1) × ( n +1) , which is an estimate for a sub-matrix of the ground-truth covariance matrix of the stochasticity. Note that under Assumption 1, for every 0 ≤ i, j ≤ n ,

D.2 Sampling Lift

Basically, sampling lift, which is designed for all semi-values, refers to any estimators based on

where S is sampled from [ n ] \ i with P ( S ) = p n s +1 .

There are two different ways to sample S . One is derived from Eq. (2). For example, to estimate the value for n ∈ [ n ] , the procedure is i) sample t ∈ [0 , 1] according to the underlying probability distribution P ∈ P ; ii) let X be a Bernoulli random variable such that P ( X = 1) = t , and sample a vector b ∈ R n -1 whose entries are independently following X ; and iii) define S ⊆ [ n ] \ n by letting i ∈ S if and only if b i = 1 . The other way is by noticing that

where q k = p n k · ( n -1 k -1 ) for every 1 ≤ k ≤ n . The corresponding sampling strategy is i) sample k ∼ { q k } 1 ≤ k ≤ n , and then ii) sample S uniformly from 2 [ n ] \ i subject to | S | = k -1 .

We comment that for the first way we do not have to calculate the distribution { q k } 1 ≤ k ≤ n . Note the implementation for { q k } 1 ≤ k ≤ n might suffer from numerical blowup induced by ( n -1 k -1 ) , and it is why Wang and Jia (2023) did not report the results of the Beta Shapley values if n > 500 .

E More Experiment Results

In this section, we provide detailed figures, see Figures 4 to 10, to supplement the results reported in Tables 3 and 2. In each figure, lr stands for the learning rate used for that dataset. In addition, we also run experiments while setting |D tr | = 200 , 10 percent of which is flipped. The corresponding results are presented in Figures 11 to 17.

Ir=0.85

2dplanes

bank-marketing

bioresponse

Spearman's rank correlation coefficient

1.0

Ir=0.01

Spearman's rank correlation coefficient

1.0

Ir=0.01

0.8

0.8

0.6

0.6

0.47

0.2

0.2

0.0

-0.2

0.2

-0.4

-0.4

semi-value

semi-value

semi-value

Ir=0.01

Ir=0.01

Ir=0.85

0.7

0.5

0.6

0.5

0.4

0.5

0.4

0.2

0.2

0.2

0.1

0.1

0.1

0.0

100000000000000000000

0.0

100000000000000000000

41050885050545453525050

SAPPS

semi-value

semi-value

semi-value

covertype

cpu

default

Ir=0.01

1.0

Ir=0.02

Spearman's rank correlation coefficient

1.0

Ir=0.02

0.8

0.8

0.6

0.6

0.6

2

0.4

0.2

0.0

-0.2

0.0

0.0

-0.4

-0.2

-0.2

semi-value

semi-value

semi-value

Ir=0.01

Ir=0.02

Ir=0.02

0.8

0.40

0.5

0.7

0.35

0.4

0.6

0.30

0.2

0.3

0.15

0.2

0.10

0.1

0.1

0.05

0.0

500000000000000000

000

S

0.00

00000000000000000000！

semi-value

semi-value

semi-value

gas

letter

Ir=0.01

Spearman'srank correlation coefficient

1.0

Ir=0.02

Spearman'srank correlationcoefficient

1.0

Ir=0.09

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0.0

0.0

-0.2

-0.2

semi-value

semi-value

semi-value

Ir=0.02

Ir=0.01

Ir=0.09

0.8

0.6

0.7

0.8

0.5

0.6

0.6

0.4

十

0.3

0.2

0.2

0.2

0.1

0.1

0.0

100000000000000000000

0.0

100000000000000000000

0.0

100000000000000000000

5

semi-value

semi-value

semi-value

optdigits

pendigits

phoneme

Spearman's rank correlation coefficient

1.0

Ir=0.03

Spearman's rank correlation coefficient

1.0

Ir=0.14

Spearman's rank correlation coefficient

1.0

Ir=0.01

0.8

0.6

0.4

0.4

0.2

0.2

0.2

0.0

0.0

0.0

-0.2

-0.2

-0.2

semi-value

semi-value

semi-value

Ir=0.03

Ir=0.14

Ir=0.01

0.8

0.8

0.5

0.7

0.6

0.6

0.4

0.3

0.2

0.2

0.2

0.1

0.1

0.0

000

S

000

semi-value

semi-value

semi-value

pol

satimage

segment

1.0

Ir=0.03

Spearman's rank correlation coeficient

Ir=0.01

1.0

Ir=0.07

0.8

0.8

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0.0

0.0

-0.2

-0.2

-0.2

0.4

semi-value

semi-value

semi-value

Ir=0.03

Ir=0.01

Ir=0.07

0.7

0.8

0.6

0.7

0.6

0.5

0.6

0.5

0.3

0.2

0.2

0.2

0.1

0.1

0.1

000

S

0.0

50000000000000000

000

semi-value

semi-value

semi-value

spambase

texture

wind

Spearman's rank correlation coefficient

1.0

Ir=0.03

Spearman's rank correlation coefficient

1.0

Ir=0.07

Spearman's rank correlation coefficient

1.0

Ir=0.02

0.8

0.6

0.6

0.6

0.4

0.4

0.2

0.2

0.0

0.0

-0.2

-0.2

-0.4

-0.2

semi-value

semi-value

semi-value

Ir=0.03

Ir=0.07

Ir=0.02

0.8

0.7

0.8

0.7

0.7

0.6

0.6

0.6

0.5

0.3

0.3

0.2

0.2

0.2

0.1

0.1

0.1

00

0.0

500000000000000o0

000

0.0

00000000000000000

0000

semi-value

semi-value

semi-value

MNIST

FMNIST

Ir=0.01

Spearman's rank correlation coeficient

Ir=0.02

0.8

0.8

0.6

0.4

0.4

0.2

0.2

-0.2

semi-value

semi-value

Ir=0.01

Ir=0.02

0.7

0.6

0.5

0.5

0.4

0.2

0.2

0.1

0.1

S

semi-value

semi-value

2dplanes

bank-marketing

bioresponse

Spearman'srank correlation coefficient

Ir=0.06

Spearman's rank correlation coeficient

Ir=0.01

Spearman's rank correlation coefficient

Ir=0.5

1.0

1.0

0.8

0.8

0.8

0.6

0.6

0.4

0.2

0.4

0.2

0.0

0.0

-0.2

0.0

-0.4

0.4

semi-value

semi-value

semi-value

Ir=0.06

Ir=0.01

Ir=0.5

0.40

0.5

0.35

0.5

0.4

0.30

0.2

0.15

0.2

0.10

0.1

0.1

0.05

semi-value

410558505055543352051050

semi-value

000000

0.00

00000000000000000000

semi-value

covertype

cpu

credit

Ir=0.02

Spearman's rank correlation coefficient

1.0

Ir=0.06

Spearman's rank correlation coefficient

1.0

Ir=0.02

0.8

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0.0

0.0

0.2

0.4

0.2

-0.2

semi-value

semi-value

semi-value

Ir=0.02

Ir=0.06

Ir=0.02

0.5

0.8

0.40

0.7

0.35

0.4

0.6

0.30

0.2

0.3

0.15

0.2

0.10

0.1

0.1

0.05

0.0

0.0

00000000000000000000

410558505055403352051050

0.00

000000000000000000000

semi-value

semi-value

semi-value

default

diabetes

fraud

Spearman's rank correlation coefficient

1.0

Ir=0.53

Spearman's rank correlation coeficient

1.0

Ir=0.02

Ir=0.05

高

0.8

0.8

0.6

0.4

0.4

0.4

.2

0.2

0.2

0.0

0.0

0.0

-0.2

0.4

-0.2

semi-value

semi-value

semi-value

Ir=0.53

Ir=0.02

Ir=0.05

0.40

0.5

0.7

0.35

0.6

0.30

0.4

0.5

0.3

0.15

0.2

0.10

0.2

0.1

0.05

0.1

4105958505055545352510150

000

semi-value

semi-value

semi-value

gas

letter

Spearman's rankcorrelation coefficient

1.0

Ir=0.02

Spearman's rank correlation coefficient

1.0

Ir=0.01

Spearman's rankcorrelation coefficient

1.0

Ir=0.17

0.8

0.8

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.0

0.0

-0.2

-0.2

-0.4

-0.4

semi-value

semi-value

semi-value

Ir=0.02

Ir=0.01

Ir=0.17

0.5

0.6

0.7

0.5

0.6

0.4

0.5

0.2

0.2

0.2

0.1

0.1

0.1

41059585050555405352050150

S

00000

semi-value

semi-value

semi-value

optdigits

pendigits

phoneme

Spearman's rank correlation coefficient

Ir=0.05

Spearman's rank correlation coeficient

1.0

Ir=0.11

Spearman's rank correlation coefficient

1.0

Ir=0.02

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0.07

0.0

-0.2:

0.0

-0.2

0.4

-0.2

semi-value

semi-value

semi-value

Ir=0.05

Ir=0.11

Ir=0.02

0.7

0.8

0.4

0.6

0.5

0.6

0.3

0.2

0.2

0.1

0.1

4105958505055405352050150

000

semi-value

semi-value

semi-value

pol

satimage

segment

Spearman's rank correlation coefficient

1.0

Ir=0.11

Spearman's rank correlation coefficient

1.0

Ir=0.04

1.0

Ir=0.1

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0.0

0.0

-0.2

-0.2

-0.2

semi-value

semi-value

semi-value

Ir=0.11

Ir=0.04

Ir=0.1

0.7

0.5

0.6

0.8

0.4

0.5

0.6

0.2

0.1

0.2

0.1

0.0

0000

semi-value

semi-value

semi-value

spambase

texture

wind

Spearman's rank correlation coefficient

Ir=0.02

Spearman's rank correlation coefficient

1.0

Ir=0.08

Spearman's rank correlation coefficient

1.0

Ir=0.04

0.8

0.6

0.4

0.4

0.2

0.2

0.2

0.0

0.0

0.0

-0.2

-0.2

-0.2

semi-value

semi-value

semi-value

Ir=0.02

Ir=0.08

Ir=0.04

0.7

0.7

0.8

0.6+

0.7

0.6

0.6

0.5

0.5

0.3

0.2

0.2

0.2

0.1

0.1

0.1

0.0

000

868888885

00000

semi-value

semi-value

semi-value

Extra References

Anguita, D., A. Ghio, L. Oneto, X. Parra, J. L. Reyes-Ortiz, et al. (2013). 'A Public Domain Dataset for Human Activity Recognition Using Smartphones'. In: Esann . Vol. 3, p. 3.

Dal Pozzolo, A., O. Caelen, Y.-A. Le Borgne, S. Waterschoot, and G. Bontempi (2014). 'Learned Lessons in Credit Card Fraud Detection From a Practitioner Perspective'. Expert Systems with Applications , vol. 41, no. 10, pp. 4915-4928.

Denneberg, D. and M. Grabisch (1999). 'Interaction Transform of Set Functions Over a Finite Set'. Information Sciences , vol. 121, no. 1-2, pp. 149-170.

Durrett, R. (2019). 'Probability: Theory and Examples'. Vol. 49. Cambridge University Press.

Frey, P. W. and D. J. Slate (1991). 'Letter Recognition Using Holland-Style Adaptive Classifiers'. Machine Learning , vol. 6, pp. 161-182.

Fujimoto, K., I. Kojadinovic, and J.-L. Marichal (2006). 'Axiomatic Characterizations of Probabilistic and Cardinal-Probabilistic Interaction Indices'. Games and Economic Behavior , vol. 55, no. 1, pp. 72-99.

Grabisch, M., J.-L. Marichal, and M. Roubens (2000). 'Equivalent Representations of Set Functions'. Mathematics of Operations Research , vol. 25, no. 2, pp. 157-178.

Grabisch, M. and M. Roubens (1999). 'An Axiomatic Approach to the Concept of Interaction Among Players in Cooperative Games'. International Journal of Game Theory , vol. 28, no. 4, pp. 547-565.

Horn, R. A. and C. R. Johnson (2012). 'Matrix Analysis'. Cambridge University Press.

Moro, S., R. Laureano, and P. Cortez (2011). 'Using Data Mining for Bank Direct Marketing: An Application of the Crisp-Dm Methodology'.

Sundararajan, M., K. Dhamdhere, and A. Agarwal (2020). 'The Shapley Taylor Interaction Index'. In: International Conference on Machine Learning . PMLR, pp. 9259-9268.

Vergara, A., S. Vembu, T. Ayhan, M. A. Ryan, M. L. Homer, and R. Huerta (2012). 'Chemical Gas Sensor Drift Compensation Using Classifier Ensembles'. Sensors and Actuators B: Chemical , vol. 166, pp. 320-329.

Xiao, H., K. Rasul, and R. Vollgraf (2017). 'Fashion-Mnist: A Novel Image Dataset for Benchmarking Machine Learning Algorithms'. arXiv preprint arXiv:1708.07747 .

Yeh, I.-C. and C.-h. Lien (2009). 'The Comparisons of Data Mining Techniques for the Predictive Accuracy of Probability of Default of Credit Card Clients'. Expert Systems with Applications , vol. 36, no. 2, pp. 24732480.

Zhang, H., Y. Xie, L. Zheng, D. Zhang, and Q. Zhang (2021). 'Interpreting Multivariate Shapley Interactions in DNNs'. In: Proceedings of the AAAI Conference on Artificial Intelligence . Vol. 35. 12, pp. 10877-10886.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes that 1) the semi-values obtained from fitting Kronecker noise to the data robustly rank data more consistently across runs and 2) we can efficiently estimate these weighted Banzhaf values with the maximum sample reuse principle. The extensive experiments illustrated the advantages of the contributions: higher sample efficiency and performance in noisy label detection and rank consistency.\n\nSTRENGTHS:\n1. It is novel to fit noise to the data to determine the semivalue weights and important to expand beyond and compare against common data valuation methods (Shapley and Banzhaf) \n2. Extensive experiments on many datasets were performed.\n\nWEAKNESSES:\nThe definition of robustness (and especially Sec 3.2), assumptions and limitations of the work are not sufficiently and clearly explained. More details are given in the questions and suggestions below. Incorporating the suggestions will improve the contextualization relative to prior work and make the claims more convincing.\n\nQUESTIONS:\nQ1. Is the definition of robustness in this paper the same as Data Banzhaf?\n\nQ2. Does the estimation according to Line 134/equation 1 get significantly less inaccurate as $w$ tends to 0 or 1? One of the terms will have much fewer samples.\n\nQ3. Does Kronecker Noise mean Gaussian noise with Kronecker covariance?\nWhat is an intuitive interpretation of each entry in $X_i$ and $\\Sigma$? How is $X_i$ related to $v$? Provide further explanation/citations on the suitability of Kronecker noise (less parameters to fit?) and why $\\sigma_{22} < \\sigma_{11}$.\n\nMy guess is that $X_i \\sim \\mathcal{N}(0, \\Sigma =\\begin{pmatrix} \\sigma_{11}\\  \\sigma_{12}\\\\\\\\ \\sigma_{21}\\   \\sigma_{22} \\end{pmatrix})$ . Instead of observing the utility function $\\bar{u} = \\begin{pmatrix} u(\\emptyset) \\\\\\\\ u(i) \\end{pmatrix}$ directly, $\\bar{u} + X_i$ is observed.\n\nIn equation 5, what is $\\phi_P$ and why is its outer product $\\Sigma$?\n\nQ4. Can you explain why Theorem 1/Remark 1 contradict Data Banzhaf? Is it due to a different notion of robustness / assumption of noise structure? Data Banzhaf approach is noise-structure agnostic as they consider users may have difficulties estimating the noise.\n\nQ5. Line 229-231 mentions that exact values are intractable on larger datasets and approximation would “draw the real noises away from following Kronecker noises”. What does the quoted part mean? Is this a problem with using Kronecker noise that limits the applicability in real use cases?\n\n**Suggestions**\n* The robustness concept should be better introduced in the introduction’s 3rd paragraph. Later, a mathematical/formal definition of the robustness concept should be included.  \nFor example, the Data Banzhaf paper explains that the stochasticity in the rankings/utilities is due to stochastic training methods. Such inconsistencies may interfere with identifying the usefulness of data. The Data Banzhaf paper gives an in-depth definition of Robustness in Sec 4. These details are missing from this paper.\n\n* The “weighted Banzhaf” and “consistency” concepts introduced here have alternative names in CGT literature: binomial semivalues and null player exclusion/hereditary property. See Domenech, M., Giménez, J. M., & Puente, M. A. (2016). Some properties for probabilistic and multinomial (probabilistic) values on cooperative games. Optimization, 65(7), 1377-1395.\n\n* In the experiments, Shapley and the Beta Shapley values are underperforming as they place large weights on the smaller/larger coalitions. I would expect Beta$(k, k)$ for large $k$ or Beta$(kp^*, k(1-p^*))$ where p* s the robust Banzhaf version to perform as well.\nThus, an alternative conclusion (instead of using binomial semivalues) is that the semivalue weights should be learnt/adaptive.\n\n* It might be possible to empirically verify that in real dataset/SGD training, training on more data leads to more or less noise in the utilities.\n\n\n\n\nMinor suggestions (not affecting score)\n* coeffient misspelling in Figure 1 caption\n* the axis tick labels in all figures are too small\n* in equation 1, s is not defined to be the size of S\n\nLIMITATIONS:\nThe limitation mentioned is that there is no universally best and most robust value for all datasets. However, other limitations (e.g. the Kronecker noise assumption or complexity) should be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends the notion of Banzhaf values to weighted Banzhaf values to improve the robustness of data ranking. The authors show under Kronecker noises, when minimizing the worse-case entropy, the most robust parameters belong to the family of weighted Banzhaf values. Similarly, as implemented in Data Banzhaf, authors use a maximum sample reuse strategy to improve sampling efficiency, as shown by juxtaposing it with other sampling techniques. The authors demonstrate the robustness of weighted Banzhaf values by comparing them with other data valuation methods over multiple datasets. Weighted Banzhaf values produce consistent data ranking. Further, the performance is presented through noisy label detection experiments.\n\nSTRENGTHS:\n+ The authors show the exact case when Data Banzhaf achieves highest robustness and cases when weighted Banzhaf values can achieve better robustness.\n\n+ The weighted Banzhaf values are successfully generalized from Data Banzhaf and demonstrates largest safe margin among compared data valuation methods.\n\nWEAKNESSES:\n+ All utility functions are set to be accuracy of simple models. Specifically, only simple models are considered, such as logistic regression models or LeNet.\n\n+ Only simple tabular and image datasets are considered.\n\n+ The sample size is also minimal, 2000 samples at most.\n\nIt seems that even though efficient sampling is employed, the current data valuation method cannot be applied in more practical settings, larger models, datasets, and sample sizes, which is due to the intractable computation of model re-trainings.\n\nQUESTIONS:\n+ How can you ensure exact Shapley value, if each evaluation is noisy?\n\n+ Since weighted Banzhaf results are based on the best Banzhaf weights, can you also provide results for the best parameters for Beta Shapley?\n\n+ What is the time complexity and actual runtime for weighted Banzhaf value at each step?\n\n+ It would be insightful to compare weighted Banzhaf values with KNN Shapley, which does not require model training. Is the data ranking, in that case, inherently robust to noise?\n\n\n+ Typo Line 242: descent\n+ Typo Line 266: flipped\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on data valuation with weighted Banzhaf values, which seems to be effective, particularly in cases where the dataset or the data valuation process is noisy. Toward that, the authors introduce and utilize a Kronecker noise model to calculate robust values*  and moreover to do it in an efficient way utilizing the maximum sample reuse principle. They show that the weighted Banzhaf value with the Kronecker noise model is optimal in minimizing worst-case entropy. They performed experiments to show the efficiency of the method with maximum sample reuse, validate their theoretical finding on optimality (to a certain extend), and evaluate the performance of weighted Banzhaf in capturing noisy labels and data ranking. Their results indicate that weighted Banzhaf shows consistently good performance across different tasks and datasets.\n\n*following the calculation of semi-values used for averaging contribution over subsets\n\nSTRENGTHS:\n**Clarity:** Overall, this paper is quite well written with clear expectations for the reader throughout. The logical flow of the paper is well-structured, although there were a few intuitive aspects that I found lacking. Nonetheless, I believe that the presentation of the work is solid.\n\n**Originality:** The concept of the Kronecker noise model, and moreover utilization of it for learning semi-values and evaluating noisy datasets are original, as far as my knowledge extends. \n\n**Quality:** I think this paper meets the quality standards starting from the introduction of the noise model and weighted version of Banzhaf indexing, to the argumentation of their optimality, and the extensive experimental evaluations.\n\n**Significance:** I believe it is crucial to align game-theoretical data valuation approaches and compare their strengths and weaknesses across different scenarios, including noisy settings. As emphasized by the authors, no universal approach can really be deemed optimal for all datasets and noise settings. However, the flexibility offered by weighted Banzhaf (or data-driven? semi-values) is promising, opening up avenues for future research in understanding the optimality of data-valuation approaches across different regimes. I find the contribution of this work to be significant in this regard.\n\nWEAKNESSES:\nI believe there is room for improvement regarding the justification of the Kronecker noise model and its implications for data valuation. It would be helpful to provide further explanations and clarifications on the following points:\n- Intuition behind the Kronecker model and the scenarios where it is (not) applicable.\n- In addition to discussing the universality of an approach or lack thereof, I believe it is important to draw conclusions or insights from the comparisons. For example, for a reader like me, it is not immediately clear why the proposed method is challenged by other approaches in particular in Table 2, and how this performance comparison is affected by the intrinsic dynamics in the data (such as higher-order interactions, which is generally captured best by Shapley value).\n\nQUESTIONS:\nThe points presented under the Weaknesses above can be considered as my questions here. I have the following additional questions:\n- In Table 3, are some portions of labels still flipped? My understanding was that they are not. If it's indeed so, I am curious to see F1 scores in addition to the correlation coefficients. Clarification would be appreciated.\n- How does the Kronecker noise model behave for large amounts of classes in data?\n- Also, the reproducibility is checked as n/a but I think it applies here.\n- Do you think overfitting may be an issue with this approach as semi-values are learned from data in the end?\n\nLIMITATIONS:\nAs the authors also addressed in the supplementary material, the concept of the Kronecker noise model poses certain limitations where it does not align well with the actual data noise and is empirically justified only on small datasets. I have not observed an additional limitation beyond this.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper looks at the standard data valuation problem, in case of noisy estimation of the value of a coalition. It proposes a model of noise, Kronecker noise, and shows that under this noise a weighted Banzhaf value (with weight that depends on parameters of the noise) is semi-value that maximizes a notion of robustness. If the weight is 0.5, it is equivalent to the Banzhaf value, but it is different otherwise. The paper shows a number of experiments that illustrate this result and the efficiency of computing the weighted Banzhaf value. \n\nNote: There is a very related paper from Wang and Jia at AISTATS '23 that proposes the Banzhaf value and shows it is the most robust semi-value under a certain definition of robustness (and also proposed an efficient estimator based on the so-called maximum sample reused principle); so this submission's novelty and significance is obviously to be judged with respect to that AISTATS '23 paper.\n\nSTRENGTHS:\nThe data valuation problem is a hot topic and moving away from the Shapley value adds interesting insights to the existing literature. In particular, considering robustness to noisy evaluation of the value function (as the paper does) is interesting. \n\nIn my opinion, the main contribution of the paper is the result of Theorem 1 that states the following: If the noise is Kronecker noise (as defined in Def 1), then the semi-value that minimizes the determinant of the semi-values covariance corresponds to a weighted Banzhaf value with weight given by (6), which depends on individual parameters of the noise. \n- To me this results is more of the type of identifying a particular kind of noise and a particular measure of robustness such that weighted Banzhaf maximizes robustness (rather than having a naturally relevant notion of noise and then showing the result), but this is already quite interesting. The notion of Kronecker noise used is somewhat general and flexible. \n- The proof is non-trivial (and very long). I was not able to follow all details. I wish the paper had included a sketch of proof to give meaningful intuition. \n- I am wondering whether this result is really specific to the data valuation problem and even if data valuation is the most relevant application. Is there any other that one could think of? \n\nThe paper shows the application of this result to the data valuation problem based on extensive experiments on synthetic and real datasets (although the real data sets are still somewhat forced to fit the Kronecker noise framework, see below). \n\nThe paper also contains another result, Proposition 2, which shows that the maximum sample reuse principle can be used to efficiently estimate the weighted Banzhaf value, but this is very incremental compared to [Wang and Jia, AISTATS '23].\n\nWEAKNESSES:\nI did not find the comparison to [Wang and Jia, AISTATS '23] to be perfectly clear in the paper. As the paper mentions, [Wang and Jia, AISTATS '23] have a result that states that Banzhaf maximizes robustness amongst all semi-values, which includes weighted Banzhaf. How can it be reconciled with the result of this paper that shows that in some cases a non-0.5 weighted Banzhaf is more robust? I do not believe that [Wang and Jia, AISTATS '23] assume isotopic noise (correct me if I am wrong). Is that not rather due to a different definition of robustness? It would be better if the paper was clearer about that. \n\nThe noise model introduced lacks motivation. Specifically, the noise is applied directly on the value function, but there is no attempt made to link this noise model to noise in the data. In fact, l. 169, the paper says that the covariance of $v(S)$ for a subset $S$ of size $s$ decreases like $1/C^{s}$ with $C=\\sigma_{11}/\\sigma_{22}>1$ if $\\sigma_{11} > \\sigma_{22}$. This brings two questions: \n- why is $\\sigma_{11} > \\sigma_{22}$ linked to a decreasing variance with the size of the subset? \n- why is this a good model? Sure, we expect that the variance would decrease, but no as fast as $1/C^{s}$, rather something classical like $1/\\sqrt{s}$. All that is to say that it is not clear (and even less justified in the paper) that the Kronecker noise model is relevant for the data valuation application. \n\nThe paper also lacks motivation for the specific robustness measure used (the determinant of the covariance of the values). There is a somewhat vague paragraph that explains that gaussian distributions maximize entropy for fixed covariance but why entropy and why fix covariance? Also, is it possible that the values are gaussian?\n(Side note: in this paragraph, the covariance $\\Sigma$ is generic whereas earlier it is specifically the individual one for Kronecker noise, that is confusing.)\n\nSome parts of the paper are a bit fast and unclear. For instance Proposition 2 and the comment below does not really explain what the convergence is, what the sample lift strategy is and why it is so bad here. This is not too crucial since this result is anecdotical but is frustrating for the reader.\n\nQUESTIONS:\n- In the experimental section: my understanding is that the paper fits the covariance matrix of the Kronecker noise to the data and then adds noise to subset with this fitted matrix. Is that correct? I wonder: is the fit good at all? This is related to my earlier comment that I do not feel that Kronecker noise is necessarily a good model here (e.g., because of the too fast decrease of the variance, see above). \n\n- More generally, in many places I was not able to know what the value fonction used is how exactly noise is added in the experiments. Maybe I missed the information?\n\n- The introduction mentions robustness to human errors and attacks. I do not believe that what is proposed here can reasonably be robust to attacks, at least not to worst-case unconstrained attacks. Can the authors comment on the robustness to attacks? \n\n- Understanding Fig 1 is basically not possible because it is not written what the value is and what noise is added. Even after reading the experimental section I was not sure what is plotted on Fig 1. \n\n- Minor: In Remark 1: \\sigma_11 = \\sigma_22 is more general than isotropic.\n\nLIMITATIONS:\nLimitations are discussed in Appendix E, which is kind of hidden after 14 pages of proofs and is not referenced in the text. It'd be better to at least put a pointer in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}