Title: Faithful Group Shapley Value
Venue: NeurIPS 2025
Accepted: yes
Keywords: Shapley value

data valuation

group evaluation
Abstract: Data Shapley is an important tool for data valuation, which quantifies the contribution of individual data points to machine learning models. 
 In practice, group-level data valuation is desirable when data providers contribute data in batch.  However, we identify that existing group-level extensions of Data Shapley are vulnerable to \emph{shell company attacks}, where strategic group splitting can unfairly inflate valuations.  We propose Faithful Group Shapley Value (FGSV) that uniquely defends against such attacks.  Building on original mathematical insights, we develop a provably fast and accurate approximation algorithm for computing FGSV.  Empirical experiments demonstrate that our algorithm significantly outperforms state-of-the-art methods in computational efficiency and approximation accuracy, while ensuring faithful group-level valuation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of group data valuation and proposes axioms that a _faithful_ group data valuation method should satisfy. In particular, for faithfulness, the total valuation of the same group of data should remain unchanged, regardless of how others are partitioned. The paper suggests the sum of individual Shapley values (i.e., the Faithful Group Shapley Value (FGSV)) uniquely satisfies the required axioms.\nNext, based on the observation that a small subset of terms dominate in FGSV’s formula, the paper proposes a more efficient approximation for individual FGSV. \nLastly, the paper empirically evaluate the approximation accuracy and computation time of FGSV and other baseline methods and compare FGSV against GSV on copyright attribution in generative AI application and explainable AI applications.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. It is significant and important to design group data valuation that satisfy the faithfulness axiom and is resistant to the splitting and shell company attacks. While summing the individual Shapley values seem like a natural solution, the paper also proposes a novel technique to compute the FGSV more directly and efficiently.\n\n### Weaknesses\n1. The clarity of the paper can be improved.\n    * A few claims need some clarifications (see questions below)\n    * Some notations are unexplained, e.g., $\\asymp$ in Theorem 3, $\\mathcal{P}\\^s$ on line 77, $\\mathcal{Z}\\^s$ in Assumption 1 and 2. Is  $\\mathcal{P}^s$ the uniform distribution over all subsets of size $s$?\n    * It would be helpful to provide the intuitive version and justification of the prudence condition (Eq 3) and axioms in Sec 3 so that the reader would not have to infer them. It is also helpful to provide some intuition about the proof of the proposition and the theorem in the main paper.\n2. There are some concerns about the experiment settings and whether it demonstrates FGSV better efficiency and approximation quality over other individual SV estimation methods (see questions below)\n\n### Minor Comments\n* Using $s_1$ instead of $s_1/s$ in the definition of $\\mu$ (Eq. 6) may be cleaner. $\\mu$ depends on the set $S_0$ not just the size $s_0$.\n* For the copier/replication attack, see Han, D., Wooldridge, M., Rogers, A., Ohrimenko, O., & Tschiatschek, S. (2022). Replication robust payoff allocation in submodular cooperative games. IEEE Transactions on Artificial Intelligence, 4(5), 1114-1128.\n\nQUESTIONS:\n1. What is the justification for the weaker null player and symmetry axioms in Sec 3 vs their GaI counterparts? Why is this weakening needed or acceptable?\n2. In line 193-6, it is stated that SOTA individual data values require quadratic number of samples to achieve the same error tolerance. However, the quadratic number of samples (or more precisely linear multiplied by the maximum group size) can be reused across all groups $S_0, \\dots, S_k$. Algorithm 1 requires linear number of samples to achieve the error tolerance but it is only for a specific group $S_0$. With more groups (maximum $n$), the total number of samples becomes quadratic as well. Is this understanding correct? If so, is the new approximation less useful and efficient in some cases with more groups?\n3. What is the issue with small input sets (Sec 3.4)? For smaller datasets, the utility might be small but that does not mean it is meaningless. Do you mean that the variance due to the algorithm randomness is high?\n4. In Sec 4.1, it is good that the total budget of utility evaluations is fixed at 20k. However, comparing the performance and time taken based on the iteration of each method seems unfair as _different methods use different number of utility evaluations per iteration_ (e.g., permutation vs single utility). Theorem 3 suggests a comparison based on the same number of utility evaluations instead. \n    * Can you compare based on the number of utility evaluations (plot of AUCC/MSE against number of utility evaluations)? \n    * Can you report the total runtime instead? Is the runtime for a single FGSV or all $n$ FGSVs?\n\nLIMITATIONS:\nThe authors only discussed another unaddressed attack strategy in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose the Faithful Group Shapley Value (FGSV) to ensure data valuation is robust to the shell company attack where different data groups (companies) strategically split their data into multiple groups to increase their value.\nThey define FGSV for a group as the sum of individual Data Shapley values of its members and prove that it uniquely satisfies the set of axioms desirable for faithful group data valuation.\nThey empirically test the proposed approach on synthetic data,  FlickrLogo-2 dataset, and diabetes dataset.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The authors try to address a key issue in data markets by using a commonly used method as a baseline. \n- Figure 1 is intuitive and clearly illustrates the problem the authors are trying to address. \n- Authors theoretically prove the existence of a fast and accurate approximation algorithm for FGSV, empirically test their approach, and avail their code in supplementary material.\n\n\n**Weaknesses**\n\n- Section 3.2 (Algorithm 1) in the main paper  and in Appendix C.\n   - None of the references in Algorithm 1 take you to the actual equation.\n   - How do you determine m1 and m2? From B.4, it's 1000>n, and in application B.2, it's 20,000, and in B.1, it's 2000. Is it the same across runs for each application?\n   - What is non-informative distribution Pnull, and how does it relate to D?\n   - In general, I think that adding a paragraph describing the algorithm(s) and or comments in the algorithm(s) would improve readability. \n\n\n- Section 3.4 (Utility function values for small input sets).\n   - Theorem 3 appears to guide whether the size of s is appropriate; however, it is still unclear to me how B is determined. It's also unclear to me the extent to which the \"non-informative data points\" affect the data valuation of a group's data. How do authors define non-informative data points? Where do they come from, and are they drawn from the same distribution?\n   - Lastly, more theoretical analysis of the implications of this padding process would improve trust in the efficacy of the faithful data valuation approach. \n\n   - The Banzhaf value is replication-robust, reliable across runs, and can handle varied sizes. Why is not used a base method instead of Shapley? \n\n\n\n- Experiments (Section 4).\n   - Both the experimental setups and the results are generally difficult to follow. For instance, in Figure 2, it is unclear which dataset was used, what the B or \\tilde{s} is, what the size of each data group(S) is, and how the dataset was sourced or generated. Many crucial details are missing, with only a few relegated to Appendix B.1. There is also a mismatch between the motivation, e.g., copyright or explainable AI, and the setup and results.\n   - The authors mention the test set for just one experiment in Section B.4 but provide no information on the data splits used in the other experiments. It remains unclear how they sourced the held-out test sets and how similar or different they are in distribution compared to the various groups used in the experiments.\n\n\n- Related works. \n   - There are several replication robust \"base\" methods. For example https://openreview.net/pdf?id=iTjSqQQ4f8, https://proceedings.mlr.press/v206/wang23e/wang23e.pdf, https://arxiv.org/pdf/2006.14583\n   - I think the related works should ideally cite papers in this line of inquiry. If possible, use these kinds of methods in the experimental comparative analysis.\n\n- Motivation and setup.\n   - There are some disconnects between the motivation and the setup. For example, drawing held-out tests and the different groups from the same distribution might not sufficiently depict the real-world setting the authors aim to address. \n   - The application of the approach to faithful explainable AI (section 4.3) might exacerbate fairness issues and further marginalize already marginalized groups. It's also unclear to me, based on the results in Section 4.3, if FGSV improves things. \n\n\n- Typos and reference issues.\n   - There are several typos. For example,  the on the reference made on line 214.  In appendix C, there is only Algorithm 1, not Algorithm 2\n   - Most of the references in the paper don't work as expected. E.g., if you click on some hyperlinked references, they take you to page one. For example, none of the references in algorithm 1 take you to the actual equation.\n\nQUESTIONS:\n- Section 3.2 (Algorithm 1) in the main paper and in Appendix C.\n   - None of the references in Algorithm 1 take you to the actual equation.\n   - How do you determine m1 and m2? From B.4, it's 1000>n, and in application B.2, it's 20,000, and in B.1, it's 2000. Is it the same across runs for each application?\n   - What is non-informative distribution Pnull, and how does it relate to D?\n   - In general, I think that adding a paragraph describing the algorithm(s) and or comments in the algorithm(s) would improve readability. \n\n\n- Section 3.4 (Utility function values for small input sets).\n   - Theorem 3 appears to guide whether the size of s is appropriate; however, it is still unclear to me how B is determined. It's also unclear to me the extent to which the \"non-informative data points\" affect the data valuation of a group's data. How do authors define non-informative data points? Where do they come from, and are they drawn from the same distribution?\n   - Lastly, more theoretical analysis of the implications of this padding process would improve trust in the efficacy of the faithful data valuation approach. \n\n   - The Banzhaf value is replication-robust, reliable across runs, and can handle varied sizes. Why is not used a base method instead of Shapley? \n\n\n\n- Experiments (Section 4).\n   - Both the experimental setups and the results are generally difficult to follow. For instance, in Figure 2, it is unclear which dataset was used, what the B or \\tilde{s} is, what the size of each data group(S) is, and how the dataset was sourced or generated. Many crucial details are missing, with only a few relegated to Appendix B.1. There is also a mismatch between the motivation, e.g., copyright or explainable AI, and the setup and results.\n   - The authors mention the test set for just one experiment in Section B.4 but provide no information on the data splits used in the other experiments. It remains unclear how they sourced the held-out test sets and how similar or different they are in distribution compared to the various groups used in the experiments.\n\n\n- Related works. \n   - There are several replication robust \"base\" methods. For example https://openreview.net/pdf?id=iTjSqQQ4f8, https://proceedings.mlr.press/v206/wang23e/wang23e.pdf, https://arxiv.org/pdf/2006.14583\n   - I think the related works should ideally cite papers in this line of inquiry. If possible, use these kinds of methods in the experimental comparative analysis.\n\n- Motivation and setup.\n   - There are some disconnects between the motivation and the setup. For example, drawing held-out tests and the different groups from the same distribution might not sufficiently depict the real-world setting the authors aim to address. \n   - The application of the approach to faithful explainable AI (section 4.3) might exacerbate fairness issues and further marginalize already marginalized groups. It's also unclear to me, based on the results in Section 4.3, if FGSV improves things. \n\n\n- Typos and reference issues.\n   - There are several typos. For example,  the on the reference made on line 214.  In appendix C, there is only Algorithm 1, not Algorithm 2\n   - Most of the references in the paper don't work as expected. E.g., if you click on some hyperlinked references, they take you to page one. For example, none of the references in algorithm 1 take you to the actual equation.\n\nLIMITATIONS:\nThe paper lacks a (detailed) limitation section", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines the usage of group level data valuation based on Shapley value. \nThe author makes this observation:  a malicious agent can game a mechanism that treats the groups as atomic entities,  using the so-called *shell company attack*, in which the malicious agent strategically splits its data between several shell companies.\n\nThe paper first result is that, to be robust to shell company attacks and satisfy the classical Shapley value axioms -- where anonymity has been adapted to the group setting--  one has no choice but to value a group of point as the sum of each point Shapley value. \n\nThis motivates the introduction of a Shapley value algorithm exploiting the group structure, in particular,  \none can group the $U(S)$ terms in the Shapley value computation based on the size of |S| and the size of the  intersection  of $S$ with the group of interest (the one we want to compute the Shapley value).\nThis observation initiates a series of tricks that ends up with a novel, efficient algorithm.\n\nNumerical experiments demonstrate competitiveness against sota as well as a scenario of shell company attack.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n* The paper raises an important question by introducing the notion of shell company attack. \n* This is a paper articulating several original observations about a very active field of research. Leading to a well motivated algorithm (+++). I find this paper very innovative.\n* The reading of the paper is eased by the fact that the important ideas are well emphasized (\"key observations..., pictures..\"). \n\n\n# Weaknesses\n* I find the current version of Theorem 3 hard to parse. I believe this should be improved. \n* The paper mentions the copier attack only at the end (and they recognize the limitation of this pape with respect of this attack). In my first reading of the paper, I almost stop reading because they did not acknowledge it earlier (and this attack is what came to my mind immediately while reading). It would be good to mention the copier attack a bit ealier. \n* (Still on the copier attack) The remark at the end falls like a clifthanger, and I think it is quite easy to find counter-measure against the authors's  defensive proposal (the \"remove the copies\" argument). I think this part should be elaborated a bit more.\n\nQUESTIONS:\nA: How do you think theorem 3 presentation could be improved? \nB: Do you have arguments to weights the pros and cons between coping against copier attack and shell company attack, are there other aspects we should have in mind? \nC: How did you came up with condition 3? What does the literature tells us about it that made you believe it was the right condition to build the argument?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article addresses the problem of dataset valuation, where the goal is to fairly assess the contributions of agents—represented by datasets—using Shapley values. Traditional state-of-the-art (SOTA) methods are vulnerable to *shell company attacks*, where an agent splits their dataset into multiple subsets to unfairly increase their total valuation.\n \nTo overcome this, the authors propose the Faithful Group Shapley Value (FGSV), which evaluates each dataset as the sum of the Shapley values of its individual data points. Intuitively, this fine-grained approach prevents agents from benefiting through artificial data splitting, as FGSV assigns value at the most granular level.\n\nFGSV is shown to be the only dataset valuation method that satisfies a set of five axioms: the four standard Shapley axioms adapted to the group setting, plus *faithfulness*—the property that ensures robustness against shell company attacks.\n\nAlthough computing FGSV exactly is computationally expensive, the authors develop a fast and accurate approximation algorithm that works under mild regularity conditions. They demonstrate scenarios where these conditions are met and validate the proposed method with extensive experiments, showing strong performance compared to various SOTA alternatives.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The article is clearly written and easy to follow.\n- The addressed problem is both interesting and novel, particularly the focus on robustness against shell company attacks.\n- The paper is mathematically rigorous, with well-justified theoretical assumptions.\n- The proposed method is validated through a comprehensive set of experiments.\n\nWeaknesses\n– The method evaluates the importance of a dataset as the sum of the valuations of its individual data points, which may fail to capture the overall contribution of the dataset as a unit.\n- The performance of the proposed method under violations of the regularity conditions is not thoroughly investigated.\n- The literature review is insufficient, making it difficult to fully assess the contribution relative to existing work.\n- The paper lacks a dedicated discussion of limitations, especially regarding scalability and applicability in less ideal scenarios.\n\nQUESTIONS:\n1. How accurate is it to evaluate the importance of a dataset by aggregating pointwise valuations? Since the value function $u$ is not necessarily linear, the dataset-level Shapley value  $SV_{S_0}(u)$ can differ significantly from the sum $\\sum_{i\\in S_0} SV_i(u)$. Could you elaborate on the implications of this discrepancy?\n\n2. How does your method perform when Assumptions 1 and 2 do not hold? Specifically, do the three empirical scenarios presented in the paper satisfy these assumptions? If not, how robust is the method in such cases?\n\n3. Why is the area under the convergence curve an indicator of accuracy? While it reflects convergence speed, does it guarantee convergence to the correct values, particularly in light of the first concern regarding the dataset vs. pointwise valuation?\n\nLIMITATIONS:\nThe article does not adequately address the limitations of the proposed method. In particular, the concerns raised in Questions 1 and 2 are critical and deserve a thorough discussion.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors address the problem of dataset valuation based on Shapley values (called group-level Shapley), where the objective is to derive Shapley semi-values for each dataset participating into a machine learning task. This differs from classical datum valuation, where one player is a data sample from a fixed dataset. Group-level valuation is subjected to a specific adversarial attack (shell attack), where other players can split their datasets to change the value of the current player. To cope with that issue, the authors are proposing a generalisation of the Shapley value for dataset valuation by introducing an additional axiom introducing a fairness constraint to address shell attacks. They provide an algorithm computationally efficient and illustrate the benefits of the latter on several experiments.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n* The Faithful Group Shapley Value proposed by the authors is, up to my knowledge, novel.\n* The proposed algorithm is original, scalable (from $O(n^2)$ to $O(n)$ computational cost up to polylogarithmic terms, where $n$ is the number of players). Note that $O(n)$ is the SOTA computational cost in terms of number of utility evaluations.\n* All claims are theoretically justified, albeit I did not check the full supplementary material.\n* Experiments are revelant and indeed show the benefits of the proposed approach.\n\n** Weaknesses**\n* Could the authors provide more insights on why they choose to focus on the shell attack compared to others? A dedicated paragraph on the most impactful and common attacks would be valuable to know at which extent this novel methodology will be relevant for the community.\n* The authors are introducing FGSV with a new faithful axiom but are considering an algorithm approximating the latter and hence potentially loosing the faithful axiom and the ability to cope with shell attacks. Could the authors comment on that point?\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-22"}