Title: Data Valuation by Leveraging Global and Local Statistical Information
Venue: NeurIPS 2025
Accepted: no
Keywords: Data valuation

Value distribution

Shapley value

Average marginal effect

Dynamic valuation
Abstract: Data valuation has garnered increasing attention in recent years, given the critical role of high-quality data in various applications. Among diverse data valuation approaches, Shapley value-based methods are predominant due to their strong theoretical grounding. However, the exact computation of Shapley values is often computationally prohibitive, prompting the development of numerous approximation techniques. Despite notable advancements, existing methods generally neglect the incorporation of value distribution information and fail to account for dynamic data conditions, thereby compromising their performance and application potential. In this paper, we highlight the crucial role of both global and local statistical properties of value distributions in the context of data valuation for machine learning. First, we conduct a comprehensive analysis of these distributions across various simulated and real-world datasets, uncovering valuable insights and key patterns. Second, we propose an enhanced data valuation method that integrates the explored distribution characteristics into the existing AME framework to refine Shapley value estimation. The proposed regularizers can also be seamlessly incorporated into various data valuation methods. Third, we introduce a novel approach for dynamic data valuation that infers updated data values without recomputing Shapley values, thereby significantly improving computational efficiency. Extensive experiments have been conducted across a range of tasks, including Shapley value estimation, value-based data addition and removal, mislabeled data detection, and dynamic data valuation. The results showcase the consistent effectiveness and efficiency of our proposed methodologies, affirming the significant potential of global and local value distributions in data valuation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper builds upon an existing work that proposes how to efficiently approximate the Average Marginal Effect (AME) such as the Shapley value. Instead of the original approach that uses the solution to Lasso Regression, the paper proposes to modify the objective to include (1) a different global term and (2) a new local term. (1) uses the L2 norm based on the observation that data values tend to follow a Gaussian distribution instead of a Laplace distribution. (2) is designed to enforce that data with similar features and labels should have more similar data values. Next, the paper proposes adapting the objective to deal with dynamic data valuation where new data points are added or removed. Instead of recomputing the AMEs, the objective is to keep the data values within some permissible variation decided by heuristics. The paper empirically compares the proposed method, GLOC, against the original AME approach and other data valuation techniques on multiple datasets and settings (mislabeled data detection, dynamic data valuation).\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. The experiments show that GLOC significantly improves the solution over AME and the heuristics proposed (e.g. L2 regularisation) are justified empirically.\n2. It is novel to consider adapting GLOC for dynamic data valuation to avoid the need for reestimating Shapley values. \n\n### Weaknesses\n1. Some claims can be made clearer. \n    * For example, in line 111 of the related work, it is stated that \"existing algorithms largely disregard the distributional characteristics of data values, leading to suboptimal performance and efficiency. On line 114, \"distributional information\" is mentioned. However, to the reader, the idea and importance of distributional information is unclear as it is not explained. Based on the experiment section, it seems that GLOC is also marginal contribution based and is able to obtain a solution with lower MSE based on different assumptions about the data value distribution.\n    * Clearer explanation of Fig 1.c-d\n2. The design choices are based on heuristics and empirically validated instead of theoretical motivations (e.g. LASSO for sample efficiency). Thus, there is no theoretical guarantees.\n\n### Minor Comments\n* line 91: \"Sharpley\" -> \"Shapley\"\n* In Fig 1, clearly state the null hypothesis.\n* cite the specific appendix\n\nQUESTIONS:\n1. Can you better explain Fig 1c-d? What dataset is used? Do you consider the pairwise difference in data values? Is there any significant difference between the blue and orange line?\n2. Can Fig 3 be modified to compare the MSE and MAE achieved by the original regression problem with no regularisation, AME, GLOC, GLOC without R_g and GLOC without R_l? Is the graph similar across different sample size? Is it meaningful to compare the Spearman correlation between each method's data value and the exact SV?\n3. Does GLOC still outperform AME when removing data points with _low_ values and adding those with _large_ values?\n4. (Limitations of GLOC) On what datasets and data samples do GLOC underperform? Is GLOC also bad at assigning low values to mislabeled data?\n\nLIMITATIONS:\nThe limitations are discussed in App A.12.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-16"}

comments: SUMMARY:
The paper studies a new data valuation method considering the distribution of value and position information. Inspired by the difference between LASSO and Ridge, the authors derive a valuation with a quadratic penalty after observing a Gaussian-like distribution. The valuation based on value distribution is quite novel, and including similarity among data is natural and reasonable. However, the experimental design is not very convincing, and the literature review doesn't cover sufficient recent papers.

STRENGTHS AND WEAKNESSES:
Studying the value distribution and including the similarity is quite insightful. However, as one of the strengths of the new valuation is its computational efficiency, the paper lacks a comparison with existing methods. 

On Lines 152-153, as you assume X comes from a Gaussian distribution, it's not very convincing to say the value distribution is Gaussian-like. For example, you could conduct another simulation with Laplace-like X and conduct the same test. 

On Line 249, "limited range" seems not rigorous. The paper only shows 10% won't change the valuation significantly. Changing the proportion and monitoring the trend will be helpful to understand the actual influence. 

Besides, the paper doesn't contain up-to-date literature. Most of the references are before 2024. For example, "The Value of Information in Human-AI Decision-making" and "An Instrumental Value for Data Production and its Application to Data Pricing" consider the dynamic case.

QUESTIONS:
Why do you use $||\beta||_2$ rather than $||\beta||_2^2$, as in Ridge regression, the penalty term is L2 norm square？How to choose $k$ in your experiments?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose AME-based GLOC, IncGLOC, and DecGLOC data valuation methods based on the identified global and local distribution characteristics of data values. \nThey theoretically define the methods and algorithms and empirically evaluate them on several real-world datasets, comparing the results to those of 11 existing data valuation methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The Authors identify interesting distributional patterns of data values and leverage the AME method to propose a global and local characteristics-based (GLOC) data valuation approach and two dynamic data valuation methods (IncGLOC and DecGLOC) designed for scenarios involving the addition of new data and the removal of existing data.\n\n- Figure 1 and 2 presents interesting insights, and the authors' perspective on dynamic data valuation is particularly relevant and could be highly valuable in the context of data markets.\n\n- The authors conducted experiments on several datasets, compared their work with several data valuation methods, and also included their code in the supplemental material. (I haven't tested it yet)\n\n- In general, the paper is well-written and easy to understand. \n\n\n**Weaknesses**\n\n- Section 3\n   - Clarifications about Figures 1c and 1d. \n       - Are the results on the random dataset?\n       - Is there an explanation for why class +1 is always higher? \n       - Is there a class imbalance?\n       - If the difference is truly in % (that is, if it's not a mistake), then those values for both inter and intra are very low.\n   - On line 152, to improve readability, I think that authors should include where in the appendix more results on this are presented.\n   - As discussed in Section 3, the authors appear to use the AME-based SV estimator. Instead of only mentioning AME, I think it would improve readability to say AME-based SV estimator because it can be confusing to see Shapley values and then AME.\n\n\n- Sections 4.2 and 4.3\n   - How does the choice of Nk (line 220) and the similarity metric affect things?\n   - Since the CS-Shapley data valuation method discriminates between training instances' in-class and out-of-class contributions, can the authors compare GLOC to Class-Shapley (Schoch et al. 2022)?\n   - Since most marginal contribution methods are not stable across runs, are affected by choice and design of utility function, and are in most cases not rank stable, is it possible that reliance on current values \\Beta^cur might lead to over or under-estimation of values for the added/removed data points?\n   - Since one of the mentioned advantages of the method is computational efficiency, can the authors add a computational analysis (runtime analysis) of the three proposed methods?\n\n- Section 5 \n  - The authors say, \"Given the benchmark Shapley values (SV),\" but it's unclear which of Shapley methods among those listed they use and how they compute the values. Also, why are Shapley values used as the benchmarks?\n\n\n- Other observations, questions, and concerns\n   - The likely limitations of the proposed framework.  1) To estimate Shapley values under L2 error, the monotonicity and sparsity assumptions must hold, which may not always be the case, depending on the data distribution and utility section, among other design choices. 2) Hyperparameter tuning Nk could make the proposed data valuation methods harder to scale. 3) Reliance on neighbors might compromise privacy guarantees. 4) There is a possibility of (over)underestimating the value of some samples. 5) There is a possibility of a mismatch between the underlying model, resulting utility, and the valuation method. 6) Since the efficiency axiom does not hold in this case (according to [25]), then it might be hard to guarantee the nice \"fairness\" properties.\n\nQUESTIONS:\nBelow are the questions, clarifications and concerns I would like the authors to address\n\n- Section 3\n   - Clarifications about Figures 1c and 1d. \n       - Are the results on the random dataset?\n       - Is there an explanation for why class +1 is always higher? \n       - Is there a class imbalance?\n       - If the difference is truly in % (that is, if it's not a mistake), then those values for both inter and intra are very low.\n   - On line 152, to improve readability, I think that authors should include where in the appendix more results on this are presented.\n   - As discussed in Section 3, the authors appear to use the AME-based SV estimator. Instead of only mentioning AME, I think it would improve readability to say AME-based SV estimator because it can be confusing to see Shapley values and then AME.\n\n\n- Sections 4.2 and 4.3\n   - How does the choice of Nk (line 220) and the similarity metric affect things?\n   - Since the CS-Shapley data valuation method discriminates between training instances' in-class and out-of-class contributions, can the authors compare GLOC to Class-Shapley (Schoch et al. 2022)?\n   - Since most marginal contribution methods are not stable across runs, are affected by choice and design of utility function, and are in most cases not rank stable, is it possible that reliance on current values \\Beta^cur might lead to over or under-estimation of values for the added/removed data points?\n   - Since one of the mentioned advantages of the method is computational efficiency, can the authors add a computational analysis (runtime analysis) of the three proposed methods?\n\n- Section 5 \n  - The authors say, \"Given the benchmark Shapley values (SV),\" but it's unclear which of Shapley methods among those listed they use and how they compute the values. Also, why are Shapley values used as the benchmarks?\n\n\n- Other observations, questions, and concerns\n   - The likely limitations of the proposed framework.  1) To estimate Shapley values under L2 error, the monotonicity and sparsity assumptions must hold, which may not always be the case, depending on the data distribution and utility section, among other design choices. 2) Hyperparameter tuning Nk could make the proposed data valuation methods harder to scale. 3) Reliance on neighbors might compromise privacy guarantees. 4) There is a possibility of (over)underestimating the value of some samples. 5) There is a possibility of a mismatch between the underlying model, resulting utility, and the valuation method. 6) Since the efficiency axiom does not hold in this case (according to [25]), then it might be hard to guarantee the nice \"fairness\" properties.\n\nLIMITATIONS:\nAuthors have a limitations section in the appendix. However, I think there are several limitations, e.g., those in the added in the \"questions\" section above that authors should consider adding or discussing.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach to data valuation based on the previously proposed AME method. There are two major improvements: (1) as observed in experiments, the distribution of data Shapley is more close to Gaussian distributions, therefore, they replace the Laplace priors with Gaussian priors; (2) they also observe that data points with the same label in the neighborhood usually have similar values, so they add a regularization term to enforce neighborhood consistency to refine Shapley value estimation. Additionally, the authors present dynamic data valuation methods that update data values without recomputing Shapley values when new data are added. They evaluate their approach on tasks like Shapley value estimation, mislabeled data detection, and dynamic data valuation.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The paper presents insightful empirical findings on the distribution of Shapley values across data points and introduces a novel method that leverages these observations to enhance existing data valuation approaches.\n- The proposed method is simple and scalable, and it appears to be effective across different tasks. \n- The experiments are comprehensive, covering diverse datasets and comparing against multiple baselines. The paper provides clear implementation details.\n\n**Weaknesses:**\n1. While the method appears to be effective, it is an extension of AME rather than a fundamentally new approach. The core novelty lies in the regularization terms (Gaussian prior + neighborhood consistency), which, while well-justified, do not represent a paradigm shift in data valuation.\n2. The MSE evaluation uses AME-estimated Shapley values as the \"ground truth\", which may favor AME-based methods (including the proposed methods) over non-AME baselines (e.g., KNN Shapley, Data Shapley). In addition,  the MSE of all methods appear to be very small, it’s unclear whether the improvements are practically significant or if alternative ground truth estimations (e.g., Monte Carlo Shapley) would yield different conclusions. A sensitivity analysis using different ground truth methods would strengthen the validity of the comparisons.\n3. In the data addition/removal experiments, it is not clear whether the rapid accuracy changes stem from: (1) improved Shapley value estimation accuracy, or (2) the neighborhood consistency regularization forcing similar data points to have similar values, causing them to be removed together and resulting in a faster performance drop.\n\nQUESTIONS:\nCould the authors address Weakness 2 and Weakness 3 in their response?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}