Title: Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retraining
Venue: NeurIPS 2025
Accepted: yes
Keywords: Data Attribution

Influence Function

Hyperparameter Tuning
Abstract: Data attribution methods, which quantify the influence of individual training data points on a machine learning model, have gained increasing popularity in data-centric applications in modern AI. Despite a recent surge of new methods developed in this space, the impact of hyperparameter tuning in these methods remains under-explored. In this work, we present the first large-scale empirical study to understand the hyperparameter sensitivity of common data attribution methods. Our results show that most methods are indeed sensitive to certain key hyperparameters. However, unlike typical machine learning algorithms---whose hyperparameters can be tuned using computationally-cheap validation metrics---evaluating data attribution performance often requires retraining models on subsets of training data, making such metrics prohibitively costly for hyperparameter tuning. This poses a critical open challenge for the practical application of data attribution methods. To address this challenge, we advocate for better theoretical understandings of hyperparameter behavior to inform efficient tuning strategies. As a case study, we provide a theoretical analysis of the regularization term that is critical in many variants of influence function methods. Building on this analysis, we propose a lightweight procedure for selecting the regularization value without model retraining, and validate its effectiveness across a range of standard data attribution benchmarks. Overall, our study identifies a fundamental yet overlooked challenge in the practical application of data attribution, and highlights the importance of careful discussion on hyperparameter selection in future method development.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates hyperparameters for data attribution and benchmarks their impact on several prominent methods: Influence Functions, TraceIN, TRAK, and LoGra, across the MNIST (MLP), CIFAR-10 (ResNet-9), and Wikitext (GPT-2) datasets. The main findings can be summarized as follows:\n\n- Most attribution methods are sensitive to certain hyperparameters.\n- The optimal hyperparameter choices vary across datasets and models.\n- The impact of hyperparameters can be entangled.\n- Implicit hyperparameters, such as training epochs, also play a significant role.\n\nFinally, the authors propose an algorithm (surrogate-indicator-based practical selection) for selecting the regularization term for influence functions without the need for retraining. The method is validated on MNIST, CIFAR, and Maestro datasets, showing that “overall, we observe that the λ chosen by the proposed algorithm generally leads to good LDS values that are significantly better than λ = 0, and in many cases, it is close to the optimal value.”\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The paper is easy to follow and addresses relevant questions that are analyzed thoroughly.\n\nWeaknesses: I do not find the contributions significant enough for NeurIPS. The insights from the study are not particularly surprising and may not provide substantial benefits to a broad set of researchers; the finding that “the best hyperparameter choices vary across datasets and models” indicates that researchers and practitioners will still need to perform hyperparameter searches after reading the paper. While the algorithm assists in selecting one parameter for influence functions, it focuses on a single parameter for one method, and I am not convinced of its generalizability to other datasets. In my opinion, this paper would be better suited for a specialized venue.\n\nQUESTIONS:\nAre there any findings or insights that are more surprising, or could any of the results be suggested as actionable steps for practitioners?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the problem of hyperparameter selection in data attribution methods, which is a topic that is severely underestimated in existing research but is crucial in practical applications. The authors first conduct a systematic large-scale empirical study to evaluate the hyperparameter sensitivity of mainstream data attribution methods on multiple models and datasets. In addition, the authors propose a surrogate indicator based on theoretical derivation, which can select the regularization hyperparameter λ without retraining the model. Experiments show that the indicator has good consistency and performance in multiple datasets and methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. This paper focuses on the practical issues in the implementation of data attribution methods.\n\n2. This paper covers multiple attribution methods and is highly representative.\n\n3. Attached with GitHub links and detailed experimental details, it is helpful for the community to verify and use it later.\n\nWeaknesses\n1. The proposed surrogate indicator is only applicable to the λ parameter in the influence function method, and cannot be extended to other key hyperparameters (such as projection dimension) or other types of attribution methods, with obvious limitations.\n\n2. The experimental indicators are single, and many results are based on the synthetic indicator LDS. End-to-end verification has not been performed in actual tasks (such as noise detection and training data screening), and it cannot be proved that the selected hyperparameters can bring better downstream performance.\n\n3. The proposed indicators are based on complex partial derivative conditions and high-order tensor expressions, which lack interpretability for non-theoretical researchers and have a high threshold for actual adoption.\n\n4. The author identified the pain points of hyperparameter tuning, but the solution he finally gave is more like an \"ad hoc fix\" rather than a systematic solution to the problem.\n\nQUESTIONS:\nOverall, I appreciate the contribution of this paper, so I gave it a positive rating. My main questions are as follows:\n1. In addition to λ, the projection dimension and the number of training rounds have also been shown to have a significant impact on performance. Why didn't you try to estimate them using a surrogate-like approach? Is it because of mathematical difficulty, non-differentiability, or other practical obstacles?\n\n2. Have you considered other performance indicators? All current experiments use LDS as the only metric. Have you considered using downstream performance as an evaluation metric in data denoising or data selection tasks? If it cannot be extended to practical tasks, the practicality of this paper will be questioned.\n\n3. Are theoretical indicators generalizable? The derivation of the surrogate is based on a series of technical assumptions. Do these assumptions hold true in large-scale deep models or strong non-convex situations? It is recommended to provide further empirical support or hypothesis sensitivity analysis.\n\n4. Is there a risk of \"misleading selection\"? The author's method attempts to fit the LDS trend with the surrogate curve, but is there a situation where the surrogate indicator fails to be monotonically in some areas, resulting in the misselection of suboptimal λ? Have the authors considered uncertainty estimates or confidence bounds to aid in the selection?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies hyperparameter sensitivity as a critical but overlooked challenge in the research of training data attribution. The authors conduct the first large-scale empirical study demonstrating that most data attribution methods are highly sensitive to key hyperparameters like regularization terms, projection dimensions, and training epochs. The paper provides a theoretical analysis of the regularization parameter in influence function methods and proposes a practical, retraining-free selection heuristic that approximates optimal regularization values. Their experiments across multiple datasets and models show that the proposed method effectively selects good hyperparameter values without the computational cost of repeated model retraining.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- This paper revisits the active research area of training data attribution and identifies a critical aspect that has been largely overlooked by the community—the computational challenge of hyperparameter tuning in data attribution methods.\n- The paper is well-written and clearly structured, making it accessible to readers (with the exception of Section 4.1, discussed below).\n\n**Weaknesses**\n- Section 4.1 suffers from poor presentation and organization. The notation is confusingly introduced in separate remark paragraphs after the main theorem statement, making it difficult to parse the theoretical contributions. This section would benefit from significant revision to include: (1) clear interpretation of the theorem statement, (2) intuitive explanations of the key insights, and (3) a proof sketch for Theorem 4.3 (and other theoretical results if space allows). \n\n- While the proposed heuristic in Section 4.2 demonstrates effectiveness in the experiments of Section 4.3, there are concerns about its generalizability. The fixed threshold of 0.5 may work well primarily because the experimental evaluation relies on relatively clean datasets (MNIST and CIFAR-10). I suspect this threshold would require significant adjustment for different types of datasets or domains. This concern is supported by the notably worse performance on the MAESTRO dataset, suggesting that the method's robustness across diverse data characteristics remains questionable.\n\nQUESTIONS:\nThere is also a hyperparameter in the metric of LDS, the subset size $a$. I wonder whether varying $a$ will also significantly change the results?\n\nLIMITATIONS:\nIt would be good to evaluate the proposed heuristic with more datasets.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the problem of hyperparameter sensitivity in training data attribution (TDA) methods, both from theoretical and empirical perspectives. The authors show that popular attribution methods such as Influence Functions (IF), LiSSA, TRAK, and LoGra exhibit significant sensitivity to hyperparameter choices. Moreover, the optimal hyperparameter settings are shown to be highly dependent on the specific dataset and model, and the effects of multiple hyperparameters can be entangled, i.e., the best value for one hyperparameter may depend on the value of another.\n\nSince accurate evaluation of TDA methods typically requires multiple retraining runs, empirical tuning of hyperparameters is computationally expensive and often impractical. To address this challenge, the authors propose a theoretically grounded framework to guide hyperparameter selection. As a case study, they theoretically derive a practical algorithm for selecting the regularization parameter in the IFFIM method, e.g. an IF variant based on an approximation involving Fisher Information Matrix. Empirical results demonstrate that this algorithm can often find hyperparameter values that are close to optimal.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n* The paper is well-written and well-structured.\n* The empirical study is comprehensive, covering a variety of popular TDA methods (Influence Functions, LiSSA, TRAK, LoGra), and convincingly demonstrates hyperparameter sensitivity across datasets and models.\n* The work brings valuable attention to a widespread, under-addressed issue in the TDA literature.\n* The paper presents a promising strategy to mitigate this issue: using theoretical derivations to guide hyperparameter selection. The case study demonstrates how this approach can help identify a promising region in hyperparameter space without expensive evaluation metric calculations.\n\n**Weaknesses:**\n\n* The proposed theoretical strategy, while appealing in principle, is less convincing in the case study. The theoretical derivation involves several assumptions (e.g., the LDS being uni-modal and concave in $\\lambda$, technical assumption in Appendix B.3) and heuristics (e.g., $\\xi_{T, \\lambda}$ in lines 263–264) that may not hold broadly in practice. The empirical results in Figure 2 show that the hyperparameter values determined by the proposed algorithm are not always close to the optimal value.\n* It is unclear how easily this theory-driven approach for hyperparameter selection can be generalized to other TDA methods beyond IF.\n* The derivation of Algorithm 1 lacks clarity:\n\n  * Theorem 4.3 introduces a large number of new definitions with little intuitive explanation.\n  * The purpose of Proposition 4.7 is unclear.\n  * The derivations steps from the theoretical results (Theorem 4.3, Proposion 4.7) to Algorithm 1 are not clearly articulated.\n\n**Minor Weaknesses:**\n\n* Possible typos:\n\n  * “The sampled subset” (line 212) — may mean “a sample of subsets.”\n  * “Trained the full dataset S” (line 258)\n* The finding that earlier model checkpoints may produce better results than final ones for some TDA methods (line 111) is quite interesting. I wish the authors gave possible explanations for this.\n\nQUESTIONS:\n1. Please clarify the interpretation of the terms in Equation 8 in Theorem 4.3. A more intuitive explanation would improve the accessibility of the result.\n2. How do Theorem 4.3, the surrogate indicator, and Proposition 4.7 connect to the derivation of Algorithm 1? Please describe this connection more explicitly.\n3. What is the computational cost of estimating the optimal hyperparameter using Algorithm 1, and how does it compare to computing the LDS metric under similar conditions?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}