Title: What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions
Venue: NeurIPS 2025
Accepted: yes
Keywords: data attribution

data valuation

influence functions
Abstract: Large language models (LLMs) are trained on a vast amount of human-written data, but data providers often remain uncredited. In response to this issue, data valuation (or data attribution), which quantifies the contribution or value of each data to the model output, has been discussed as a potential solution. Nevertheless, applying existing data valuation methods to recent LLMs and their vast training datasets has been largely limited by prohibitive compute and memory costs. In this work, we focus on influence functions, a popular gradient-based data valuation method, and significantly improve its scalability with an efficient gradient projection strategy called LoGra that leverages the gradient structure in backpropagation. We then provide a theoretical motivation of gradient projection approaches to influence functions to promote trust in the data valuation process. Lastly, we lower the barrier to implementing data valuation systems by introducing LogIX, a software package that can transform existing training code into data valuation code with minimal effort. In our data valuation experiments, LoGra achieves competitive accuracy against more expensive baselines while showing up to 6,500x improvement in throughput and 5x reduction in GPU memory usage when applied to Llama3-8B-Instruct and the 1B-token dataset.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a data valuation framework for large-scale language models by introducing **LoGra**, a low-rank gradient projection method that significantly reduces the computational and memory overhead of influence functions. Alongside, the authors present **Logix**, a practical software toolkit that simplifies the integration of data valuation into existing training pipelines. Compared to existing methods, LoGra achieves orders-of-magnitude improvements in efficiency without sacrificing accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper provides numerous concrete examples to demonstrate the effectiveness of the proposed method, making the results more convincing.\n\nIt introduces a practical toolkit (Logix) to lower the barrier of adoption for data valuation in LLMs.\n\nAdditionally, the authors analyze failure cases, offering insightful guidance for future improvements.\n\nWeaknesses:\n\nThe dataset used for large-scale experiments is limited both in variety and size, as only a 1B-token subset of OpenWebText (OWT) is tested. This may affect the generalizability of the results, especially to broader domains such as multilingual or multimodal datasets.\n\nQUESTIONS:\nThe experiments are limited to models like Llama3-8B-Instruct, GPT2-XL, and Pythia-1.4B. Is it possible to evaluate the approach on larger-scale models (e.g., 30B or 65B) to further validate scalability?\n\nThe analysis of failure cases on Pythia-1.4B is insightful. However, could the authors propose concrete remedies or practical strategies to mitigate such issues in future applications?\n\nIs the proposed acceleration method hardware-agnostic? For example, can similar improvements be expected on TPUs or different GPU architectures?\n\nIf the authors are able to address the above issues, I would be willing to raise my score.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a low-rank gradient projection algorithm named LOGRA, aiming to extend influence functions to LLM-scale data valuation tasks without significantly compromising accuracy. The authors also design LOGIX, a modular framework that seamlessly integrates data valuation procedures into standard training pipelines. Experiments demonstrate that this method substantially reduces computational and memory overhead while maintaining reasonable estimation quality.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths*\n\n1. Practical problem formulation: Data valuation is a highly relevant issue in the context of LLMs, especially with regard to legal attribution and compensation. The paper addresses the core challenge of extending influence function methods to modern large-scale models.\n\n2. Complete engineering implementation: The LOGIX framework is carefully designed with compatibility in mind, supporting modern PyTorch ecosystems such as FSDP. This adds practical value to the work.\n\n3. Well-executed quantitative experiments: The authors provide solid empirical evidence through standard metrics such as LDS and brittleness tests on small models.\n\n*Weaknesses*\n\n1. Limited generalizability: The effectiveness of LOGRA appears to rely on structural assumptions about the Hessian. For instance, PCA initialization performs poorly in Transformer architectures without weight sharing, raising concerns about broader applicability.\n\n2. Subjective qualitative analysis: The identification of “valuable data” relies heavily on human inspection of semantic similarity between training samples and model outputs. A more systematic and quantitative evaluation metric is lacking.\n\n3. Lack of ethical discussion: Despite the centrality of data ownership and provenance, the paper lacks a thorough discussion on potential misuse, legal concerns, or fairness implications of the proposed system.\n\nQUESTIONS:\n1.LOGRA's performance depends on the initialization of the Kronecker projection matrices (PCA or random). Is the method inherently less effective for architectures with less structured activations (e.g., Transformers with multi-head attention)?\n\n2.Does LOGIX support more complex training pipelines, such as multi-stage pretraining followed by SFT or RLHF? \n\n3.How do you assess whether the “valuable data” identified by LOGRA truly affect model behavior? Have you considered introducing metrics such as embedding coverage or goal-oriented perturbation?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an efficient data attribution algorithm by leveraging the gradient-projection approximation of the influence function with the special gradient structure for linear layers. Besides the algorithmic advancements, the authors also demonstrate how the algorithm can be efficiently implemented with the existing PyTorch library and provide a library that is capable of large-scale attribution.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The research problem is well-motivated, and the scope is clear.\n2. The overall writing and the illustration are professional and easy to follow.\n3. The contribution is sound: in particular, the algorithmic design by exploiting the linear gradient structure, as well as the backward pass for the sample gradient calculation.\n4. The evaluation is complete and thorough.\n\n**Weaknesses**\n1. The theory is not instructive and not entirely rigorous (in terms of the statement as well as the proof).\n2. After some investigation, the structure of the codebase does not seem to be as extensible as claimed: the logic is scattered throughout the codebase, hard to track the lineage of definition for data/function.\n3. The discussion around throughput gained is overclaimed: it comes from the dot products part, instead of the key technical advancements of this work, i.e., the logging part.\n\nQUESTIONS:\n1. I have played around with the provided library, and I have some questions: In the `LoraLinear` class, why is the initialization of the projection matrix using `nn.init.kaiming_uniform_`? In addition, it seems like the implementation does not handle the biases in the linear layers.\n2. (**Weakness 1**) Can the authors provide a rigorous statement of the Lemma and also the rigorous version of the proof? In particular, is it possible to avoid using $\\approx$ in the statement (and also the proof)?\n3. (**Weakness 3**) While I understand that the authors try to \"propose\" an overall logging/dot-product framework for data attribution, specifically the influence-function-style attribution, I think the key advancement lies in the introduction of the efficient gradient-projection in the logging stage. Hence, it is misleading to claim the speedups during the dot-product stage, especially when the baseline is largely limited by memory constraints. I suggest that the authors state the scenarios in which the speedup is obtained.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper build on the use of influence functions to determine the relationship between model outputs and a vast set of training data. In particular the paper introduces an efficient method called LoGra that uses a low rank projection that can be efficiently integrated with the training process. The authors consider tradeoffs in the size and selection of the low rank projection (e.g. initialization through a PCA-like process). The method is first validated against competing methods in a small scale test using smaller models on typical datasets like CIFAR-10. The authors then test this method against a 1B token sized training set (a proxy used for access to actual training sets of models) and test realistic model LLM model sizes (i.e. billions of parameters). The authors show throughout where prior influence function methods may fail, often due infeasible computations and enormous systems costs.\n\nSTRENGTHS AND WEAKNESSES:\nIn general I found this paper to be very strong and convincing, with huge improvements over similar methods that allow this work to scale to both billion parameter data and models. While very dense, it is still clearly written and touches on a broad range of areas to show that their method is both theoretically sound and practically scalable. \n\n## Strengths:\n\nS1: The paper is very well motivated, discussing shortcomings in both \"accuracy and efficiency\" of existing systems (such as difficulties in computing solutions to the inverse hessian products). Continually considers practical limitations.\n\nS2: Considers tweaks to the core method such as the PCA-like initialization of the low rank projections. The explanations here (3.2) motivate this idea and help understanding of the overall method. \n\nS3: Considers systems aspects like maximizing available compute on typical GPU setups, recomputation, and storage costs of projected gradients. \n\nS4: Gives some details about a software implementation of this method through the pytorch hooks interface. If this works as advertised (see W2), it makes this method even more relevant as it could be easily applied in many complex existing training stacks. \n\nS5: Good baselines and comparisons of why some competing methods could not even be evaluated. I appreciate that the authors validate and compared the method on both small scale experiments similar to the literature and larger scales. \n\n## Weaknesses\n\nW1: I am a bit unclear on the selection of the 1B train tokens from OWT. This is meant to be a standin for direct actual access to a sample of the LLM training data, correct? Is it valid or accepted in data influence literature to use a proxy for training in this way? I believe these experiments could have used a subset of the dolma data and the olmo models, as these are both modern (i.e. compared to your issues with the pythia models) and open access. You note that OWT may be higher quality than the overal dolma mixture, but it also exists as high quality filtered subsets. \n\nW2: Since the software package is considered a key contribution of this work (i.e. mentioned in the abstract) it would be good to show a practical end-to-end implementation of the package. Was the same package used to produce the LLM experiments? \n\nW3: Could provide more details about the comment in 255-257 about complications (\"... (i.e., no weight sharing) may not successfully keep larger components of the GPT2 Hessian...) with regards to \"Transformer architecutre ... lacks the specialized KFAC Hessian approximation\", as part of the central premise of the paper is that the method can be applied to billion param scale transformer models.\n\nQUESTIONS:\n- Clarify the decisions around the 1B OWT data selection\n- Clarify the usability of the software package\n- Since qualitative comparisons (the example prompts) are part of the results for this method, are there possible experiments that could show that the data marked as valuable is of further use in model training? One version of this could use high value documents as ICL exemplars (or simply in context knowledge) to test the effects on evaluations.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}