Title: GraSS: Scalable Data Attribution with Gradient Sparsification and Sparse Projection
Venue: NeurIPS 2025
Accepted: yes
Keywords: data attribution

influence function

gradient compression
Abstract: Gradient-based data attribution methods, such as influence functions, are critical for understanding the impact of individual training samples without requiring repeated model retraining. However, their scalability is often limited by the high computational and memory costs associated with per-sample gradient computation. In this work, we propose **GraSS**, a novel gradient compression algorithm and its variants **FactGraSS** for linear layers specifically, that explicitly leverage the inherent sparsity of per-sample gradients to achieve sub-linear space and time complexity. Extensive experiments demonstrate the effectiveness of our approach, achieving substantial speedups while preserving data influence fidelity. In particular, **FactGraSS** achieves up to 165% faster throughput on billion-scale models compared to the previous state-of-the-art baselines. Our code is publicly available at https://github.com/TRAIS-Lab/GraSS.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GraSS , a new method for making gradient-based data attribution more efficient. The key idea in this work is to exploit the sparsity of these per-sample gradients, most of the values are close to zero, and use that to compress the gradients. They also introduced a specialized version called FactGraSS , designed specifically for linear layers, which gives even better performance. In experiments, their method was significantly faster up to 165% faster throughput on huge models while still keeping the accuracy of influence estimation.\n\nSTRENGTHS AND WEAKNESSES:\nstrength: \n\nThe method demonstrates a significant improvement in computational efficiency. \n\nWeaknesses: \n\nThe presentation of the paper is somewhat difficult to follow.\n\nQUESTIONS:\n1. Most current attribution methods, especially those based on Leave-One-Out (LOO) influence analysis, are built upon the Influence Function introduced by Koh and Liang. The core computation involves the product of the inverse Hessian matrix and the gradient. This framework has two main issues: \n\n**Efficiency** : Computing or approximating the inverse Hessian is time-consuming and difficult, which has led to many follow-up works focusing on how to accelerate this part. \n\n**Accuracy** : Influence Function-based methods assume the loss function is strongly convex, which theoretically limits their maximum achievable accuracy. \n\nLater, more accurate methods emerged, such as SGD-Inf. My question is: Can newer acceleration techniques like GraSS also be applied to speed up these more advanced attribution methods?\n\n---\n\n2. Building on the first question, if the issue of speed is largely resolved, does the accuracy of current attribution methods still fall significantly short of what is required in real-world applications? Take SGD-Inf, one of the most accurate methods available today, as an example: its error grows exponentially with the number of training steps (exp(T)), meaning the error deteriorates rapidly as training progresses. \n\n---\n\n3. Is such a level of error still acceptable in practical scenarios? The third question again concerns the computational bottleneck. Even with efficient algorithms like GraSS or DataInf, the overall computation time for attribution still requires at least one full pass over the training data — that is, it has O(N) complexity. Meanwhile, in many large-scale model trainings today, the model is often trained for only one or a few epochs. This implies that the cost of attribution may already approach or even exceed the cost of the entire training process.\nIn such cases, how viable is the application of data attribution methods in practice?\n\nLIMITATIONS:\nsee Questions", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of scalable, gradient-based data attribution. Influence functions, commonly used for scoring the importance of samples in a dataset, require computing a (inverse) Hessian-gradient product, which is extremely costly for very large models. The authors propose two novel methods, GRASS and FACTGRASS. GRASS sparsifies the gradients using their proposed \"selective mask\" technique, after which they apply the sparse Johnson-Lindenstrauss transform. FACTGRASS enables the use of the compression technique via an interesting modification of the LoGRA technique applied to layerwise sparsified gradients of linear layers. The proposed data attribution techniques are evaluated with the LDS benchmark on million scale models and simple vision/music datasets, as well as on Llama3-8B models using the OpenWebText dataset, showing improved efficiency of computation over current methods.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n* The paper is generally quite well written\n* The methods proposed address the real problem of the scalability of gradient-based data attribution, and the proposed method has a significantly lower wallclock time than the baselines\n* The computational speedup of the GrASS method is shown rigorously in sections 3-4.\n\n## Weaknesses\n\n* Theoretical analysis could be written more clearly. The computational complexity resutls could be stated as formal results, at least in the appendix. Moreover, a table comparing computational costs of the different methods would have helped the exposition significantly/.\n* The GRASS and FACTGRASS influence functions should be written as numbered equations for clarity.\n* There is no discussion on the statistical properties of the GRASS estimator. Please refer to the \"Questions\" section as well.\n* The quantitative analysis with the LLM is not very clear, and should be more clearly expressed.\n* I feel the scope of this work is slightly limited - please refer to the \"Limitations\" section.\n\nQUESTIONS:\nI have the following questions:\n\n* How does the gradient sparsification impact the accuracy of the influence function based attribution? That is, given a dataset, can you discuss how the GRASS/FactGRASS influence functions compare with a full influence function? \n* Can you provide a bound on the (expected) deviation from the \"true\" influence function score (as given in Koh and Liang's paper)? Can you quantify the manner in which gradient sparsity influences this?\n\nLIMITATIONS:\nI feel a key limitation to this work is the scope. The work, as presented, only proposes a method for efficiently computing influence functions. While the results appear to be technically correct, this work does not address any deeper question other than a simple improvement in cost. Moreover, there are no rigorous theoretical results - stated as Lemmas or Theorems - of interest from a statistical standpoint, though the computational complexity guarantees are nice to have.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes GRASS, a new gradient compression method to make influence functions faster and more memory-efficient for large neural networks with sparsification and projection.\n\nSTRENGTHS AND WEAKNESSES:\nStrong Points\n----\nS1. The paper tackles the important problem of scaling influence functions to very large models.\n\nS2. Combining gradient sparsity and sparse projection is a novel way to speed the influence function computation.\n\nWeak Points\n----\nW1. The paper mentions RapidIn (Lin et al., 2024), which also uses random projection, but does not include a direct experimental comparison or discussion of pros and cons versus RapidIn.\n\nW2. The experiments focus on only a few models, especially only one billion-scale model.\n\nW3. The importance and stability of mask selection in Selective Mask is not analyzed much.\n\nW4. The effect of different hyperparameter settings (like k and k') is not fully explored.\n\nW5. Random masking loses accuracy compared to other methods.\n\nW6. For the billion-scale model, the paper reports only speed and a single qualitative example, but does not provide any quantitative evaluation of data attribution accuracy.\n\nQUESTIONS:\nQ1. Can the authors provide a direct quantitative comparison between GRASS/FACTGRASS and RapidIn (Lin et al., 2024) on common benchmarks?\n\nQ2. Can the authors add quantitative accuracy evaluation (such as LDS or other metrics) for the billion-scale model instead of only reporting speed and one qualitative example?\n\nQ3. Can the authors analyze how the Selective Mask is chosen and whether its stability or importance affects the attribution results?\n\nQ4. Can the authors include experiments or analysis showing how sensitive GRASS/FACTGRASS is to key hyperparameters like k and k'?\n\nQ5. Can the authors explain or empirically study why random masking leads to lower accuracy and in what situations it is still a good choice?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper proposes to leverage the per-sample gradient sparsity inherit in neural networks with relu activations to speed up influence function computation.  In particular, the sparse JL lemma is leveraged to accelerate random projections to take advantage of the sparsity.  Furthermore, additional extensions are proposed: 1) the parameters are first further sparsified via a one-time optimization problem 2) extending Logra, the input activation and output gradients in linear layers are sparsified, and a similar approach is applied to their respective Kronecker product for computational speedup.  The resulting method is able to achieve up to 165% speedup over Logra on Llama-3.1-8b-instruct.

STRENGTHS AND WEAKNESSES:
Strengths
- the paper proposes a novel core idea - leveraging / introducing parameter sparsity to speedup influence function computations.  this idea could likely be extended to other problems involving per-sample gradient compression
- the experiments show show significant speedup over Logra on billion-parameter models

Weaknesses,
- analysis focuses only on 1 billion-parameter model, unlike Logra (which considered 3), though this is understandable given computational resources.

QUESTIONS:
- is it possible to perform some quantitative analysis of FactGrass for the 8B model beyond throughput?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-06