Toward Efficient Influence Function: Dropout as a Compression Tool
Authors: Yuchen Zhang
Mohammad Mohammadi Amiri
Venue: Accepted by TMLR ()

Abstract: Assessing the impact the training data on machine learning models is crucial for understanding the behavior of the model, enhancing the transparency, and selecting training data. Influence function provides a theoretical framework for quantifying the effect of training data points on model’s performance given a specific test data. However, the computational and memory costs of influence function presents significant challenges, especially for large-scale models, even when using approximation methods, since the gradients involved in computation are as large as the model itself. In this work, we introduce a novel approach that leverages dropout as a gradient compression mechanism to compute the influence function more efficiently. Our method significantly reduces computational and memory overhead, not only during the influence function computation but also in gradient compression process. Through theoretical analysis and empirical validation, we demonstrate that our method could preserves critical components of the data influence and enables its application to modern large-scale models.

Published in Transactions on Machine Learning Research (02/2026)
Toward Efficient Influence Function: Dropout as a Compression Tool
Yuchen Zhang
shangyy4 Orpi. edu
Department of Compuler Science Rensselaer Polytechnic Institute
Mohammad Mohammadi Amiri
mohamm / /rrpi. edu
Department of Computer Science
Rensselaer Polytechnic Institute
Reviewed on OpenReview:
Abstract
Assessing the impact the training data on machine learning models is crucial for understand- ing the behavior of the model, enhancing the transparency, and selecting training data, In- fluence function provides a theoretical framework for quantifying the effect of training data points on model's performance given a specifie test data. However, the computational and memory costs of influence function presents significant challenges, especially for large-scale models, even when using approximation methods, since the gradients involved in compu- tation are as large the model itself. In this work, introduce novel approach that leverages dropout as gradient compression mechanism to compute the influence function more efficiently. Our method significantly reduces computational and memory overhead, not only during the influence function computation but also gradient compression process. Through theoretical analysis and empirical validation, we demonstrate that our method could preserves critical components of the data influence and enables its application to modern large-scale models.
1 Introduction
Large loundation models such as GPT-4 AAchaam el al.j 2023). Llama (Grattafiori el al. 2024), and DeepSeek (Liu al.) 2024), have showcased remarkable capabilities across a variety of tasks. Despite their success, the statc-of-the-art models face persistent challenges, including hallucination (Lin ct al.
2021 Huang al. 2025 and the generation of toxic biased content (Abid et al. 2021 Wang et al. 2023
A critical factor underlying these shortcomings is the compasition and quality of their training data Park
et all.101 2023). Furthermore, training data also impart the knowledge that forms the foundation of a model's 's capabilities (Wang et al.] 2024 Meng et al. 2022 Mirzadeh et al. 2024). This raises a critical question: which data contribute positively to a model's s performance, and which ones negatively impact it? Addressing this highlights the need for robust methods evaluate the influence of training data.
Influence function, theoretical method rooted in statistics (Hampel 1974 Law 1986). which originally used to assess the robustness of statistical estimator (Huber & Ronchetti 2011), provides powerful tool for assessing the impact of training data on model's parameters and subsoquently on the model's perfor- mance. It offers a framework to understand how modifications to the training dataset propagate through the model. The concept has since been adapted to deep learning (Koh Liang 2017 Koh et al., 2019). enabling application to modern largo-scale modcls. This method has been wildly used in training data selection (Xia al. 2024 Yu et al. 2024 Hu 2024), data synthesizing (Li et al. 2024), and mislabel data detection (Koh Liang 2017
Kwon al.] 2023).

---

Published in Transactions on Machine Learning Research (02/2026)
Although the inffuence function provides a robust framework and has demonstrated promising results. their practical application is often hampered by high computational costs (Kwon et al. 2023 Zhou et al. 2024 Choe et al.) 2024). Computing influence function involves calculating an inverse Hessian-vector product (iHVP) and the gradients of the loss function with respect to both training and testing data. Since the Hessian matrix's dimensionality scales quadratically with the size of the model, and each gradient is large as the model itself, these pro- cesses become prohibitively expensive for large-scale model, both in terms of computation and mem- ory. Previous methods have attempted to mitigate the overhead of influence function through approx- imation or compression techniques. Approximation methods typically rely on itcration (Agarwal al., 2017 Koh & Liang]
Memory Efficiency
Performance
2017) closed form approx- imation (Kwon et a.. 2023), but the large size of stor-
gradients poses a fundamental challenge, ing all gradients can be impractical for large-scale models. Compression methods, such as those us- ing random Gaussian projections (Prrk et al. 2023 or principal component analysis (PCA) (Choe et al., 2024), alleviate the challenge posed by the large size of gradients, but they introduce additional memory and computational overhead during the compression process, as shown in Figure[] These approaches still face challenges in scaling to modern large-scale mod- cls or adapting to diverse model architectures and structure, which further limits their practicality.
PCA
Gaussian Dropout
Computational Efficiency
Figure 1: Comparison of different compression meth- ods for influence function estimation. PCA identifies important directions but incurs high computations ional overhead. Both PCA and Gaussian projection require storing a compression map, which can be memory in- tensive. In contrast, Dropout avoids both computa- tional and memory overhead, making it more effi- cient alternative.
In this work, we focus on compression-basee ap- proaches to influence estimation. Research has shown that modern machine learning (ML) mod- els are highly overparameterized (Balaji et al., 2021] Fischer et al. 2024), with only a small subset of pa- rameters playing a critical role in their performance previous studies indicate that the influence of training data is closely tied to the high spectrum of the Hes- sian matrix, where the majority of eigenvalues are concentrated near zero, and only few outliers deviate significantly from the bulk (Sagun et al.) 2016 2017). These findings highlight that tracking data influence does not require exhaustive computation over the entire parameter space, but can focus few critical directions or small subset of parameters.
Fedus et al. 2022 Xue et al. 2024 Furthermore,
Our Contributions. We observe that the influence of training data on the performance of a ML model can be effectively tracked through a small subset of parameters, reducing the need to consider the full parameter space. Building on this, we propose a novel dropout-based compression method compress gradients, which is both straightforward to implement and scales elficiently large-scale ML models. This significantly reduces both memory and computational complexity associated with computation of influence function and the compression process itself. Through theoretical analysis and empirical experiments, we validate the effectiveness of the proposed method, demonstrating its ability to capture data influence while offering efficiency.
2 Preliminaries
We denote the input space and the output space by X' and y. respectively. Let Dur = (2t 22 ... + represent the training datasct, where each training data point 2 = (ofp v(r) E Xxy. For a given data point (z. y) and a model with parameters 0 θ, let 1 (g. fo()) denote the loss function, where fo X + y is the model parameterized by 0, and 1 y y R measures the discrepancy between the output and

---

Published in Transactions on Machine Learning Research (02/2026)
the ground truth. The gradient of loss function evaluated the data point with respect to 0 is denoted as Vol (v, fo(x)) Additionally, let Dial { "vub>". , validation data point 2 (1 Veai) X Finally, denote the number of parameters in the model by d.
F denote the validation dataset, where each
2.1 Influence Function
The influence function quantifies how the model parameters change in response to upweighting a specific
training data point, and how the change affects the model's performance (Hampel) 1974 Law 1986 Koh
& Liang. 2017). Formally, given an infinitssimllly small 6 > 0, the upweighted empirical risk minimization problem formulated by increasing the weight of the k-th training data point = = (2 v() function. The optimization problem given by:
the loss
(c) = arg min #Ee 12
Σ l (virs fo(xx()) + (yir fo(ati))
Assuming the loss function twice-differentiabl and strongly convex 0 the influence of the k-th training data point on the empirical risk minimizer 0* is defined as the derivative of (())(e) at (Koh & Liang, 2017
Io- (2/r)) = (c) -- H H
de
is the empirical Hessian matrix and gi. = Val (vii: So(ati)) B=a. represents the gradient of the loss function evaluated at the k-th training data point =
where H = + Σ. Vill (yies fo(((())))) A=0-
For the validation dataset Dval
{2 vaal: 2 }. the influence of the training data point z on the
validation loss is Koh Liang 2017 Kwon et al. 2023
# I(
1
(mas) Io. (2(+)
gval
H-
H
(1)
m
m
where P.m Vol (Rait fola (x(()/a)))))
the gradient of loss function evaluated at 2
8=00
The influence function I(2 provides an intuitive method to evaluate whether training data point &tr is beneficial or detrimental to the performance of the model on the validation dataset D When the loss function is cross-entropy loss, the Hcssian matrix could be approximated with the
Matrix (FIM), which is equivalent to the Gauss-Newton Hessian (Martens 2020 Bae et al.] 2022 Grosse et all, 2023). Note that H not invertible if the dimension of 0 exceeds the size of training dataset n, which is common in many modern ML models. To address the issue, damping term is added to H, i.e., replacing H with H + MLs where X is small constant, and Ia is d d identity matrix.
2.2 Compressing Gradients for Influence Function
Computing the influence function faces several challenges when fo is large-scale deep learning model (Bauu et al. 2020 Bae et al., 2022 Kwon et al.] 2023). A key obstacle is that the size of the Hessian becomes prohibitively large to compute directly, as its dimensionality scales quadratically with the number of the model parameters.
To address this challenge, several methods (Schioppa et al. 2022 Park al. 2023) propose projecting gradients onto low-dimensional subspace using random Gaussian projection matrix (Johnson et. al.
3

---

Published in Transactions on Machine Learning Research (02/2026)
1984 and computing the inffuence function in the subspace follows:
IGaussian (+++)
m Ival
1
m
n P
Pgint (
)
pg
71
m
(mm) P (PHP)) Pg
(2)
m
where P E Rexd is random Gaussian projection matrix. Here, represents the dimensionality of the compressed subspace and < d. While influence function computation becomes more efficient in terms of computation and memory complexity, the use of a projection matrix introduces additional computing and memory overhead for compression (Choe et al. 2024). Specifically, computing a gradient via backpropagation has cost of Odd), whereas projecting it into lower- dimensiinai subspace using P incurs a cost of (rd)). Even efficient projection methods, such as FJLT (Ailon & Chazelle, 2009) have a higher computational complexity than O(d), making the compression process more expensive than the gradient computation itself. Additionally, storing these compression maps can incur memory overhead which exceeds the memory usage of the model itself. These limitations highlight the need for new compression strategy for influence function computation, which minimizes both computing and memory costs, while preserving koy information necessary for reliable influence estimation.
3 Method
To address the computational and memory challenges associated with influence function computation, we propose a novel approach that leverages dropout as a gradient compression mechanism. We demonstrate that the influence of training data on a small subset of parameters can effectively reflect its influence on the entire parameter space. Unlike traditional compression methods, which require random Gaussian matrix as the compression map or use PCA to obtain the important components, incurring significant memory and computational costs, our method simply drops a random subset of gradient entries. As shown in Figure [ this technique reduces the dimensionality of gradients without incurring the additional memory and computational overhead typically associated with the compression process.
3.1 Dropout as a Compression Mechanism
Dropout is a widely used regularization technique in deep learning (Srivastava et al., 2014). where a subset of model parameters or activations is randomly set to zero during training. We apply a similar approach the gradient vectors during influence function computation, compressing the gradient by retaining only a small subset of its entries. Let g Rd represent the gradient of the loss function with respect the model parameters for data (x, y), iee. g Vol (v, fo(x)) 0=0-" To compress the gradient, randomly sample entries of the gradient g. Mathematically; this process is equivalent to using a binary matrix I E gg compress the gradient g and get the compressed one g:
g - ig
Each row of i has exactly one entry equal to 1, while all other entries are 0, and there is only one non-zero entry each column:
d-1
i (0xd)
Ii,=1 Vie 10. 1}.
(3)
1=0
Specifically, I. = indicates that the j-th entry of the gradient is retained.
It is important to note that the introduction of used for theoretical analysis, and in practice we do not need to explicitly construct this. This avoids unnocessary computational and memory overhead, thereby simplifying the implementation while maintaining efficiency. For detailed implementation pipeline and step-by-step algorithmic description, please refer to Algorithm T in Appendix A.2
4

---

Published in Transactions on Machine Learning Research (02/2026)
After getting compressed gradients g, use compressed gradients for influence function computation:
1
I Dropout (zir)
H H-gg..
Dir
m7
j=1
1
((HHT)) -igh
(4)
The matrix H is the Hessian/Gauss-Nevvv Hessian caleulated using compressed gradients.
3.2 Efficiency Comparison
Even though traditional gradient compression methods, such as random projection (Johnson et al. 1984] used in TRAK PPark al., 2023), reduce the complexity of computation of influence function, they rely on explicit projection matrices to compress. That will introduce significant memory and computational overhead, because these hods use dense projection matrices with memory complexity O(rd) and computal ional complexity dominated by matrix-vector multiplication, which is O(rd) for each gradient. PCA requires even more resources to obtain the principal components. In contrast, our dropout compression method avoids the need for explicit projection matrices, and reduce memory and computational costs to (rr) as only r entries of gradients are sampled and stored.
Other efficient influence function computation methods, such LiSSA (Agarwal et al. 2017 and LOGRA (Choe et al. 2024), employ stochastic iterative approaches or Kronecker product for gradient computation, respectively. While these methods reduce the computational cost of iHVP, they still require expensive iterative algorithm (Klochkov & Liu 2024) or are hard to expand to all deep learning architectures (Kwon
at all 2023 Grosse et al. 2023). The comparison of computational and memory costs of different hods
for gradient compression and influence function computation are detailed in Appendix A.3
3.3 Error Analysis
While the dropout-based compression method in equation Jfffers a more efficient alternative for computing inffuence functions than the standard Gaussian approach equation 2 it is important to understand the trade- offs in accuracy. Intuitively, Gaussian projects compresses gradient information into a lower-dimensionall subspace, whereas dropout retains only a random subset of gradient entries, potentially discarding significant information. To address this, we thcoretically analyze the upper bound of the error incurred by our method. For both methods, the compression error is defined as the difference I(= IGaussian Dropout (= The spectral norm of this error is expressed as:
1
[[(+)t] IGaussian or Dropout (2 11
Peal
AHg
Σ
||1
AH
9
8
(5)
m
2
and it is governed by the term ||H where AH represents the difference between the full inverse Hessian and its compressed counterpart.
Our theoretical analysis indicates that the error introduced by Caussian-based compression can actually have higher thcorctical upper bound compared to dropout-based approach. Specifically, Gaussian-basod compression error is bounded by O (d + dd' (H)), while dropout-based compression error bounded by O (amax H)). These results provide a worst-case stability guarantee, ensuring that the approximation remains robust even at high compression rates. We provide the formal statements in Appendix A.4 This theoretical framework supports the utility of dropout as lightweight and practical tool for influence esti- mation, particularly when computational resources are constrained.
5

---

Published in Transactions on Machine Learning Research (02/2026)
Table 1: Performance of mislabeled data detection on some GLUE benchmarks (MRPC, QNLI, QQP, SST2) using various methods for inftuence function computing. The reported results are avcraged over independent runs; with slandard deviations shown as subscripts. The best results (excluding Orig) are highlighted in bold, and the second-best results are underlined.
Rank=2
Rank=8
Method
MRPC
QNLI
QQP
SST2
MRPC
QNLI
QQP
SST2
Orig
0.832p.005
0.794
0.807
0.802..028
LiSSA
0.665g.018
0.498g.014
0.578p.079
0.5099.056
0.617g.075
0.500p.015
0.609
0.4788.046
Hessian-free
0.687p.011
0.639b. 128
0.6666.016 0416
0.8149.116
0.6919.009
0.697o.132
0.627
0.6100.184
DataInf
0.753p.144 0.789p.013
0.723g.076
0.934 0.06
0.794g.022
0.7936.100
0.821 0.785p.059
Gaussian
0.786u. 0.823p.012
0.793g.0s .031
0.933
0.8449.026
0.833p.082
0.821 0.006
0.828 206
PCA
0.833p.013
0.792..125
0.803g.030
0.932o.007
0.850p.024
0.832 0.841..061
0.827 n.20
Dropout (Ours)
0.791p.u37 0.825p.010
0.791..08
0.9350.007
0.842o.025
0.836p.073
0.825
0.838o.1e
4 Experiments
In this section, we evaluate the effectiveness of our method: Using dropout as compression tool for influence function computation, in terms of accuracy and efficiency, key factors in real-world data attribution tasks. We conduct three experiments to evaluate the effectiveness of our approach: (1) mislabeled data detection 4.1 which uses influence function to identify mislabeled data points in a noisy training dataset; (2) model retraining 4.2 which identifies the most influential training examples and retrains the model either with only these data points or with them removed to observe their impact on model performance; and (3) cross-source inftuential data identification 4.3 which investigates whether influential training examples typically originate from the same source as their corresponding test examples. To comprehensively evaluate our method, we start with relatively small experimental setups and then scale up to pallion- models. This allows us to assess how well our method generalizes across settings and to demonstrate its scalability. Additional experimental details are provided in Appendix A.5
4.1 Mislabeled Data Detection
Mislabeled data points often negatively impact a model's performance. It is expected that the influence values of these mislabeled data points will be larger than clean data points, as their unweighting tends to increase the testing loss.
In this experiment, we use five binary classification datasets from GLUE benchmark WWag et al. 2018) and synthetically generate mislabeled training data points similar to (Kwon et al. 2023). flipping the binary label for 20% of randomly selected training data points to simulate the situation where a part of data points are noisy. We use the RoBERTa model (Liu et al.) 2019) and fine-tune the model on those noisy dataset using LoRA (Hu et al. 2022) with 2-rank and 8-rank separately. As for the baselines, we investigate the performance of five efficient methods, including three approximation methods and two compression mothods as well as the Orig influence function equation [ For approximation methods, we consider LiSSA (Koh) & Liang 2017) with 10 iterations, Hessian-free which only computes the dot product of gradients (Pruthi et all 2020), DataInf which uses an closed form approximation version of influence function (Kwon et al., 2023 For compression methods, we consider Gaussian, which uses random Gaussian matrix to compress gradients similar to (Park et al., 2023), , and PCA that uses PCA to obtain principal components to compress gradients. Some details of these methods are attached in Appendix A.1 For compression methods, including Gaussian, PCA, and Dropout, we use r = 16 for both 2-rank and 8-rank LoRA.
For evaluation metrics, we use the area under the curve (AUC). The AUC quantifics the ability of the inffuence function to distinguish between mislabeled and clean data points. Specifically, it measures the probability that a score selected from a class of mislabeled data is greater than that of class of clean data.
6

---

Published in Transactions on Machine Learning Research (02/2026)
An inftuence function that reliably assigns larger infiuence values to mislabeled data points will achieve high AUC soore, reflecting its effectiveness in identily mislabeled data.
Results. Table compares the mislabeled data detection performance of various influence function com- putation methods using LoRA with ranks 2 and 8. Results are averaged over 5 independent runs. Dropout achieves comparable, and in some cases superior detection performance compared to Gaussian and PCA, despite requiring no additional computation or memory overhead for compression. Furthermore, there is a consistent trend indicating that compression-bssee methods outperform approximation- based ones. Similar to findings in Kwon et al. (2023), best performance (Orig). This is potentially because the whole gradients contain some redundant informa- tion which has negative impacts on the performance. In terms of rumning time, compression based met demonstrate superior computational efficiency for iHVP computation. For example, on QNLI dataset with 8-rank LoRA, compression methods take an average of 4.83 seconds, whereas DataInf takes 13.38 seconds. More results on time usage of iHVP computation of these methods across various benchmarks is provided in Appendix A.5.2
observe that original uncompressed gradients do not always have the
4.2 Model Retraining
The retraining process begins by identifying the most influential data points. The new training dataset constructed by these influential data points or removing these highly influential data points from the original dataset. The performance of the retrained model is evaluated on the original test dataset. Retraining could demonstrate the critical role of these influential data in model's learning process and the effectiveness of methods used to identify these influential data.
4.2.1 Small-Scale Setups
We initiate this experiment with small-scale setups: (1) ResNet-9 (He et al. 2016 with CIFAR-10, in which we train a ResNet-9 model from scratch using randomly selected subset containing 10000 data points and evaluate on a test dataset containing 256 data points by accuracy. Then we remove a specific amount of influential data from the training dataset and retrain the model. Large performance decrease indicates greater effoctiveness of the method in identifying the influential data. (2) GPT-2 (Radford et al., (019)) with CNN daily mail, in which we full fine tune a GPT-2 using 1000 text samples and evaluate on a test dataset containing 512 text samples by perplexity. Then we only use specific amount of influential data to retrain the model. Large perplexity decrease indicates greater effectiveness of the method in identifying the influential data. We use influence function to compute the influential score of each training data and rank them by these score. On these benchmarks, we compare Dropout against baselines include; Random Ranking, which randomly rank training data; LiSSA (Koh & Liang 2017) uses an itcration method to get influential scores; DataInf (Kwon et al.) 2023) uses a form of approximated influence function; Hessian- free (rrtti et al.] 22200) computes the dot product of gradients directly; LOGRA (Choe et al. (024)) uses Kronecker product for gradients computation and compression; Gaussian uses random Gaussian matrix to compression gradients; FJLT, an efficient compression method proposed in Ailon & Chazelle (2009); ;; and PCA uses PCA to obtain principal components and do compression. Some details of these mothods are attached in Appendix A.1 For LOGRA we use Tin = Tout VF = 64, and for Gaussian, PCA and Dropout we use r y= 64 to do compression.
Results. The retraining results of ResNet-9 and GPT-2 are in Table 2 and Table 3 separately. We observe that Dropout achieved performance comparable to or better than other methods across both settings. LOGRA achieves strong performance in ResNet-9, it does not perform as well in GPT-2. We hypothesize that this is due to the Kronecker structure used in LOGRA, which may not generalize well across architectures. In terms of efficiency, compression methods demonstrate impressive performance in computing iHVP same as previous. Moreover, the efficiency of gradients compression becomes more crucial when dealing with large- scale models. Notably, the Dropout excels in efficiency during the compression process. For example, in GPT- experiment, FJLT uses an average of 453.87 seconds to compress all gradients, in contrast, Dropout requires only 9.76 seconds, representing 46x speedup in the compression process. Additionally, Dropout eliminates the extra memory overhead associated with storing the full compressing map, which is a requirement for
7

---

Published in Transactions on Machine Learning Research (02/2026)
Table Accuracy (%) of ResNet-9 on the test dataset after removing a specific amount of the most in influential training data from the training datasct. The reported results are averaged over 5 independent runs, with standard deviations shown subscripts. The testing accuracy of the model trained on the full training dataset is 78.83%. The best results are highlighted in bold, and the second-best results are underlined.
Methods
5%
15%
25%
35%
Random Ranking
79. 79.141.00
76.682.39
75.66o.42
74.060.9
LiSSA
78.01 78.01
75.351.79
69.732.84
65.161.48
Hessian-free
78.01
74.30p.97
70.270.81
65.51
DataInf
77.730.39
73.280.83
69.301.29
63.91g.45
LOGRA
76.09o.74
71.33 1.53
67.97 1.22
64.30p.87
Gaussian
77.811.18
72.731.59
68.40
63.32
PCA
77.621.10
72.34
69.061.21
63.
Dropout (Ours)
p6.88
72.27 1.48
68.871.02
62.30o.57
Table 3: Testing perplexity of GPT-2 trained with specific amount of the most influential training data.
The reported results are averaged over 5 independent runs, with standard deviations shown subscripts. The testing perplexity of the model trained on the full datasct is 13.67. The best results are highlighted in bold, and the second-best results are underlined.
Methods
5%
10%
15%
20%
Random Ranking
17.43p.20
17.38p.07
17.100.04
17.05n.05
LiSSA
17.40p.11
17.31o.08
17.11o.06
17. 17.05p.or
Hessian-free
17.43p.26
17.13g.05
16.91p.us
16.90a.0e
DataInf
18.04..6
17.119.co
16.91 p.02
16.87 .03
LOGRA
17.65p.43
17.249.17
17.03o.13
16.89
FJLT
17.25 17.25p.06
17.140.05
16.91 16.98o.10
Dropout (Ours)
17.27
17.10p.09
16.95
16.84g.05
compression methods like Gaussian, FJLT, PCA and LOGRA. The time consumption of these methods is detailed in Appendix A.5.2
4.2.2 Large-Scale Settings
We now evaluate our approach for data attribution in billion- parommter models, using Pythia-1.413 and Pythia-6.9B (Biderman et al., 2023). For Pythia-1.4B, we perform data attribution subset of OpenWeb- Text (OWT) (Gokaslan et al. 2019) with 2,000 training and 200 testing text sample. For Pythia-6.9B, we use a heterogeneous dataset with six sources, including CNN daily mail, math reasoning, wikisql, java code, and others, using 10,700 training and 300 testing samples.
In both cases, we fully fine-tune the model and then remove the top-k percent most influential data from the training datasct and retrain the model. A larger performance drop indicates more effective method for data altribution. Notably, the gradients used in the computation of influence function is the same size the model itself. This makes most muudient-based methods, including Hessian-free impractical for billion- scale models due to the prohibitive resources required to store ( per-example gradients. Therefore, we evaluate only the methods feasible under our setup: Random Ranking, LOGRA with Tin = Tout vF 64, and Dropout with r= 128.
Results. Table presents the retraining results for Pythia 1.4B and Table 5 for Pythia 6.9.. We should realize that when the training dataset becomes large, the marginal contribution of each individual data point diminishes. As result, removing small subset of data has only marginal impact the model's
8

---

Published in Transactions on Machine Learning Research (02/2026)
Table 4: Testing perplexity of Pythia 1.4B on test dataset after removing a specific amount of the most influential data from the training dataset. The reported results averaged over 5 independent runs. The testing perplexity of the model trained on the full training dataset is 25.23. The best results are highlighted in bold, and the second-best results are underlined.
Methods
10%
20%
35%
40%
Random Ranking 25.52
25.91
26.28
26.49
LOGRA Dropout(Ours)
25.68
25.89
26.62
26.79
25.51
26.05
26.55
26.80
Table 5: Testing perplexity of Pythia 6.9B on test dataset after removing a specific amount of the most influential data from the training dataset. The reported results are averaged over independent runs. The testing perplexity of the model trained on the full training dataset is 2.54. The best results are highlighted in bold.
Methods
10%
20%
35%
40%
Random Ranking
2.54
2.55
2.55
2.56
Dropout(Ours)
2.55
2.56
2.57
2.58
performance when fine-tuning Pythia 6.9B on over ten thousands of text samples. Nonetheless, our mothod still shows slightly better results under these conditions.
4.3 Cross-Source Influential Data Identification
To assess whether our method can effectively identify the influential training examples in large-scale set- tings, we apply the influence function with Dropout compression to examine whether the identified most influential samples for training originate from the same class or data source as their corresponding validation examples. We conduct. this experiment using the Pythia-6.9B model, fine-tuned on a dataset composed of six heterogeneous sources, including CNN daily mail, math reasoning, wikisql, and others, containing 10,700 training and 300 testing samples, consistent with the previous setup. Importantly, our method remains computationally feasible under this setup, as it avoids the need save full per-example gradients large compression matrices, making it both memory and compute efficient.
Results. Intuitively, for a large model like Pythia trained on heterogeneous data sources (such as CNN Daily Mail, Math Reasoning, WikiSQL, and others), a successful data attribution method should be able trace given test sample back to training samples that originate from the same source. To quantify this, we measure the source alignment using Top-1 and Top-3 accuracy. Top-1 calculates the proportion of test samples where the most influential training sample originates from the same source. Top-3 is stricter metric that calculates the proportion of test samples where all three of the most influential training samples originate from the same source. Given our six-source datasct, the Top-1 and Top-3 accuracy for Random Ranking baseline (randomly selecting training samples) is 16.7% and 0.7%, separately. Our method's performance (see Table 6 substantially exceeds this, confirming that dropout preserves essential semantic signals even at high compression ratios. Table reports the results and examples of identified influential training data are shown in Appendix A.5.3
5 Analysis
In this section, we provide an additional error analysis to complement the results in Section 3.3 Our
thcoretical results in Section 3.3 indicated that the spectral norm of the error for Dropout possesses a tighter upper bound compared that of the Gaussian approach. While these bounds offer a worst-case stability guarantee, they do not necessarily reflect the expected error the variance of the estimates across different

---

Published in Transactions on Machine Learning Research (02/2026)
Table 6: Proportion of top-1 and top-3 most influential training examples that belong the same class the corresponding test example.
Methods
top-1
top-3
Random Ranking
0.167
0.007
Dropout (Ouro)
0.831
0.642
Dropout Mean
Gaussian Bound
Gaussian Area
Random
F=2
r=4
r=8
r=16
Detection Rate (16)
20
380
Inspection
Inspection
Figure 2: Mislabeled data detection on COLA (one benchmark GLUE) with 2-rank LoRA. We compare Orig with gradient compression methods Gaussian and Dropout with different compression size r (2, 4, and 16). For Gaussian and Dropout, the shaded regions represent the range (min/mast of performance, and the solid lines indicate the average detection performance across independent runs.
random seeds. Intuitively, Dropout may cause significant and random information loss, potentially leading to larger and unstable error, especially when the compression size is small. Therefore, is valuable to investigate how influence function performance with Dropout varies with different compression size r. For this, we use mislabeled data detection as case study.
In Figure D. the bounds represent the best and the worst performance observed across runs. Empirical results indicate that while Dropout exhibits higher variance than Gaussian at extreme compression levels (e.g. = 2), this instability diminishes rapidly as the compression size r increases. In particular, for r 8 and r 16, our method exhibits significantly reduced variance, matching the stability of the Gaussian baseline while maintaining comparable average detection rate. This demonstrates that once minimal threshold of the parameter space is sampled, Dropout effectively captures the necessary signal for reliable influence ostimation. Furthermore, while the upper bound provided in Section 3.3 may be loose, it still provides a meaningful indicator of robust performance.
6 Related Works
Data attribution aims quantify and understand the impact of each training data point the performance of the model (Albalak et al. 2024). In Ghorbani & Zou (2019), the authors proposed Data Shapley, which quantifics the value of cach training data point by leveraging shapley value as metric. Despite its conceptual appeal, Data Shapley is computationally prohibitive, particularly for modern large-scale ML models (Jia gt all. 2019). Furthermore, several works proposed frameworks to do data attribution by retraining a model multiple time evaluate the impact of some data point [myys et al., 20022 Park et al. (202)). Although efforts such as (Park et al.] (023)) strive to balance computational cost and effectiveness, the necessity of retraining models remains significant drawback, especially for resource intensive deep learning applications.
10

---

Published in Transactions on Machine Learning Research (02/2026)
Influence function is another approach to data attribution, adapted from robust statistics (Law 1986 Ham-
pel. 1 and introduced to the deep learning context in KKoh & Liang (2077); ; Koh et al. (2019) It
addresses the counterfactual question: how would the model's parameters or loss change if a specific train-
ing data point were removed? While inftuence function offers a theoretically groundod framework for data altribution, the high computational cost has limited its applicability to large-scale models. To miligate the computational burden of influence function, various methods have been proposed. In Koh & Liang (2017): Agarwal al. (2017), the authors introduced LiSSA, which approaches iHVP computation iteratively, reduc- ing the cost of influence function computation. Other approaches include LOGRA (Choe et al. 2024 and EK-FAC (Grosse et al. 2023) proposed using Kronecker product for gradients computation, and compress gradients using Kronecker product structure or eigen decompositions for efficiency. However, the Kronecker product structure cannot be universally applied to all deep learning models and eigen decomposit ions will be expensive in large-scale matrix. Datalnf (Kwon et al., 2023) proposed an approximation of the influ- ence function by approximating the inverse Hcssian matrix. However, this method introduces crrors which scale quadratically with the size of the model. Consequently, DataInf is primarily optimized for low-rank adaptation (LoRA) (Hu et al.) p022) and becomes computationally intractable for hull-parameter attribu- tion in large-scale models. (Zhou et al. (0024) proposed method which approximates the Hessian matrix using Generated Fisher Information Matrix (GFIM). This approach relies on strong assumption that each column of the gradient matrix is independent and has a zero mean, which often fails to hold in practice.
7 Conclusion
In this work, we demonstrate that the influence of training data on a small subset of parameters can ef- fectively reflect its influence on the entire parameter space. Building on this, we introduce dropout as a compression tool to enable efficient infiuence function computation, addressing the computational and memory challenges that. hinder the application of influence function in large-scale models. Our approach leverages this simplicity and scalability of dropout to selectively retain gradient information, thereby signif- icantly reducing computational and memory overhead compared to methods relying on dense compression map such as gaussian compression.
Through thcoretical analysis, we demonstrated that the error upper bound of influence function with dropout compression is smaller than gaussian compression methods. Our empirical results validate these findings, showing that dropout compression method could achieve comparable or superior performance in data at- tribution while maintaining high efficiency. This work highlights the potential of dropout lightweight, efficient, and practical compression tool in influence function computation, paving the way to extending application of influence function in large-scale artificial intelligence systems.
8 Limitations
While we demonstrate the potential of dropout as an efficient gradient compression method for influence function computation, several limitations remain to be addressed. Our method does not alleviate the resource requirements for gradient computation. Computing gradients for all data points, particularly in large-scale models and datasets, remains a bottleneck. This limitation highlights the need for further optimizations to make influence function methods more resource-efficeent.
References
Abubakar Abid, Maheen Farooqi, and James Zou. Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAT/ACM Conference AI, Ethics, and Society, pp. 298-306, 2021.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXio preprint arXie:2303.088 08774, 2023.
Naman Agarwal, Brian Bullins, and Elad Hazan. Second-order stochastic optimization for machine learning in linear time. Journal of Machine Learning Research, 18(116):1-44 2017.
11

---

Published in Transactions on Machine Learning Research (02/2026)
Nir Ailon and Bernard Chazelle. The fast lindenssaas transform and approximate nearest neigh- bors. STAM Journal computing, 39(1):302-33) -222, 2009.
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models. arXiv preprint arXiv:2402. 16827, 2024.
Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger B Grosse. If influence functions are the answer, then what is the question? Advances in Neural Information Processing Systems, 15:17953-17967 2022.
Zhi-Dong Bai, Yong-Qua Yin, al. Limit of the smallest eigenvalue of large dimensional sample covariance matrix. Ann. Probab, 21(3):1275 1294, 1993.
Yogesh Balaji, Mohammadmahdi Sajedi, Neha Mukund Kalibhat, Mucong Ding, Dominik Stöger, Mahdi Soltanolkotabi, and Soheil Feizi. Understanding overparameterization in generative adversarial networks. arXiv preprint Xix:21110 056655, 2021.
Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions in deep learning are fragile. arXiv preprint ar Xiv:20006 14651, 2020.
Stella Biderman, Hailey Schoclkopf, Quentin Gregory Anthony, Herbic Bradley, Kylc O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Sang Koun Choc, Hwijeen Ahn, Juhan Bac, Kewen Zhao, Minsoo Kang, Youngscog Chung, Adithya Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, et al. What is your data worth to gpt? Ilm-scale data valuation with influence functions. arXiv preprint arXiv:2405. 13954, 2024.
William Fedus, Barret Zoph, and Noam Shaveer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23110): 2022.
Tim Fischer, Chris Biemann, et al. Large language models are overparameterized text encoders. anXiv preprint XrXiv:2410. 14578, 2024.
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. International conference on machine learning, pp. 2242 2251. PMLR, 2019.
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus, 2019.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llaaa 3 herd of models. arXiv preprint 24X7:2240.21788 2024.
Roger Grosse, Juhan Bac, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. Xiv preprint arXin:2308.03200 2023.
Frank R. Hampel. The influence curve and its role in robust estimation. Journal of the american statistical association, (0(346))38333331 1974.
David A Harville. Matrix algebra from statistician's perspective, 1998.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
12

---

Published in Transactions on Machine Learning Research (02/2026)
Yuzheng Hu, Pingbang Hu, Han Zhao, and Jiaqi Ma. Most influential subset selection: Challenges, promises, and beyond. Advances in Neural Information Processing Systems, 17:119778- 119810, 2024.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, al. survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):
1 1-55, 2025.
Peter J Huber and Elvezio M Ronchetti. Robust statistics. John Wiley & Sous, 2011.
Andrew llyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry. Datamodels: Understanding predictions with data and data with predictions. In International Conference on Machine Learning, pp. 9525 9587. PMLR, 2022.
Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J Spanos. Towards efficient data valuation based on the shapley value. In The 22nd International Conference on Arlificial Intelligence and Stalislics, pp. 1167 1176. PMLR, 2019.
William B Johnson, Joram Lindenstrauss, et al. Extensions of lipschitz mappings into a hilbert space. Contemporary mathematics, 26(189-206):1, 1984.
Yegor Klochkov and Yang Liu. Revisiting inverse hessian vector products for calculating influence functions. arXiv preprint arXiv:2409.11333 13577, 2024.
Pang Wei Koh and Percy Liang. Understanding black- box predictions via influence functions. In International conference machine learning, pp. 1885-1894. PMLR, 2017.
Pang Wei W Koh, Kai-Siang Ang, Hubert Teo, and Percy S Liang. On the accuracy of influence functions for measuring group effects. Advances in neural information processing systems, 32, 2019.
Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora-tuned Ilms and diffusion models. arXiv preprinl ar arXi:2231.00000,, 2023.
John Law. Robust statistics the approach based on influence functions, 1986.
Xiaochuan Li, Zichun Yu, and Chenyan Xiong. Montessori-instrruct: Generate influential training data tailored for student learning, arXiv preprint arXiv:2410.14208, 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint ar Xrx::21000 079587 2021.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 11.:2442.19437, 2024.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemooyr, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arxiv preprint 1rXto:1907.16690 2019.
James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21 (146):11-76, 2020.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359 17372, 2022.
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar, Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.00 0,5.222,0..0229,, 2024.
Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclere, and Aleksander Madry. Trak: At- tributing model behavior at scale. Xiv preprint ar arXio:2303.12 14 2023.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33:19920-19930 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models unsupervised multitask learners. OpenAT blog, 1(8):9, 2019.
Levent Sagun, Leon Bottou, and Yann LeCun. Figenvalues of the hessian in deep learning: Singularity and beyond. orXiv preprint orXio:1611.074400 2016.
Levent Sagun, Utku Evei, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of over-parametrized neural networks. arXiv preprint arXiv:1706. 2ee11006.00444 7445, 2017.
Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up influence functions. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 8179-8186, 2022.
Jack Sherman. Adjustment an inverse matrix corresponding to changes the elements of a given column or row of the original matrix. Annu. Malh. Stulist., 20:621, 1949.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ila Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929 1958, 2014.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understandigg arXiv preprint arXir:180,007661, 2018.
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. In NewrIPS, 2023.
Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, and Jundong Li. Knowledge editing for large language models: A survey. ACM Computing Surveys, 53(3:11337, 2024.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data targeted instruction tuning. arviv preprint arXiv:2402.0 04333, 2024.
Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. Openmoc: An early effort on open mixture-of-experts language models. arXiv preprint arXiv:2402. 01799, 2024.
Zichun Yu, Spandan Das, and Chenyan Xiong. Mates: Model-aware data selection for elficient pretraining with data influence models. Advances in Neural Information Processing Systems, 37:108735-108755 2024.
Xinyu Zhou, Simin Fan, and Martin Jaggi. Hyperinf: Unleashing the hyperpower of the schulz's method for data influence estimation. XrXiv preprint or 2iv:2 10. 05090, 2024.
14

---

Published in Transactions on Machine Learning Research (02/2026)
A Appendix
A.1 Efficient Methods for Influence Functions.
LiSSA Agarwal et al. (0117) proposed an iterative method for computing iHVP (H + Ala) which was later utilized by (Koh Liang 2017) to compute the influence function. For 8u = uv, LiSSA recursively
computes the following equation: 8,+1 v + (Ia - (H + (T))ss Agarwal et al. (2017) proved that if
+ AIa Id. s; will converge the iHVP (H + XIa) -1 as increases. Then, the iHVP could be approximated as:
(H + A
(6)
and the influence function could be calculated using this approximation iHVP:
I( - -TT Tt.
(7)
Here of Volluir Sa( 1==-
is the gradient of the loss function calculated at the k-th training data m Eド= P 1 m. Ej= Vol(v fa(s(aa)) 0==-
point with respect to parameters of the model, and -
is
the average of gradients of the loss function calculated at evaluation dataset with respect to parameters of the model.
Datalnf Kwon et al. (202)) proposed an approximated version of the influence function. The key approxi- mation in DataInf involves swapping the order of matrix inversion and the averaging in (H + AI))= Using this approximation, the inverse Hessian matrix becomes:
Σ Ligh RT + +Maa)
Σ Ligh + +Ala)
(8)
n
1 ad
Firger KT
(Ia
(9)
+ g g
k
where g is the gradient of the loss function calculated at the k-th training data point with respect to parameters of the model. The Anterman-Morrisee formula (Sherman 1949 is utilized to compute the matrix inversion in equation Based on this approximation, the inffuence function can be computed efficiently, reducing the operation to linear complexity.
LOGRA Grosse al. 2023 Choe et al. (2024) proposed using Kronecker product to approximate gradients and (Choe et al. 2024 compresses gradients make use of Kronecker product structure. For the g-th layer of deep learning model with parameter 0q. let h2 represent the output and 9q represent the pre-activated output of the g-th layer. The gradient of loss function evaluated on (z, y) with respect to fg is given as
the following:
Vy, fo(x)) = h1 &
V ge (yy.
(10)
where represents the Kronecker product. LOGRA (Choe al. 2024 imposes additional Kronecker product structure the projection matrix P as follows:
PV e. So(x)) = (Pmn @ Pout)(hg-1 @ V (I, = Pi @ Pout V. My, fo(x)),
(11)
(12)
where Pi E Xdu, Pout E Fou Roouixdon ddout equation 12 LOGRA first projects forward and backward activations onto low-dimensional space using Pm and Pout respectively, and then reconstructs projected gradient directly from these projected activations. It important note that dn dout vd and Tin Tout Vr, making it be easy to use relatively large compression size
A.2 Algorithm Details
In this section, we provide the detailed implementation pipeline for efficient influence function estimation using Dropout compression. Algorithm explicitly outlines the procedure, demonstrating how the Dropout
15

---

Published in Transactions on Machine Learning Research (02/2026)
compression method defined in Section 3.1 is realized in practice through efficient indexing. The procedure consists of three main phases: i) Mask Generation: We first sample set of indices M of size This set represents the active parameters that are retained. Unlike dense projection methods (e.g.. Gaussian) that require storing a projection matrix of size r x d, our method only requires storing the list it Gradient Compression: For both validation and training data, we compute the full gradient but immediately discard all entries not M. This ensures that the storage requirement for each data point is strictly O(r). iii) Influence Estimation: The inverse Hessian vector product (iHVP) is computed entirely within the oiw-dimeniionll subspace Rxxr, The inversion of the compressed Hessian has complexity of O(r9), which is negligible for small r. By directly operating the indices, we avoid the overhead of sparse matrix multiplication, achieving true O(r) memory and computational efficiency.
integers.
Algorithm Efficient Influence Function Estimation via Dropout Compression
Input: Training dataset Der, Validation dataset Deats Model parameters 0*, Compression size r, Damping
Output: Approximate Influence scores (- 1: // Step 1: Generate Compression Mask
2: Randomly sample set of indices M C {l....,d} d} such that [M] =
3: This corresponds to the non-zero diagonal entries of Í'i.
// Step 2: Compress Validation Gradient
6: Store only sampled indices: Ival + Joat E( Be Vol(yral: lo=
Compute full validation gradient: Gool f- m
(Memory cost: 0(r)
// Step 3: Compute Compressed Hessian Inverse
Compute Hessian sub-matrix on indices M: H.
Compute inverse term: Hine (H AI,) {Compute cost: ((r²)}
10: // Step 4: Compute Influence for Training Data
11: for each 2 E Dr, do
12: Compute gradient gir-
13: Compress immediately: fir + (ir(M)
(Memory cost: 0(r))
14: Compute score: I(2(rr) + - Tvat Hinugrr
{Compute cost: (
15: end for
16: return (I(=tt))x
A.3 Complexity Comparison.
Table presents comparison of the computational and memory complexity of iHVP computation for influence estimation across Orig, LiSSA, DataInf and gradient compression methods with compressed size r, such as Gaussian, LOGRA, FJLT and Dropout. While gradient compression methods cannot reduce the complexity to linear, the compression size r could be very small, making these methods efficient in practice.
Although the computational and memory complexity involved iHVP computation are the same for gradient compression methods with the same compressed size r, the complexity of compression process itself differ. Table provides a comparison across Gaussian, LOGRA, FJLT and Dropout.
A.4 Proof of Theorems.
In this section, we provide the formal statements and detailed proofs for the error bounds discussed in Section 3.3 We begin by introducing the fundamental lemmas and identities required for analysis.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Table 7: Comparison of computational and memory complexity involved in iHVP computation for influence function estimation across Orig, LiSSA, DataInf and gradient compression methods with compression size (e.g. Gaussian, LOGRA, FJLT and Dropout). The number of parameters in the model is d and the number of data is n.
Method
Computational Complexity
Memory Complexity
Orig.
O + d (ndd)
(
LiSSA
(
DataInf
((nd)
(
Compressing Methods
( + r
0 (r²)
Table 8: Comparison of computational and memory complexity involved in performing compression across Gaussian, LOGRA, FJLT and Dropout. The number of parameters in the model is d, the number of data is n, and the compressed size is r.
Method
Computational
Complexity
Memory Complexity
Gaussian
(nnrd) (nyrr)
O(rd)
LOGRA
O(vrd)
FJLT
(d d) (nrr
(d))
Dropout
O(r)
Theorem A.1: Woodbury Matrix Identity (Harville, 1998)
Given a square invertible n x matrix A, an n. matrix U, and a kxn matrix V, let B = A+UV. If (1+ + VA-UU is invertible, then:
-ー -0
(13)
The thcorem of woodbury matrix identity A.1 not only allows cheaper computation inverses but also provides closed form expression of matrix inversion.
Theorem A.2: Convergence of Extreme Singular Values (Bai et al. 1993)
Let A be a l x n random matrix whose entries are independent copies of a random variable with zero mean, unit variance, and finite fourth moment. k, n + oo with n/k + x (0, 1], then:
maxe (A) 1 + vie almost surcly. Vi
(14)
Theorem A.2 implies that for n random Gaussian matrix, the largest singular value Tmax converges O(VR).
17

---

Published in Transactions on Machine Learning Research (02/2026)
A.4.1 Theorem for Gaussian Compression
Theorem A.3: Gaussian Compression Error Bound
For gaussian based compression method in equation 2 if A PT PH is invertible and the dimension of 9 exceeds the size of training datasct n, the spectral norm of AH, i.c. |A 2
bounded by:
O + de omax H));
where max H) denotes the largest singular value of H.
Proof. Using the Woodbury identity in equation 13 we express the error matrix AH as:
AH (A& + H 1 P (AI, PHPT)-1P
1 (Aa + H) H)- | PT P P PHP P.
(15)
Applying the triangle inequality and properties of the spectral norm:
2 & [I + 1
X | Pl + X
P PH + P P'PH>>"|||1"" Pll
1
max (PTP) P)
maxe (P P) max H
(16)
min (M + H)
do (Ma + P PH)
Since the model parameters exceed the dataset size n, H is not full rank, so min (Al + H) = X. From Theorem A.2. we have mmax [ P) S Omax (P)2 d. Substituting these into equation 16
+ d d a H
A
O O(d + d o max H)).
(17)
A
X2
A.4.2 Theorem for Dropout Compression
Theorem A.4: Dropout Compression Error Bound
For dropout based compression method in equation if the dimension of A exceeds the size of training dataset n, the spectral norm of AH, i.e. ||AH|] is bounded by:
O (H)),
where Tnnax (H) denotes the largest singular value of H.
Proof. Let I be the binary selection matrix defined in equation 3 Using the Woodbury identity:
AH (Ala + H)
iH + ITi.
(18)
Taking the spectral norm:
1
T Fmax [T
Tmax [ (i max (H)
|H 2 5
+
mmin (Ala + H)
A
Aa (Al ÍT iH)
max (H)
(19)
2
18

---

Published in Transactions on Machine Learning Research (02/2026)
Table Hyperparameters used for model training in experiments.
RoBERTa
ResNet-9
GPT-2
Pythia 1.4B Pythia 6.9B
Optimizer
AdamW
SGD-M
AdamW
AdamW
AdamW
LR Scheduler
Lincar
Lincar
None
None
Cosine
Learning Rale
3 x 10-1
0.4
5 10-5
2 x 10
5 x 10-5 0.01
Weight Decay
None
5 x 10
0.01
0.01
Batch Size
32
64
64
16
2 1024
Sequence Length
None
None
256
128
Epochs
10
3
Table 10: Average time usage (seconds) computing iHVP in mislabeled data detection tasks.
Rank=2
Rank=8
Method
MRPC
QNLI
QQP
SST2
MRPC QNLI QQP SST2
Orig
319.22
384.26
335.40
418.29
LiSSA
43.05
53.65
53.77
53.46
74.19
83.76
84.54
84.68
DataInf
7.97
9.85
10.16
9.89
11.17
13.38
13.54
13.45
Gaussian
3.74
4.58
4.62
4.60
3.83
4.72
4.75
4.71
PCA
3.80
4.59
4.67
4.87
3.86
4.77
4.82
4.82
Dropout(Ours)
3.84
4.68
4.75
4.73
3.93
4.83
4.87
4.85
Because I is subset selection matrix, its singular values exactly 1 0, thus T m results in:
1.
This
|AH 12 ( H)).
(20)
This bound is independent the parameter dimension d, explaining the stability of dropout- compres- sion for large-scale models.
A.5
Experiments.
A.5.1 Details of Experiments.
For each methods, set the damping term in influence function as Ar 0.1 En Vo, Va for layer 4, where Q represents the parameters of the I-th layer, Va,4 represents the gradient of the loss function calculated at the i-th data point with respect to 0,, and dy represents the number of paramcters in this layer. For model training, we use hyperparameters Table
A.5.2 More Results.
In this section, we include more results of experiments. Table 10 contains the average time usage for computing iHVP in mislabeled data detection tasks. Table II presents the average time usage for gradients compression in retraining ResNet-9 and GPT-2. ResNet-9 has fewer than million parameters, whereas GPT- 2-small excceds 100 million parameters. As a result, applying PCA or Gaussian becomes computationally expensive for GPT-2. To address this, we use the more efficient FJLT method for comparison. The results demonstrate that Dropout is highly efficient during the compression process while preserving the effectiveness of the influence function.
A.5.3 Examples of Identified Influential Data.
To qualitatively assess whether our method identifies meaningful influential examples, we present represen- fative cases where the most. influential training samples retrieved by our method align with the content
19

---

Published in Transactions on Machine Learning Research (02/2026)
Table 11: Average time usage (seconds) of compression gradients in retraining ResNet-9 and GPT-2.
Model
PCA
Gaussian
FJLT
Dropout
ResNet-9
1344.34
235.64
8.44
GPT-2
453.87
9.76
or source of a given validation example. We list some examples in the following. These examples were selected from experiments using the Pythia-6.95 6.9 model fine-tuned on heterogeneous dataset composed of six sources.
Example A.1
Q Test Example
Absolute quiet descended on the square. Fourteen mech wolves lay out of commission and more than thirty freedom fighters. Mech bird corpses were scattered everywhere. On the platform, Ajax curled his upraised hands into fists. "I, Ajax, warrior hero, hereby assume my rightful place as leader of Mech City!" No one dared disagree. "I order all wolves to stand down! All two-legged robots cease fighting immediately!" The freedom fighters lowered their weapons. The birds took to the sky en masse. The wolves glanced sheepishly each other, unwilling to disobey Ajax's command despite their previous orders. Looking out from his place of concealment, Fascista Ultimo viewed their hesitation with alarm. He uttered directive into his transmitter, and the mech wolves retreated to the square's s periphery.
Most Influential Training Example
This is my home you so arrogantly took up residence in." Michael nodded at Liz. "This my mate that the lot of you tried to kill." With another nod to Avery, he finished. "And last but not least, this is my charge that you beat and tried to feed upon. What do you propose we do about that, Carl?" Words gushed from the tattered young vampire's mouth. "We thought you were dead. All of us did. We were told that you died in the fortress!" "Who told you was dead??" "The Council. They said that you, and all the rogues with you, died when the fortress collapsed." Michael laughed at his answer, and Liz laughed with him. "So Monroe is calling us the rogues now, is he? months old, she would bet. She felt no malice from them, just fright and insecurity.
20

---

Published in Transactions on Machine Learning Research (02/2026)
Example A.2
Q Test Example
Carl Froch's hopes of fighting Julio Cesar Chavez Jnr in a 50 million Las Vegas send-off have suffered a potential setback. The Mexican's promoter, Bob Arum, is currently negotiating new deal with his fighter and expects him return in September or October. But he wants Chavez to face the hard-hitting Kazakh Gennady Golovkin, rather than Froch, reported by Boxing Scene. Out cold: Carl Froch stopped George Groves in the eighth round of their rematch at. Wembley 'Gennady Golovkin will continue to be the first choice assuming he beats [Daniel] Geale, and if Chavez says no to Golovkin, then we'll look to Froch,' Arum told Boxing Scene. Froch retained his hpper-middleeweightt world titles last weekend with stunning eighth-round knockout of George Groves front of 80,000 fans Wembley. The Nottingham Cobra. has admitted he could hang up his gloves but dreams of finishing his career in the fight capital of the world. But he might have to wait until next year if Chavez agrees to fight Golovkin, fight that had been mooted for this summer before negotiations stalled. Option: Froch has targetted Las Vegas send-off against Julio Cesar Chavez Jnr. Golovkin instead defends his WBA Super and IBO middleweight titles against former world champion Geale in New York's Madison Square Garden on July 26. And Golovkin's 's trainer Abel Sanchez is confident his charge would beat Froch if the two were to meet. 'Froch is gallant warrior, but makes too mistakes and if the fight. can be made, see Golovkin capitalising on them stop him in the last part of the fight,' he told World Boxing News. Another option for Froch is another domestic dust-up with mandatory challenger James DeGale. The IBF has ruled that Froch must face the Londoner within the nine months or be stripped of the belt. Do- mostic: James DeGale is Froch's mandatory challenger after beating Brandon Gonzalez last weekend.
Most Influential Training Example
(CNN) Some of the president's men are still working. In golf's Presidents Cup, that is. And while U.S. President Barack Obama wasn't 't in attendance in Ohio he has more important things to worry about former president, George W. Bush, greeted both teams Thursday at the Muirfield Village Golf Club. The biemnial competition, which pits the U.S. against the rest of the world minus Europe, has been kinder to the Americans than the more prestigious Ryder Cup. Indeed since the lournament started in 1994, only once has the International Team prevailed, and the U.S. has won four in row. The U.S. featured six players in the top 10 in the rankings, including world No. and PGA Player of the Year Tiger Woods. The International Team, by contrast, had one Masters champion Adamn Scott. Early indications suggested the U.S.s superiority in the rankings would translate to an easy victory they led all six fourballs in the early stages. But after about a 90-minute delay because of thunderstorms, the International Team fought back. By day's 's end, it was 3.5 to 2.5 for the U.S., slender advantage. Jason Day's 's dramatic pult al the 18th gave the Australian and Graham DeLaet win over Hunter Mahan and Brandt Snedeker after Woods and Matt Kuchar routed Angel Cabrera and Marc Leishman 5 and 4. "It's awfully fun partncring the No. player in the world," " Kuchar said in a televised interview. The U.S. fell behind as Phil Mickelson and Keegan Bradley lost to Louis Oosthuizen and Charl Schwartzel and before Scott and Hideki Matsuyama. halved with Bill Haas and Webb Simpson. Jason Dufner and Zach Johnson eased past Branden Grace and Richard Sterne and to level the match. In the decider, Steve Stricker and Jordan Spieth edged Ernic Els and Brendon de Jonge. Stricker's 's short putt at the 18th gave the U.S. the lead heading into Friday's foursomes.
21

---

Published in Transactions on Machine Learning Research (02/2026)
Example A.3
Q Test Example
1. Let n be the total number of students, which given as 33. 2. Let A be the initial average mark of the class. The total sum of marks for all students
A. The total sum marks for the 3 students with an average mark of 40 is 3 x 40 120. After excluding these students, the remaining number of students is n - 33 - 3 30. 6. The new average mark for the remaining 30 students is given 95. 7. The total sum of marks for the remaining 30 students is 30 95. 8. The total sum of marks for all 33 students is the sum of the marks of the 3 excluded students plus the sum of the marks of the remaining 30 students:
33 x A = 120 + 30 x 95
9. Calculate the sum:
30 x 95 = 2850
10. Substitute back into the equation:
33 x A = 120 + 2850 = 2970
11. Solve for A:
2970 90
A
33
12. Therefore, the initial average mark of the class was 90.
Most Influential Training Example
1. Let V be the total number of valid votes. 2. Let T be the total number votes cast in the election. 3. Given that one candidate got 60% of the total valid votes, this means the other candidate got 40% of the total valid votes. 4. We know that the other candidate got 2519. .999999999999 valid votes, which is 40% of the total valid votes. Therefore, can set up the equation:
0.40 V 2519.99999000099907
6. Solving for V:
2519. 2.....99999990000 999999999999999 6299.9999999989999
0.40
Since V represents the total valid votes, we can approximate V to 6300 (as the fraction extremely close to integer). 8. We also know that 30% of the total votes were invalid, 70% of the total votes were valid. 9. Therefore, the valid votes V are 70% of the total votes T:
0.70 T 6300
10. Solving for T:
6300
9000
0.70
11. Thus, the total number of votes in the election is 9000.
22

---

Published in Transactions on Machine Learning Research (02/2026)
Example A.4
Q Test Example
1. **Identify the total number of marbles**; There are 4 red marbles, 3 blue marbles, and 6 green marbles. Total number of marbles (red) + (blue) + (green) 13 marbles. 2. **Identify the number of favorable outcomes**; We are interested in drawing marble that is either red or bluc. Number of red marbles = 4. Number of blue marbles = 3. Total number of red blue marbles 4 (red) + 3 (blue) 7 marbles. 3. **Calculate the probability
The probability of drawing marble that is either red or blue is the mumber of favorable outcomes divided by the total number of marbles. Probability = Number of red or blue marbles / Total number of marbles = 7 13. 4. *xxpress the probability as a fraction*** ** The fraction is already in its sim- plest form, which 7 Therefore, the probability of drawing marble which either red or blue is f3-
13-
Most Influential Training Example
1. **Tenttify the total number of students**; Total students = 300 2. **Caluulate the number of students who got first division**
Percentage of first division students 29% Number of first division students = 29% x 300 = TO0 x 300 = 87 students 3. **Calculate the number of students who second division students = 54% Number second division 54 300 162 students 4. **Calculate the number of students who just passed** The remaining students are those who just passed. Total students who got first and second division 87 (first division) 162 (second division) 249 students Number of students who just passed = Total students Students with first and second division Number of students who just passed = 300 249 = 51 students 5. ** *Conclusion The number of students who just passed is 51.
29
got second division **: Percentage
students 54% x 300 I00
Example A.5
Q Test Example
The best treatment advice for a 69-year-old male with coronary artery disease who has asymptomatic gallbladder stones, discovered during a routine ultrasound and with no history of biliary colic or jaundice, is: C. No surgery for gallbladder stones In cases where gallbladder stones are asymptomatic, the standard approach is usually conservative management, monitoring the condition without immediate surgery. This strategy helps avoid unnecessary surgical risks, particularly for patients with additional health concerns such coronary artery discasc. Surgery is generally considered only if the stones become symplomatie.
Most Influential Training Example
Given the clinical presentation and history, the most appropriate immediate treatment for this patient is intravenous hydrocortisone. Her symptons-seve severe abdominal pain, confusion, vomiting, high temperature, rapid pulse, and dangerously low blood pressure unresponsive fluids suggest adrenal crisis, especially in the context of an autoimmune condition like Hashimoto thyroiditis, which is associated with an increased risk of other auloimmune disorders such as Addison's 's disease (adrenal insufficiency). During times of stress, such as after surgery, an adrenal crisis can occur if there is inadequate cortisol production. Administering IV hydrocortisone would address the cortisol deficiency and help stabilize her condition.
23

---

Published in Transactions on Machine Learning Research (02/2026)
Example A.6
QTest Example
table: 1-1137718-2 columns: Rd., Grand Prix, Date, Location, Pole Position, Fastest Lap, Winning Driver, Constructor, Report Q: Who was the winning driver on 13 August? A: SELECT Winning
Driver FROM 1-1137718-2 WHERE Date '13 August"
Most Influential Training Example
table: 1-1139087-2 columns: Rd., Grand Prix, Date, Location, Pole Position, Fastest Lap, Winning Driver, Constructor, Report Q: What day was the grand prix in jerez? A: SELECT Date FROM 1-1139087-2 WHERE Location 'Jerez'
Example A.7
TTest Example
Given the clinical scenario of young woman experiencing significant cardiac symptoms such syncope, bradycardia, and palpitations, especially with exertion, in the context of confirmed Lyme discase, the most appropriate treatment is intravenous antibiotics. The bradycardia and syncope suggest that Lyme carditis may have caused high-degree atrioventricular (AV) block, which requires prompt and aggressive treatment to prevent complications. In these cases, intravenous ceftriaxone is typically recommended as it is highly effective addressing more severe manifestations of Lyme discase, particularly those involving the heart. This treatment approach ensures that the infection rapidly controlled, reducing the risk of further cardiac issues. Therefore, for this patient with confirmed Lyme discase and significant cardiac involvement, the most appropriate treatment is IV celtriaxone.
Most Influential Training Example
Given the clinical presentation and history, the most appropriate immediate treatment for this patient is intravenous hydrocortisone. Her symptons-seven severe abdominal pain, confusion, vomiting, high temperature, rapid pulse, and dangerously low blood pressure unresponsive fluids suggest adrenal crisis, especially in the context of an autoimmune condition like Hashimoto thyroiditis, which is associated with an increased risk of other auloimmune disorders such as Addison's disease (adrenal insufficiency). During times of stress, such as after surgery, an adrenal crisis can occur if there is inadequate cortisol production. Administering IV hydrocortisone would address the cortisol deficiency and help stabilize her condition.
24

---

Published in Transactions on Machine Learning Research (02/2026)
Example A.8
Q Test Example
By Daily Mail Reporter PUBLISHED: 00:57 EST, 6 November 2012 UPDATED: 10:57 EST, November 2012 Four years on and Katie Holmes was back on the Broadway stage on Monday night. Reaction to her turn in Dead Accounts is yet to get going, but the actress certainly ensured she got back to work with bang by sharing steamy smooch with co-star Josh Hamilton. In one scene, the 33-year-old actress is seen locking lips with the fellow Thespian and throwing her arms around him. First night: Katie Holmes took to the stage on Monday evening for the preview opening night of her Broadway play Dead Accounts in New York Passionate: The actress shared on steamy clinch with co-star Josh Hamilton in one scene Predictably, perhaps, gossipy reports have surfaced saying Hamilton who is actually married producer Lily Thorne has a 'crush' on Katie, according the Daily News at least. However, such talk has been laughed off by other publications, with the Chicago Sun-Times quoting source brandishing the rumours as 'just more scandal press garbage'. Katie looks like girl next door in the stills released from her performance; she is wearing a casual ensemble of jeans, purple top and floral cardigan, tcamed with pair.
Most Influential Training Example
By Jessica Jerreat PUBLISHED: 21:14 EST, August 2013 UPDATED: 22:39
EST, 10 August 2013 After concluding one last bit of official business, the Obamas departed The Disabled American Veteran National Convention in Florida for their their annual eight-day vacation Martha's Vineyard, where they staying at 7.6 million vacation home with views of the Atlantic. Getting on the planc from Orlando the Obama's sported smart look with the President domning a suit and Michelle Obama meticulously attired with pearls and belt. around her sun dress. However, wheh the first couple disembarked in Martha's Vineyard they were ready for their vacation to start, as Obama had changed into a pair of chinos and Michelle ditched the pearls and belt. Before: President Barack Obama and first lady Michelle Obama wave goodbye as they leave Orlando for family vacation at Martha's Vineyard After: The President and first lady arrive in Martha's Vineyard in more casual attire, wearing a pair of chinos and no jewelry, respectively.
25


Reviews:
[{"review_id": "sHwRf4hG1B", "reviewer": "Reviewer_9ggR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper the author present a method for reducing the cost of computing the Influence function (IF). The influence function can be potentially used in either dataset design or in network understanding by showing the impact of individual data samples on the converge of learning and the network performance. The authors introduce the dropout approach, that is instead of computing the IF in order to reduce the number of parameters (coefficients) that needs to be evaluated. The paper is well written and it seems that it is theoretically correct. \n\nThe authors explicitly show that using dropout the error rate does not changes drastically during the computation of the IF it also allows to simplify the gradients compression by reducing the complexity of the projections of the gradients onto a sparser low-dimensional subspace (eq 2 vs eq. 4) \n\n\nHowever my main concern is about the fact that while the dropout effectively set some parameters to 0 and reduces the computation, the matrices used for computation remains of the same size. Thus unless one uses specific operators for sparse matrices the gain can be minimal at best. In addition, the paper aims at the efficient computation but the authors did not evaluate the time of the computation which should reflect the efficiency of the method", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors main claim is that using the dropout reduces the cost of the computation of the IF. By the definition of the IF computation, this is a natural consequence, because the authors statistically remove a selected number of parameters used in gradient computation and thus affecting the overall cost of the IF evaluation.", "audience_yn": "Yes", "audience_explanation": "The IF function is an important tool in the analysis of the ANNs in general. With the advent of the explainability and the need for understanding in the decision process of the Deep Models this work has a merit to be published.", "requested_changes": "Time evaluation of the proposed method with the analysis of the impact of the dropout on the speed of IF computation. This should be contrasted with the IF without the drop out.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "N5pIg7G2Kv", "reviewer": "Reviewer_79da", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies computationally efficient ways to compute the influence function, a concept taken from robust statistics and got modernised in recent years. Compared to previous methods which uses PCA or random Gaussian matrix, the authors proposed to use dropout as their compression mechanism to speed up the computation of the Hessian, which is the main computational bottleneck when computing the influences. Some preliminary analysis in error was provided, and thorough experiments have shown that the utility of their recovered influence scores is often on par, if not better than, other methods, while enjoying further computational efficiency.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The various experiments (e.g. mislabeled data detection, model retraining) have demonstrated the utility of their method, and their reported clock time in computation has demonstrated the efficiency of their method compared to the other methods.", "audience_yn": "Yes", "audience_explanation": "Yes, I think research on the influence function is an important direction as it help people understand how data affects the model, and computational methods making such method easier and faster to run in large scale is valuable.", "requested_changes": "It is quite interesting to see that in Theorem 3.2, there is no mention of the number of entries being \"dropped out\" in the bound. I find that odd and would request an explanation in this regard. Having this will strengthen the work in my view.", "broader_impact_concerns": "There is no ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nutx7xa5Dv", "reviewer": "Reviewer_ncDR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Modern machine learning models are heavily dependent on the quality of training data. In the statistical literature, influence function measures the influence of a single data point to the optimal parameter. However, computing the influence function requires computing the inverse hessian vector product of the parameters with respect to the loss function, which is prohibitively expensive for modern machine learning models. To address the efficiency issue, existing works propose approximation methods and compression methods to reduce the computation. This work fall into the second category. It reduces the computation by randomly dropping parameters to evaluate, and works with a much smaller hessian matrix, similar in spirit as Dropout. Theories suggest that the error of their compression method is controlled.\n\nThe experiments have three parts. First, on benchmarks of mislabeled data detection, it is found that compression methods generally work better than approximation methods, and the proposed Dropout scheme works similar to random projection with similar hyperparameter settings. Then, the behavior of the models after removing the most influential data are studied under different approximated/compressed influence functions. Dropout performs comparable to existing methods on both small- and large-scale settings. Finally, as the theoretical results are not strong enough, experiments that establish the theoretical properties are presented. The proposed Dropout scheme has a bound that is better than random projection on average, but may have higher variance with a high compression rate.\n\n### Strengths\n\n- The motivation of the idea is strong. Machine learning models are over-parameterized, and dropout is a good tool to utilize the power of over-parameterized neural networks while being more efficient in complicated analysis such as hessian matrices. I like that the idea is motivated by interesting properties of deep neural networks but has a different consideration.\n- The proposed scheme is motivated by influence function, but can generalize to other methods. In particular, the hessian of the loss function has been used in second-order optimization, Bayesian deep learning, and many other ideas. It would be interesting to see how the proposed method could influence other areas, and what are the fundamental principles of second-order methods.\n\n### Weaknesses\n\n- I think the theories are quite weak. The two theorems are proved without consideration of the important hyperparameter $r$. Without the term, the comparison of the two methods could not be established and the rate of efficiency is not established.\n- The exact procedure of the proposed method is unclear to me. For practitioners to use this work, there should be an algorithmic block that takes inputs and hyperparemeters, and gives the outputs. Moreover, the connection of Dropout and random projection could be inferred from the texts, but is not seriously descussed.", "claims_and_evidence_yn": "No", "claims_explanation": "- I am frustrated by the the two theoretical results. Theorem 3.1 for random projection is extremely loose. Random projection wouldn't work at all if it follows this bound. Also, intuitively as $\\frac{r}{d}\\to 1$, the error should go to $0$, which is not established in the theories. \n\n- Section 5 is used to compare the two bounds empirically. However, it only tested $r=1,2,3.4$, but the models were tested with $r=16$ in other experiments. \n\n- Section 4.3 is very confusing. I don't understand why \"the influence signal should align with the semantic or source-level structure of the\ndataset\", and what top-1 means in the context. More explanation should be provided in this part.\n\nOther than the above points, I think the claims are supported by good evidence, and I am convinced that this is a valuable work.", "audience_yn": "Yes", "audience_explanation": "People working on anomaly detection would be interested in novel methods of detecting outliers with the method. The dropout idea could also be potentially added to the toolboxes of modern statistical workflows, once the theoretical properties are more seriously established. \n\nAlso, in applied AI, data cleaning is an important step but the methods are rather unsystematic. The line of works gives an insightful guidance of how to automate data cleaning in this era, when companies are working with trillions of datapoints daily.\n\nIn Bayesian deep learning, the hessian of the model log likelihood is used for obtaining the posterior distribution of the parameters. I believe the line of work could give insights to other methods that use the hessian of the loss functions in modern machine learning models.", "requested_changes": "- If it is possible, I would like to have the theorems that include the term $r$ in the bound. \n- An algorithmic block of the Dropout pipeline for computing the influence functions would be helpful for practitioners to use the methods. There is a claim \"and reduce memory and computational costs to O(r) as only r entries of gradients are sampled and stored\", which is not clear from the presentation.\n- I'd like to see more experiments in Section 5. For example, $r=1,2,4,8,16$ would be a better selection of hyperparameters to compare.\n- Please explain Section 4.3 better. Also, why is Dropout quite different from random projection in this experiment? I don't have the intuition here.\n\nMinors:\n- At the end of page 2, \"mode\" -> \"model\".\n- The notations of the methods are not consistent. Both Dropout and random projection use the same $\\tilde{\\mathcal{I}}$, which complicates the description in the texts. Would it be possible to add subscripts to the notations, or change one of them to $\\hat{\\mathcal{I}}$?\n- There is a minor error in the proofs of the theorems. When Woodbury matrix identity is used, the sign of the longer term is minus, which should give a plus term after plugging in. This doesn't affect the results, but should be corrected.", "broader_impact_concerns": "The broader impact concerns are minor.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)