HyperINF: Unleashing the HyperPower of Schulz's Method for Data Influence Estimation
Authors: Xinyu Zhou
Simin Fan
Martin Jaggi
Venue: COLM 2025 (2025)
Keywords: language model; data; influence score
TL;DR: We propose HyperINF, an efficient and accurate influence function approximation which leverages the hyperpower method, specifically Schulz's iterative algorithm.

Abstract: Influence functions provide a principled approach to assess individual training samples' contributions to specific targets. However, their high computational costs have limited applications in large-scale models and datasets. While existing approximation methods have reduced computational overhead, they often suffer from inaccurate estimation due to weak convergence guarantees. Hyperpower methods offer rigorous convergence guarantees for matrix inverse approximation, but their matrix multiplication operations typically involve intractable memory and computation costs for large-scale models.
We propose HyperINF, an efficient and accurate influence function approximation leveraging the hyperpower HyperINF, specifically Schulz's iterative algorithm. To address computation-intensive matrix multiplication, we incorporate generalized Fisher information (GFIM) as a low-rank Hessian matrix approximation, reducing memory and computation overhead to constant costs.
Through comprehensive convergence simulations on matrix inversion, we demonstrate HyperINF's superior accuracy and stability compared to baselines. We further validate its efficacy through extensive real-world data attribution tasks, including mislabeled data detection and data selection for LLM and VLM fine-tuning. On LoRA-tuned models, HyperINF achieves superior downstream performance with minimal memory and computational overhead, while other approaches suffer significant degradation. Our code is available at https://github.com/Blackzxy/HyperINF .

Published as a conference paper at COLM 2025
HyperINF: Unleashing the HyperPower of Schulz's Method for Data Influence Estimation
Xinyu Zhou", Simin Fan', Martin Jaggi
Machine Learning and Optimization Lab, EPFL.
(first_name. last_name past_name]@epfl.. (@epfl.ch ch
Abstract
Influence functions provide principled approach to assess individual training samples' contributions to specific targets. However, their high computational costs have limited applications in large-scale models and datasets. While existing approximation methods have reduced compu- fational overhead, they often suffer from inaccurate estimation due to weak convergence guarantees. Hyperpower methods' offer rigorous con- vergence guarantees for matrix inverse approximation, but their matrix multiplication operations typically involve intractable memory and com- putation costs for large-scale models. We propose HYPERINF, an efficient and accurate influence function approximation which leverages the hy- perpower method, specifically Schulz's iterative algorithm. To address computation-intensit matrix multiplication, we incorporate generalized Fisher information (GFIM) as low-rank Hessian matrix approximation, reducing memory and computation overhead to constant costs. Through comprehensive convergence simulations on matrix inversion, we demon- strate HYPERINF'ss superior accuracy and stability compared to baselines. We further validate its efficacy through extensive real-world data attribu- tion tasks, including mislabeled data detection and data selection for LLM and VLM fine-turin tine-tuning. On LoRA-tuned models, HYPERINF achieves supe- rior downstream performance with minimal memory and computational overhead, while other approaches suffer significant degradation. Our code is available at https: thub. com/ Bllaakkk///yyprrIII.
1 Introduction
Large foundation models have demonstrated remarkable capabilities across language, vision, and audio modalities (Touvron et al., 2023; Liu et al., 2023a; OpenAl et al., 2024; Bai et al., 2023). Recent data-centric studies emphasize that training data plays an essential role in downstream model performance (Hoffmann et al., 2022; Gao et al., 2020; Penedo et al., 2023; Wang et al., 2018; Gunasekar et al., 2023; Lee et al., 2023; Longpre et al., 2023b). Consequently, there is growing need for efficient and effective data attribution methods that can identify the most beneficial training samples without introducing substantial computational overhead on large-scale models and data pools. Influence functions, one of the most principled data attribution methods, quantify each training sample's impact on model predictions for a validation set (Hampel, 1974; Koh Liang, 2020). Despite the efficacy (Kwon et al., 2024; Koh & Liang, 2020; Pruthi et al., 2020; Guo et al., 2021; Wang et al., 2019b; Kong et al., 2021),the the natural Hessian inverse operation in the influence function formulation introduces intractable memory and computational costs, hindering widespread application to large models.
To mitigate these computational challenges, various estimation methods have been pro- posed. Agarwal et al. (2017) introduced LISSA, which iteratively estimates Hessian-vector products. However, the algorithm's 's convergence is not guaranteed, potentially resulting
'These authors contributed equally this work.
A hyperpower method is defined as function (A,,)) on matrices A and X, where A is the targeted matrix inverse (Petković, 1995).
1

---

Published as a conference paper at COLM 2025
Table 1: Complexity Comparison between Exact (Gaussian Elimination), LiSSA, Datalní and HyperINE wttpuaatoooaaaaiiiii Computational and memory complexities are obtained on a LoRA-tuned model with dimension d IN and rank r E IN. Assume the dimension the LoRA matrices is identical across L different layers.
Complexity
Exact (Gaussian Flimination)
LISSA
Datalnf
Hypor|NF w. GFIM
Hyper|NF
Computation
A(ALL+
O(rdi)
O(PL)
O(PPL)
H H- g Compulation
O(APL) 0(9P))
OLddL+ O(rd)
O O(f))
P(PdL++ O(Al))
Memory
O(AAF)
in significant deviation from correct values after several iterations. More recently, Kwon et al. (2024) proposed DATAINF as closed-form Hessian matrix approximation, further reducing complexity. Nevertheless, its error bound grows quadratically with matrix scale, making it susceptible to downstream performance degradation. Hyperpower methods offer a promising alternative for improving Hessian-inverse estimation accuracy with rigorous convergence guarantees (Garnett et al., 1971; Behera et al., 2024). However, these meth- ods involve iterative matrix multiplication operations, introducing substantial memory and computational costs, especially for large-scale networks. To address this challenge, we introduce HYPERINF, novel approximation method incorporating Schulz's 's iterative algorithm (Petković, 1995). To manage the large matrix multiplication costs, we employ the generalized Fisher information matrix (GFIM) (Hu & Li, 2024) as a low-rank Hessian approximation, with theoretical justification. Notably, on LoRA-tuned models, memory and computational costs are reduced to constant value independent of LoRA ranks. Our experimental results demonstrate that HYPERINE with GFIM achieves superior accuracy due to its rigorous convergence guarantees while maintaining low computational over- head compared to baseline methods. Through extensive experiments on LLMs and VLMs, we show that HYPERINF effectively identifies both helpful and mislabeled data points, enhancing data attribution interpretability and fine-tuning efficiency.
conplexxtt
Our Contributions. We summarize our main contributions as follows:
We leverage the generalized Fisher information matrix (GFIM) to derive novel low-rank formulation of influence functions Equation 5, significantly improving computational efficiency for large-scale models;
We demonstrate that Schulz's method substantially (Equation 6) enhances the stability and accuracy of Hessian inversion approximation, yielding more accurate influence scores for large-scale data attribution;
We verify HYPERINN's empirical effectiveness across extensive large-scale experi- ments, including mislabeled data detection (6 4), data selection for LLM fine-tuning ($ and VLM instruction-tuning (S 5.3).
Preliminaries
We first revisit the influence function formulation and two variants, LISSA and DATAINE.
Setup. The data attribution problem aims to assess each data point in the training set Dirain = {xx, }= according to their impact on the model's performance on a targeted validation set Dval {xx]ra ) Given a model f parameterized by 0, the loss function on the t sample (xx, is denoted as fey We assume that the loss function is differentiable and strongly convex, the gradient on the 7th sample can be represented as Vol; == Vo (xx)) with respect to 0. The empirical risk minimizer on the entire training set is denoted as 8* = arg mingce 1 E"1 fe(:)))
Influence Functions. The influence function quantifies how fast the model parame- ters would change corresponding to the wp-weight of specific data point. Follow- ing Koh & Liang (2020), given an infinitesimally small e > 0, we upweigh the con- tribution of the tt datapoint (xx)ys) by increasing its portion in the loss function: (()) == arg minoce Ei= E foxx)) + el fo(xx)) Assume the loss function
2

---

Published as a conference paper at COLM 2025
HypertNF N=
HyperikFF No800
HyperINF N=6400
HypertNF N=12800
din-512
dmm-1224
500
dmm=2245
1099
Ernor
dinw4295
180
23
25
Iterakion
Feration
Iteration
Iteration
LISSA N= 200
USSA N=800
USSA N=6400
LSSA N=12800
Log E0101
dm-1124
dmm22442
17-
30
dine4295
121
18
15
26
Iteration
Beration Datalinf N=800
Dotaint N-200
Datainf Ni 6400
Dataint N=12800
1000
1.21
Error
2007
600
600)
14h
150
300
2503
156
Dimension
Dimension
Dimension
Dimension
Figure Convergence test of HYPERINE, LISSA and DATAINE. We construct M + EN ttis Al and apply various methods approximate the inverse hessian-vector product M where sj Rd. E Rd are randomly generated from standard normal distribution. Only HYPERINF can converge to from
low error with increasing matrix dimension and sample size while the approximation LISSA and DATAINF significantly diverge from the target values. For LISS A, it does converge but only in limited circumstances (e.g. when N is large). We include the results with other distributions in Appendix I.
(x)) is twice-differentiable and strongly convex in 0, the influence of the t data sample (x)) E Dirain on 6* is defined as the derivative of ( (c) at =
d)(k)
Lo (xkr de
H ( Vol
where H()) = e( is the Hessian matrix of the empirical loss com- puted on the flattened gradient vectors (Koh & Liang, 2020; Kwon et al., 2024).
We further score the contribution from each training sample according to model's performance on the validation set Dval. For simplicity, we define I (xx,ye) == - ((0))(0) (E(00)-1Vol as the influence from the th datapoint (xxxyz) E Dtrain on Dval, where = -. Vee (y)a (f(("""))) representing the gradient on the validation set, the datapoints assigned with largest negative values? of influence function would lead to the sharpest drop of validation losses, which contribute the most to the training process. In contrast, the datapoints with largest positive values could be the toxic samples which sabotage the model training.
Agarwal al. (2017) proposed an iterative method to compute the inverse Hessian vector product H(0+)-lv. For Vo v, LISSA recursively computes the following iteration:
LISSA.
Vj a (I H(O)))v; Agarwal et al. (2017) proved that vi converges to H(@*) (10 as increases, when H(0*) < In practice, it is often assumed that LISSA converges to H(0*)-1 after several reasonable numbers of iterations, and applies the approxima- tion vi ~ H(6-)-1o to compute the influence function I (xx, Vı.) = -v, V However, some works have shown that the stability and convergence from the iterative update are questionable (Basu et al., 2021; Ko et al., 2024).
We refer largest negative values here as negative scores with the largest absolute value.
3

---

Published as a conference paper at COLM 2025
DATAINE. Kwon et al. (2024) proposed closed-form approximation of the Hessian inverse, which greatly improves the computation efficiency. Firstly, following George et al. (2021), when applying the negative log-likelhhood loss function fey,fe(x) - log (o))) the second-order Hessian is equivalent to the Fisher Information Ma- trix (FIM) in expectation (Bartlett, 1953), which only involves first-order computations. Consequently, Kwon et al. (2024) approximate the Hessian inverse leveraging the Sherman- Morrison formula 3:
(
1
H 11
1
Vol.Vol A+ Vol, Vol.
(2)
where G( VoL, V
stands for the Fisher Information Matrix (FIM). With N representing the number of tunable parameters, the complexity of DATAINF is reduced to (N) operations with O (NV memory expense. However, compromise, the reverse-order operation Equation 21 incurs a O(N2) approximation error (Kwon al., 2024). When applying to large-scale models, it could risk a large approximation error.
3 HYPERINF: Efficient and Accurate Data Influence Approximation via the Hyperpower Method
We introduce HYPERINF as an accurate yet efficient approximation method for influence function, which leverages generalized Fisher Information Matrix (GFIM) proposed by Yang et al. (2022) and Hu & Li (2024), and Schulz's hyperpower method (Petković, 1995). We begin by providing theoretical proof of Hessian matrix approximation for large models using GFIM, followed by demonstration of Schulz's iteration for approximation of the matrix inverse.
3.1 Large-scale Hessian Approximation using Generalized Fisher Information
The second-order gradients often incur intensive computations and instability on large-scale networks. Therefore, we conduct several approximations on Hessian matrix when applying Equation 1 on LoRA-tuned models.
Block-wise Diagonal Approximation. In deep transformer-structured networks, the Hes- sian matrix is observed to be approximately block-wise diagonal according to (Zhang et al., 2024a;b). We, therefore, apply block-wise diagonal approximation on the Hessian inverse in Equation 1. Given a neural network as a compositional function fa(x) = fez ....o fo, (x) where for E [L], we compute the hessian inverse on each parameter block which yields sparse estimation as diag H1 (0)-1, ...., ()) (Grosse et al., 2023b).
Connection between Generalized Fisher Information and Hessian Matrix. Suppose that we train the model to minimize the negative log-lik ligliielihood objective: f(y.f((x) - llog ply fx)) for all (x X' Y, where p(.) is the probability density function and X' are input and output spaces, respectively. According to Bartlett's ss second identity
(Bartlett, 1953), the second momentum of first-order gradient (i.e. Fisher Information Matrix)
is equivalent to the second-order gradient matrix (Hessian) in expectation:
1x Exxx-p(()))XXXXXX
V
(3)
= -Exx=x(X0
V (o()) (Vol(,,))xx)) [6
Since Equation 3 replaces the second-order gradient with stable and tractable first-order gradients, the Fisher Information Matrix (FIM) is widely adopted as an approximation of the Hessian matrix in deep networks (Grosse et al., 2023a; Kwon et al., 2024; Barshan et al., 2020). We further extend the Generalized Fisher Information Matrix (GFIM) (Hu & Li, 2024) to yield low-rank formulation of influence function. With some idealized assumptions, we claim the Lemma 3.1 following the insights from Yang et al. (2022) and Hu & Li (2024).
For simplicity, denote & := (y;, fo(x;)
4

---

Published as a conference paper at COLM 2025
Lemma 3.1. Given the matrix-form gradient on a parameter block @ as g which can be flattened to vector by vec(g) E R1xni Let & denotes the Kronecker product, ly denotes r X r identity matrix. Assume that each column of the sample gradient g = g0;;,yy) E Raxx is independent and identically distribuled random vector with zero mean under the distribution ply x, 0) for any We have:
g((xx,,) Raxr,
(788)) (-88*)
E vec(g) vec(g)
E I
In addition (Equation 3), holds:
1 I ②
E
y 88
[E(((vec(0)).
Following Lemma Theorem 3.1, we further estimate a hessian-gradient product using GFIM, corresponding to the - term in Equation 1. Given an invertible matrix A, we have (4, @ A)-1 = I, @ A-1 Therefore, denote the GFIM matrix as G(0) A (gg ) E Raxd for any matrix E Rd it holds that:
1 (7888) vec(o) vec(G(0) ((vec())))vve((() &
(4)
Consider a LoRA-tuned model with LoRA dimension d and rank r. We assume that each column in one LoRA block AW E Rdxr corresponding i..d. with zero mean. In the ideal case that the model is trained to converge with B(-Velog p(y) 0)) = O, the zero-mean assumption on the columns of gradient matrices could stand. Thus, we apply Equation 4 to approximate the original Hessian-gradient product. To further guarantee that G(0) is invertible, we add a damping factor f4 to the GFIM matrix following Martens (2010). We eliminate the constant in Equation 4 then derive the final formula of HYPERINF influence score. On specific datapoint {* Dirain denote the unflattenedd gradient on a parameter block as g: (0), we compute:
each rank, is distributed
8: T THYPERINE VR) -- (G(0*) + A1a) (g)))))))))
1 Vol(y) val
where 8+ m
E !xx representing the average unflattened gra-
dient 0 the validation set.
3.2 Matrix Inverse Approximation with Schulz's Method
Schulz's method (Petković, 1995). To compute the inverse of one matrix A, the hyper- power iterative family of matrix iteration methods has attracted the attention of many researchers due to its rigorous convergence guarantee (Altman, 1960; Garnett III et al.,
1971; Bazán & Boos, 2018): X, XI(I+ T; + T7 +..++ ) where T; - AX, The iterative approach requires matrix-matrix multiplications per iteration and has an order of convergence (Bazán & Boos, 2018). When choosing p = 2, it yields the Schulz iteration, which can also regarded as by-product of the Newton method applied to the non-linear
equation f(X) = A
X = X; X,Y,, Y=1- AX,
(6)
proved by Ben-Israel Cohen (1966) and Petković (1995) that with a proper initializa- tion, Schulz's method would converge A 1 in the order of convergence. We provide complete proof of convergence in Appendix D. Compared to other conventional matrix inverse algorithms (e.g. gaussian elimination, conjugate gradient, GMRES), Schulz's method demonstrates superior accuracy in terms of error rate and significant efficiency gains from the GPU acceleration on matrix multiplications. We include more details in Appendix H. With comprehensive matrix inversion experiments (section 4), we show that starting from a small identity matrix or random gaussian initialization, Equation could converge to desirable error rate in finite steps (L 20). We provide the pseudo-code in Algorithm 1.

---

Published as a conference paper at COLM 2025
Summary. We hereby provide the pipeline of influence score computation using HYPER- INF. Firstly, we compute the generalized fisher information G(0) on tunable parameter blocks (e.g. LoRA modules); Secondly, we compute the inverse of the damped GFIM (G(0) + Ala) with Schulz's iterations (Equation 6); Last, we compute the influence score with cached validation gradient and the unflattened gradient on each training sample, i.e. ZHVPERINF (x2, ykk) (Equation 5). We provide the pseudo-code in Appendix (Algo. 2).
Complexity Analysis. Compared to the original influence function formulation in Equa- tion 1, the generalized fisher information matrix G( E R reduces the memory com- plexity from (f?dd) to Odd)). On computation complexity of Hessian-gradient product, the matrix multiplication between (G(0*) + Ala)-1 Rdxd and 8k E Rdxr only requires O(rd²) FLOPS, instead of O(r2d²) with flattened gradient vectors. Specifically, with LoRA rank 16, HYPERINF only requires 0.39% memory complexity and 6.25% computations comparing to original Hessian-vector product operations. We include the complexity com- parison to other existing approximation methods in Table 1, where HYPERINF with GFIM showcases outstanding memory and computation efficiencies. In addition, we report the time costs for Hessian inverse-vector product in subsection E.1, where HYPERINF demon- strates superior efficiency on GPU, It underscores the superior compatibility of HYPERINF with modern GPU computations.
4 Synthetic Convergence Test of Matrix Inverse Approximation
Setup. We first examine the accuracy and stability of Schulz's algorithm on matrix inverse approximation by convergence test.
Specifically, to simulate the FIM matrix in the influence function A = (G(0*) + Ala) on training set with scale |Dtrain| = N and model with number of parameters as d, we construct M N EN s Al Rdxd by randomly generating s; E Rd. We then compute the exact value of M E Ixd and the approximated value M using DATAINF and Schulz's algorithm. For LISSA, since it directly approximates the inverted matrix-vector product, we randomly generate another vector v E I2d and compute the exact value of the matrix-vector product Q = M-1v E Rd as the target. We denote the approximated value from LISSA as Q.
For all the methods, we measure the error as the Frobenius norm of the matrix I|Q - Q1 where Q= for DATAINF and HYPERINF.
We run the convergence test across various d (512, 1024, 2048, 4096) and N e {200, 800, 6400, 12800), emulating different scales of model and amount of data samples respectively. In all settings, the dampling factor À is set as 0.01. The initialization for iter- ative methods is set as Xo = 5e 44 We provide more results with matrices from various distributions in Appendix I, which demonstrates the similar pattern as in Figure 1.
HYPERINF solves matrix-inversion approximation with great convergence performance. We present the results from the synthetic experiments in Figure 1, where HYPERINF with Schulz's s algorithm demonstrates a remarkable accuracy and stability compared to the other two methods. Specifically, on high-dimensional matrices M with large d, both LISSA and DATAINF tend to diverge with increasing approximation errors.
For LISSA, the error would not converge but explode exponentially according the number of iterations. Even when applying on a small dimension of matrix with N = 200, LISSA is not able to give an accurate approximation with large error rate ~ 10 This might comes LISSA algorithm to the initialization conditions, which could be
from the sensitivity hard to tune when apply on large-scale models. In comparison, HYPERINE with Schulz's algorithm could always converge to a low error rate within finite iterations across all scales of d and N. It implies that our proposed HYPERINE could consistently achieve satisfying accuracy on large-scale models and datasets, while both LISSA and DATA INF could significantly diverge from the exact value.

---

Published as a conference paper at COLM 2025
Table Mislabeled Data Detection Accuracies across the GLUE Benchmark with rank 16 for rsLoRA finetuning. When probing 20% and 40% data points, HYPERINE can consistently outperform other baselines by a large margin 27% f).
Method (LoRA) (k%)
DATAINE
LISSA
TRACIN
HYPERINE
COLA
20%
39.66-616
32.181936
40.25 +:20
51.55-1.
40%
50.59.
16.0
49.
66.04-1xt 1.84
MRPC
20%
58.52
24.461124
57.75
60.89
40%
68.89
37.88
67.34
79.17=0.2
QNLI
20%
48.92
43.70
45.37
64.77
40%
56.51 +2.00
50.18
49.51.a70
76.66 1.44
QQP
20%
51.11
38.1412.3
52.18
57.85=1k2
40%
62.07
44.74.2.71
=1.59
73.07
RTE
20%
36.74.1.9
35.07
35,14 11:3
41.90=00.0
40%
47.85
47,85 47.85
45,51
57.96=035
SST2
20%
74.96=433
44.93
66.51
69.00
40%
80.50
46. 46.62.334
71.96
78.44=1.17
Average
20%
51.65
36.41
49.53
57.66
40%
61.07
46.01
57.65
71.89
Influence Function Approximation on Large-scale Models
In this section, we further apply HYPFRINF on influence function approximation on large- scale foundation models and demonstrate its effectiveness on various data attribution tasks. We compare HYPERINE with two existing baseline methods LISSA (Agarwal et al., 2017) and DATAINF (Kwon et al., 2024), as well as the Hessian-free method TRACIN, which replaces the second-order term H in Equation with the identity matrix I4 (Pruthi al., 2020). Across all mislabeled data detection, data selection for LLM fintuning and VLM pretraining, HYPERINF shows promising performance compared to all baseline methods.
5.1 Mislabeled Data Detection
We first apply HYPERINF on the mislabeled data detection task following (Koh & Liang, 2020; Yang et al., 2024; Kwon et al., 2024). We construct corrupted dataset by flipping the label of 20% randomly sampled data points, which is considered as the mislabeled subset. After fine-tuning the model the corrupted training dataset, we rank all data points according to their influence scores from HYPERINF, LISSA and DATAINE respectively and then identify the top-p% samples with the highest scores the mislabeled ones. We define as the inspection rate. Denote the real mislabeled subset as Dmis and the identified top-p% percentage subset using influence function as D(p), the detection ratio r((p) can then be measured as the recall between Dmis and D()): rt(p) = [DminD(pp)
Dunee E (0,min(p/20) 1.0)]. We assess the mislabeled data detection accuracy according to the detection ratio rt with respect to the inspection rate p. We run the experiments across six tasks in the GLUE benchmark (Wang al., 2019a) with the Roberta-large model. We finetune the pretrained checkpoint each corrupted training set using rsLoRA (Kalajdzievski, 2023), a rank-stabilized variant of LoRA (Hu et al., 2021). ,. We provide more implementation details, ablations with various LoRA ranks and complexity analysis in Appendix E.
Results. According to Table 2 and Figure 2, HYPERINF outperforms all baselines most of tasks with better accuracy and less variance. On SST2, the accuracy of HYPERINF comparable DATAINF and TRACIN method while the variance is largely reduced when applying HYPERINF. In contrast, we find that LISSA does not perform well on the mislabeled data detection task, which barely better than the random guess. Additionally, with the low-rank formulation from GFIM, HYPERINF achieves remarkable efficiency comparable to all the other baselines with modern GPU computing (subsection E.1).
Comparison between HYPERINF with GFIM and FIM. Notably, HYPERINE with GFIM does not lead to performance degradation compared to FIM. According to Figure 5, HYPER-
7

---

Published as a conference paper at COLM 2025
INF with GFIM consistently achieve comparable or better performance than FIM, while being (1/r)² more efficient in compulation and (1/r)² in memory (Table 1).
5.2 Data Selection for LLM Finetuning
We further show the effectiveness of HYPERINF on data selection for LLM finetuning (Pruthi et al., 2020; Kwon et al., 2024; Xia et al., 2024; Albalak et al., 2024). Given downstream task, we aim to select the most beneficial data points from the train dataset which yields a better accuracy on the held-out test set. Specifically, we fine-tune pretrained Llama2-7B' check- point (Touvron et al., 2023) on four reasoning tasks: QASC (Khot et al., 2020), HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020) and LogiQA (Liu et al., 2020). We consider both sparse (LoRA) and dense finetuning strategies. When applying LoRA, we start with warmup run on the training set for 1 epoch to prevent using gradients from randomly initialized LoRA modules. We apply LoRA with rank r = 64. We compute influence scores from HYPERINF, DATAINF, LISSA and TRACIN and select the top-k% (k 5,20) datapoints with the lowest (i.e. largest negative) scores respectively. We continually train the model after warmup run using the selected data points. For dense finetuning, we use the gradients from the last transformer block to compute influence scores, which is observed to be the most influential layer within the autoregressive ressive language model architecture (Men et al., 2024). Since the EKFAC (Grosse et al., 2023c) method only applicable on linear or convolution layers, it is not trivial to apply it on LoRA finetuning experiments. Thus, we only compare to E.FAC results with dense finetuning. We report the test accuracy of the finetuned model. We include more implementation details in Appendix F. The model is tuned for N (resp. N = 3) epochs on LoRA (resp. dense) finetuning.
Results LoRA finetuning. According to Table 3, HYPERINF achieves the best per- formance comparing to other baselines. Notably, with 5% finetuning datapoints selected by HYPERINE, the reasoning accuracy outperforms the train with the full dataset, which requires 20x data samples and 4x FLOPs. With 20% selected data points, HYPERINF greatly improves the accuracy by 2.0% above the random selection baseline.
Results on dense finetuning. Although the theoretical analysis in Theorem 3.1 is inspired by LoRA finetuning context, show that HYPERINF also significantly benefits data selection in dense finetuning. According to Table 4, with 5%, 20%, 40% selected data points, HYPERINF consistently improves the reasoning accuracy across all tasks above the random baseline. In contrast, all three baselines could lead to degradation when selecting a small portion of data points (5, 20%). Compared to training on the full dataset (1 epoch), using 40% HyPeRINFFseleeted samples improves the average accuracy by 12.9%, which also performs other baselines by large margin.
Discussions. The effectiveness of data selection methods varies across tasks (see Table and Table 4), possibly due to differences in the inherent quality of the training datasets. For instance, on tasks such as HellaSwag and LogiQA, all data selection approaches yield only marginal gains, possibly because the training data is already of high quality. Moreover, sample-level data selection often prioritizes similar high-quality examples, which, if reused across multiple training epochs, may lead to redundancy or overfitting.
Compared to LoRA fine-tuning, last-layer representations may capture more mask-relevant and informative signals (Men et al., 2024), enabling the selection of more "influential" examples that better drive model performance. This observation aligns with, rather than contradicts, the earlier point about data quality. It suggests that even within a high-quality dataset, the effectiveness of selection can depend on the underlying representation used.
5.3 Data Selection for VLM Pretraining
We further consider to apply it on multimodal instruct-tuning data selection for Vision- Language Model (VLM) pretraining (Liu et al., 2023c; Bai et al., 2023; Chen et al., 2023; Karamcheti et al., 2024). Following LL (Liu et al., 2023c), we adopt the commonly used VLM architecture which consists of three components: vision backbone Vp, a projector Fg and a language backbone LMg. In our experiments, we apply CLIP ViT-Large (Radford et al., 2021) with a patch size of 14 and input resolution of 336px as the vision backbone and
http://gggggggac...ccataaaaaaaa. co/meta- |lama/LLaa 1ama-lmmaaaaa
8

---

Published as a conference paper at COLM 2025
Table Evaluation accuracies (%) for LLM data selection with LoRA finetuning. The best results Bolded and the second-best are Underlined. On average, HYPERINE shows the larger improvements as k increases and performs better than all other baselines. The 1 ()) indicates the improvement (degradation) compared to the Random baseline.
Method (LoRA) (R%)
Random
DATAINF
LISSA
TRACIN
HYPERINE
14.0
10.6
12.9
QASC
20%
16.2
18.7
16.7
16.3
19.7
100%
14.1
88.9
88.5
88.5
HellaSwag
20%
88.7
89.8
89.5
89.3
89.7
100%
91.7
5%
51.3
53.7
52.9
52.9
54.1
PIQA
20%
52.6
52.7
55.6
54.8
56.0
100%
50.6
5%
27.0
28.7
25.4
24.8
28.0
LogiQA
20%
26.8
27.0
25.6
27.0
27.0
100%
27.6
5%
45.4
46.0(066)
44.4
44 (4.6)))1))
46.2
Average
20%
46.1
47.1
46.9(o.9))
46.9(o.1)
48.12207)
100%
46.0
Table 4: Evaluation accuracies (%) for LLM data selection with dense finetuning. The best results are Bolded and the second-best are Underlined. The (4) indicates the improvement (degradation) compared to the Random baseline.
Method (dense) (K%)
Random
DATAINF
LISSA
TRACIN
EKFAC
HYPERINF
5%
11.3
12.5
11.2
11.4
10.8
14.3
QASC
20%
11.0
40%
18.1
35.6
13.2
40.1
22.0
56.1
100%
11.9
5%
71.5
70.8
70.6
72.5
27.5
81.3
HellaSwag
20%
84.7
82.8
83.8
82.6
68.3
83.2
40%
100%
92.4
5%
46.5
42.3
48.7
47.8
41.5
53.2
PIQA
20%
53.2
57.3
57.0
40%
55.0
60.8
60.9
57.1
53.8
58.0
5%
25.5
25.0
27.2
25.4
26.5
28.3
LogiQA
20%
28.6
22.3
26.4
27.4
27.4
30.2
40%
30.6
28.2
34.3
33.2
30.5
40.1
100%
27.0
3%
61.1111)
39.4977)
(your))))
39. 19.11
44.3
Average
20%
44.9
45.6
43.7
44.6)033
44.8 44.8p.11)
46.4..51
40%
47.4
53.16771
49 49.4g200
54.8774
49.1(.21)
60.3(12.9)
Llama2-7B (Touvron al., 2023) the language backbone. For the projector Fe, initialize a two-layer GELU-MLP (Hendrycks Gimpel, 2023). Along the suggested setting from Karamcheti al. (2024), freeze the vision backbone Vp throughout the entire training process while only tuning the projector Fy and the language backbone LMg. We provide more implementation details in Appendix G.1.
Setup. We adopt the two-phase pretraining scheme following LLaVa (Liu et al., 2023c). In the alignment phase, we tune the projector Fp and LoRA modules of the language backbone on separate alignment dataset (Karamcheti et al., 2024). In the instruction-tuning phase, we select the most influential data samples from a generic dataset consisting of 665K datapoints (Karamcheti et al., 2024). We compute the influence score using the gradients from the projector and LoRA modules and then select subsets with the lowest scores (i.e. largest negative). We train the VLM on the selected instruct-tuning subsets for one epoch and four cross-modal reasoning tasks: VQAv2 (Goyal et al.,
evaluate the model's performance 2017), GQA (Hudson & Manning, 2019), POPE (Li et al., 2023) and Text-VQA (Singh et al., 2019). We provide more details in Appendix G.2 and G.3.

---

Published as a conference paper at COLM 2025
Results. We present the downstream accuracies across four reasoning tasks in Table 5. In terms of average accuracy, HYPERINF consistently outperforms all the baselines, achieving 2.3% improvement over the random baseline with 20% of data selected. In contrast, with only 5% selected data points, LISSA shows significant (8%) performance degradation. We also conduct the experiments without training the projector and study the significance of the alignment phase. Our results indicate that the alignment phase is essential for effective instruction-tuning data selection.
Table Downstream evaluation accuracies (%) on VLM instruct-tuning data selection experiments. The best results are Bolded and the second-best are Underlined. We applied the gradients from Projector+LoRA modules to compute influence scores.
Method (Projeclo=+1oRAA) (k%)
Random
DATAINF
LISSA
TRACIN
HYPERINE
VQAv2
5%
60.2
60.7
53.2
59.2
60.3
20%
64.5
64.7
65.1
66.4
67.3
GQA
5%
42.2
42.5
20%
45.5
45.1
49.8
POPE
5%
72.2
76,9
57.9
78.9
80.6
20%
83.4
84.0
82.6
84.2
84.5
TextVQA
5%
32.0
32.0
27.4
26.2
26.4
20%
35.8
35.9
34.3
31.7
36.1
Average
5%
51.6
53.0
43.6
51.9(03)) 58.0gu.71)
53.2(1
20%
57.3
57.4
57.0go
59.6(231)
6 Conclusion
In this work, we propose HYPERINE influence function-based data attribution algo- rithm that achieves efficient and accurate Hessian approximation by leveraging generalized Fisher information and Schulz's algorithm. Through comprehensive matrix inversion tests, we demonstrate Schulz's 's algorithm's 's superior accuracy and stability compared to alternative approaches. Our method also shows remarkable efficacy in large-scale data attribution appli- cations, including mislabel detection, data selection for LLM and VLM training. HYPERINF consistently outperforms existing baselines, validating the benefits of our precise Hessian estimation approach. Furthermore, we show that HYPERINF effectively utilizes modern GPU clusters through matrix multiplication operations, delivering significant computational acceleration compared to prior baseline methods. These results establish HYPERINF as powerful tool for influence-based data analysis in contemporary machine learning pipelines, offering both theoretical rigor and practical advantages.
10

---

Published as a conference paper at COLM 2025
References
Naman Agarwal, Brian Bullins, and Elad Hazan. Second-order stochastic optimization for machine learning in linear time, 2017.
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. A survey on data selection for language models, 2024.
M Altman. An optimum cubically convergent iterative method of inverting a linear bounded operator in hilbert space. Pacific Journal of Mathematics Vol. 10, No. 1960.
Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023.
Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second PASCAL recognising textual entailment challenge, 2006.
Elnaz Barshan, Marc-Etienne Brunet, and Gintare Karolina Dziugaite. Relatif: Identifying explanatory training examples via relative influence, 2020. URL https:////arx org //bss/ 2003. 11630.
M. Bartlett. Approximate confidence intervals. Biometrika, 40(1/2):12-1,, 1953. ISSN 00063444. URL htttp://wwe hwwpowww..sr.ooooo.oo..... wtt./stablle//3330000
Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions in deep learning are fragile, 2021.
Fermín S.V. Bazán and Everton Boos. Schultz matrix iteration based method for stable solution of discrete ill-posed problems. Linear Algebra and its Applications, 554:120-145, 2018. ISSN 0024-3795. doi: http:///bbl.gg//11006////aaa
2018.000.....2222220 URL https:
//wwwscieeccdin http://wwww....eeata.taaooo////a
Ratikanta Behera, Krushnachandra Panigrahy, Jajati Keshari Sahoo, and Yimin Wei. m- qr decomposition and hyperpower iterative methods for computing outer inverses of tensors, 2024. URL https://arxiv http://aaaiiiaaaaaaaa///a abs//4/s// 070007.
Adi Ben-Israel and Dan Cohen. On iterative computation of generalized inverses and associated projections. SIAM Journal on Numerical Analysis, 3(3):410-419) 1966.
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth PASCAL recognizing textual entailment challenge, 2009.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference Arlificial Intelligence, 2020.
Yihan Cao, Yanbin Kang, Chi Wang, and Lichao Sun. Instruction mining: Instruction data selection for tuning large language models, 2024. URL /RKL:ttts////rrvv.or////////3/ 06290.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. Alpagasus: Training better alpaca with fewer data, 2024. URL htttps:/aarxwv.oo///bbbbbb 287700
Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, Daniel Salz, Xi Xiong, Daniel Vlasic, Filip Pavetic, Keran Rong, Tianli Yu, Daniel Keysers, Xiaohua Zhai, and Radu Soricut. Palí-3 vision language models: Smaller, faster, stronger, 2023.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictioe uncertainty, visual object classification, and recognising lectual entailment, pp. 177-190. Springer, 2006.
11

---

Published as a conference paper at COLM 2025
William B Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the International Workshop on Paraphrasing, 2005.
Qianlong Du, Chengqing Zong, and Jiajun Zhang. Mods: Model-oriented data selection for instruction tuning, 2023. URL httpt::/arxivv..oo abs/2311 15653.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish T'hite, Noa Nabeshima, Shawn Presser, and Connor Leahy, The pile: An 800gb dataset of diverse text for language modeling, 2020.
James M. Garnett, Adi Ben-Israel, and Stephen S. Yau. A hyperpower iterative method for computing matrix products involving the generalized inverse. SIAM Journal on Numerical Analysis, 8(1):104-109, 1971. ISSN 00361429. URL stable/2949526.
org/
James M Garnett III, Adi Ben-Israel, and Stephen S Yau. A hyperpower iterative method for computing matrix products involving the generalized inverse. SIAM Journal Numerical Analysis, 8(1):104-109 1971.
Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent in kronecker-factoree eigenbasis, 2021.
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning, 2019.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pp. 1-9. Association for Computational Linguistics, 2007.
Yash Goyal, Tejas Khot, Douglas Summers- Staa,, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering, 2017.
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilé Lukošiuté, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel Bowman. Studying large language model generalization with influence functions, 2023a. URL https:///axiv> https://arxxw.org//bb/// 2308. 03296.
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilé Lukošiute, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions, 2023b.
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilé Lukošiūté, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions, 2023c. URL https://arxiis org/abs/22n 2308. 03296.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023.
Han Guo, Nazneen Fatema Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. Fastif: Scalable influence functions for efficient model interpretation and debugging, 2021.
Frank Hampel. The influence curve and its role in robust estimation. Journal of the american statistical association, 69(346))383-393 1974.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), 2023.
12

---

Published as a conference paper at COLM 2025
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
Jiang Hu and Quanzheng Li. Adafish: Fast low-rank parameter-eefficient fine-tuning by using second-order information, 2024.
13128.
Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019.
Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas J. Spanos, and Dawn Song. Efficient task-specific data valuation for nearest neighbor algorithms, 2020.
Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora, 2023.
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually-conddttonee language models, 2024.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. ReferltGame: Re- ferring to objects in photographs of natural scenes. In Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 787-798, Doha, Qatar, October 2014. As- sociation for Computational Linguistics. doi: 10.3115/v1/D44-100
URL https:
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. Qasc: A dataset for question answering via sentence composition. ar arXic:1910.1147322, 2020.
Myeongseob Ko, Feiyang Kang, Weiyan Shi, Ming Jin, Zhou Yu, and Ruoxi Jia. The mirrored influence hypothesis: Efficient data influence estimation by harnessing forward passes, 2024.
Pang Wei Koh and Percy Liang. Understanding black-box prediction via influence functions, 2020.
Shuming Kong, Yanyan Shen, and Linpeng Huang. Resolving training biases via influence- based data relabeling. International Conference on Learning Representations, 2021.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual genome: Connecting language and vision using crowdsourced dense image annotations, 2016.
Frederik Kunstner, Lukas Balles, and Philipp Hennig. Limitations of the empirical fisher approximation for natural gradient descent, 2020. URL hnttps://arxias wwtpixiv.org/bb////r// org/abs/1905. 12558.
Yongchan Kwon and James Zou. Beta shapley: a unified and noise-reduced data valuation framework for machine learning, 2022.
Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora-tuned Ilms and diffusion models, 2024.
Alycia Lee, Brando Miranda, and Sanmi Koyejo. Beyond scale: the diversity coefficient as a data quality metric demonstrates llms are pre-trained on formally diverse data, 2023.
13

---

Published as a conference paper at COLM 2025
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models, 2023.
Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxi Yang, Min Yang, Lei Zhang, Shuzheng Si, Ling-Hao Chen, Junhao Liu, Tongliang Liu, Fei Huang, and Yongbin Li. One-shot learning as instruction data prospector for large language models, 2024. URL https: 10302.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635-651, 2023a. doi: 10.1162/tacla.005500 URL https://aclanthologg wwtp:///hh.oo...oo/222233/ tacl-1.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023c.
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint ar Xie::2007 088124 2020.
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning, 2024. 15685.
URL http:ttp.///avvv.rg//bb////////
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning, 2023a. URL https://arxit: org/ /bss/ 2301. 13688.
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. A pre- trainer's 's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity, 2023b.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cof conference on computer vision and pattern recognition, pp. 3195-3204, 2019.
James Martens. Deep learning via hessian-free optimization. In Proceedings of the 27th International Conference on International Conference on Machine Learning, pp. 735-742, 2010.
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect, 2024.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, 2019.
OPenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tlge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brak- man, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen,
14

---

Published as a conference paper at COLM 2025
Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew P'eng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Fran- cis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thomp- son, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergyaa, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sher- win Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.
Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. 1m2text:
Describing images using million captioned photographs. In Neural Information Processing Systems, 2011. URL https://api. wew.nticscholar........ htt.ro/aar.rrroooooooioiii
14579301.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: Outperforming curated corpora with web data, and web data only, 2023.
Miodrag Petković. Iterative methods for bounding the inverse of matrix (a survey). Filo- mat, 9(3):543-577, 1995. ISSN 03545180, 24060933. URL http:///:/w::tttttto http://www..sstrroor//tttbb.ll org/ 43999236.
Omead Pooladzandi, David Davini, and Baharan Mirzasoleiman. Adaptive second order coresets for data-efficient machine learning, 2022.
Garima Pruthi, Frederick Liu, Mukund Sundararajan, and Satyen Kale. Estimating training data influence by tracing gradient descent, 2020.
15

---

Published as a conference paper at COLM 2025
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100000+ questions for machine comprehension of text. In Proceedings of EMNLP, pp. 2383-2392. Association for Computational Linguistics, 2016.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs, 2021.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge, 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL, 2018.
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension, 2020.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read, 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic com comositionality over sentiment treebank. In Proceedings of EMNLP, pp. 1631-1642, 2013.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremmy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
John Tukey. Bias and confidence in not quite large samples. Ann. Math. Statist., 29:614, 1958.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bow- man. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackhoxNLLP Analyzing and Interpreting Neural Networks for NLP, Pp- 353,355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 20/653//11/1885446 URL https:://eclanthoooo...or////
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019a.
Hao Wang, Berk Ustun, and Flavio P. Calmon. Repairing without retraining: Avoiding disparate impact with counterfactual distributions, 2019b.
16

---

Published as a conference paper at COLM 2025
Jiachen I. Wang and Ruoxi Jia. Data banzhaf: A robust data valuation framework for machine learning, 2023.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments, 2019.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint axXit:2109.01652, 2021.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning, 2024.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions, 2023. URL https:
arggabs/2304.4 1223444.222.. 12244.
Minghan Yang, Dong Xu, Qiwen Cui, Zaiwen Wen, and Pengxiang Xu. An efficient fisher matrix approximation method for large-scale neural network optimization. IEEE Transac- fions on Pattern Analysis and Machine Intelligence, 45(5):5391-5400) 2022.
Yu Yang, Siddhartha Mishra, Jeffrey N Chiang, and Baharan Mirzasoleiman. Smalltolarge (s2l): Scalable data selection for fine-tuning large language models by summarizing training trajectories of small models, 2024.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions, 2016.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, 2019.
Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhi-Quan Luo. Why transformers need adam: A hessian perspective, 2024a. URL https://arxxv.orr//aat 2402.16788.
Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. Adam-mini: Use fewer learning rates gain more, 2024b. URL https://arxiv hrrg/bbs/2440/ 16793.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini lyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment, 2023. htt.httt////aivvoor//aa/////// 11206.
17

---

Published as a conference paper at COLM 2025
A Related Works
Gradient-based Data Attribution Methods. Assessing the importance of each datapoint based on the model's performance is widely studied problem. Traditional methods based on Sharpley-value and LOO (leave-one-out) mechanism often need to train numerous models to get reliable score, which limits their application on large models nor datasets (Ghorbani & Zou, 2019; Jia et al., 2020; Kwon & Zou, 2022; Wang & Jia, 2023). In comparison, by tracing the gradient information from the model, one can value the contribution of each datapoint along the optimization process. Various methods are proposed to assess the data influence tracing first-order gradient (Pruthi et al., 2020). However, those methods risk biasing towards dimensions with larger gradient scales and the uncertainty from stochasticity (Pooladzandi et al., 2022). This could be mitigated by influence function-based methods (Koh & Liang, 2020; Kwon et al., 2024; Agarwal al., 2017), which leverage the second-order curvature information to balance the uncertainty of the first-order gradients.
Data Selection for Foundation Models. High-quality datapoints are shown to improve the base LLM's performance dramatically. Increasing datapoint's quality and diversity can effectively induce the instruction-folloiing ability for large language models (Cao et al., 2024; Chen et al., 2024; Du et al., 2023; Li et al., 2024; Liu et al., 2024). Furthermore, researches on both task-based traditional NLP tasks and open-ended instruction tuning datasets have demonstrated its effectiveness (Longpre et al., 2023a; Zhou et al., 2023; Xu et al., 2023; Wei et al., 2021).
18

---

Published as a conference paper at COLM 2025
B Derivations of Influence Function and its variants
B.1 Influence Function
We provide the proof for Influence Function based on the work of Koh Liang (2020). We have 0 denoted as the minimizer for the empirical risk:
R(0) Σ Elyi, fo(x))
We also assume that the R is twice-differentiable and strongly convex in 0, therefore:
1 R 11
H(0)= = R2) V
E
Ely. fo(x)))
(8)
exists and is positive definite. Then upweighing the contribution of the w datapoint, we have:
"
(() == arg min 8
Σ
L fo(x)) + +el foxx))
(9)
arg min ( <((xx, 8) oce
(10)
Define the change of the parameter A = 0 (c) - 0 and notice that 0 does not depend on e, the quantity we want compute Equation can be re-written as:
( dAe de de
(11)
From previous definition, ((*) is the minimizer for Equation 10, therefore we have the first-order optimality condition:
VaR(o (e)) + eV 0²x) (e)) 0
(12)
We then perform the first-order Taylor expansion of the left-hand side since (() - 0* as
8 - 0:
0 ~ [eR((*) + -Vol(> ( ( A*
(13)
We can further obtain:
Ae - R()) + ev (V) + 0')]
(14)
Because 0* is the minimizer for R(0), we plus (OR(0*) 0 and drop the e-term in the first term of the right-hand side in Equation 14:
A - [())())) (xxx 0r
(15)
Lastly, combining Equation 8 and Equation 11 we can get:
(k(
H Vol
(16)
de
B.2 Influence Function on Validation Loss
In particular, the influence of the upweighing datapoint (xx, Vk) on the loss at a validation datapoint (xyal yyal) also has closed-form formula:
I Iupal ugaa (Xt, yx) == d ((k) (c))
(17)
de
c=0
= Vol(xpal , dl(k)
(18)
de
vol(xpal @+) H (0*) ===0
4
(19)
19

---

Published as a conference paper at COLM 2025
Therefore, when want to evaluate the influence the whole validation dataset, can get a similar formula:
(x) --
V Vol fo(())2)))o0
H (00) Volk
(20)
m
B.3 Full derivation of DATAINF
Kwon al. (2024) proposed a closed-form approximation of the Hessian inverse, which greatly improves the computation efficiency. Firstly, following George et al. (2021), when ap- plying the negative log-likelihood loss function E(y, fx()) log (()]ff(x)), the second- order Hessian is equivalent to the Fisher Information Matrix (FIM) in expectation (Bartlett, 1953), which only involves first-order computations. Consequently, Kwon et al. (2024) approximate the Hessian inverse leveraging the Sherman-Morrison formula 5
H ( -1
V + +Ala
(G(0) +1
Approximation with FIM
Vell Voll + +ala)
Reverse the order of summation and inverse
(21)
Vol.Vli nd A+ Vol, Voli
Sherman-Morrison formula
(22)
where == Vol.Vee ge, stands for the Fisher Information Matrix (FIM). While the computation complexity of Equation 2 is reduced to (d), in compromise, the reverse-order operation Equation 21 incurs a O (d²) error (Kwon et al., 2024). When applying to large-scale models, could risk a large approximation error.
Soor simplicity, denote 8;== fx(:))
20

---

Published as a conference paper at COLM 2025
C Pseudo Code for HYPERINE
C.1 Matrix inverse by the Schulz's Method
We provide the sudo-code for matrix inverse approximation using Schulz's Iterations as in Algorithm 1.
Algorithm 1 Matrix Inverse Approximation via Schulz's Iterations
Require: A matrix A needed to be computed for its inverse, an initial guess X ~ A-1, maximum iteration number Niter-
for E [Nter] do
Iteratively update X X1-1(21 - AX1-1)
end for
return The final approximation A + XN
21

---

Published as a conference paper at COLM 2025
C.2 Influence Score Computation
We provide the complete pseudo algorithm using HYPERINE in Algorithm (2) to compute influence function for each datapoint in training set Dirain according to the impact on the validation set Dvvll
Algorithm 2 Influence Score computed by HYPERINE
Require: A training dataset Ditrain)
(:)) validation dataset D (val) J an objective function f, deep neural network fe(x) fer fei-1 {xx:ra)), yy
o fe, (x), where = ..., } and 0g E R4 for / E [4], HYPERINF's initial guess Xo.1 for [L], HYPERINF's iteration number Naer-
Ensure: Influence Score for each training data point: Zhyperine (x for k = 1,
Step 1: Compute the first-order gradients from validation datasets
for E [I.] do for E [m] do
Compute Ve, Ey, (val) end for
E Rdixr, unflattened gradient
Compute dj 1 T (yy (aal), fo(x)(a)))
end for
# Step 2: Compute the inversion using Schulz's method
for E [L] do
for i E [n] do
Compute (y fx;))) Rdixr, unflattened gradient
end for
Compute E; 0.1 (ndj) Ei_ (o.(y; V (y,
Compute A (0) + e
Compute approximated inversion for Ay: Å -1 + SCHULZ INVERSE Xo,, Niter)
Compute the Hessian-Vector Product: h + tf - E R end for
Step 3: Compute the Influence Score for E [n] do
TyYPERINE (Xx. yk) + E (y
end for
Function to compute an inversion matrix via Schulz's method
procedure SCHULZ X1, Niter)
# Input: A matrix A needed to be computed for its inverse, initial guess Xo for A-1 a maximum iteration number Niter-
# Output: The final approximation X for A
for f E [Nter] do Iteratively update X, = (2-1111-AX-i) end for
Get the approximation for A +- X end procedure
22

---

Published as a conference paper at COLM 2025
D Convergence Analysis of Schulz's Method
In this section, we provide convergence analysis of the Schulz's method. We first give the setup with notations:
Let A R be non-singular matrix, and X₂ be the k-th iteration of the Schulz's method, defined as:
X Xx(2I xx(2I-AXp)
(23)
where Xo is the initial approximation of -1 Define the error at 1 iteration as: R = AXІ- We provide the proof for the following convergence theorems: Theorem D.1. The matrix of error R₃ satisfies quadratic relation. Lee,
R
R
Proof. According to Equation 23, at t iteration, we have:
AX++1 AXx(21 AXx) AXx(I Rk).
Plug into R++1 - AX we have,
R = AX 11 = - AX R2) R = - AX AX
By definition, R I AX 7 AX = - Rs, which gives:
R - (I- RR) - R.
Theorem D.2. The spectral norm of the error decreases quadratically:
| < |x
(24)
Proof. Taking norms on both sides:
R ill ||]]]]
Applying the submultiplicane property of matrix norms:
|| | < || ||&]]
Thus we obtain:
| < |
This proves that the error decreases quadratically with each iteration, provided || Roll <
1.
Theorem D.3. Given the initial condition that the spectral norm of Ro I AXo satisfies Roll < 1, then lo R
lim Xt A-1
Proof. Given Roll < 1, then
|R satisfies:
| S ||
As k - co, |R+ll following the above proved iterative relation || ill S |R exponentially fast. Consequently, as k + co,
X₁ + A
23

---

Published as a conference paper at COLM 2025
E Details for Mislabeled Data Detection Task
Implementation Details. In this task, we choose rank-stabilized LoRA (Kalajdzievski, 2023) instead of original LoRA (Hu al., 2021), for it corrects the one limitation of LoRA (i.e. the performance did not improve further with increasing rank) by simply dividing LoRA adapters by the square root of their rank, which unlocks the effectiveness of higher adapter ranks in LoRA.
We conduct mislabeled data detection experiment on six binary classification tasks based on GLUE benchmark (Wang et al., 2019a), which are GLUF-COLA ((Warstadt et al., 2019), detecting whether a sentence is grammatical acceptable) GLUE-MRPC ((Dolan & Brockett, 2005), detecting whether the sentences in the pair are semantically equivalent), GLUE-QNLI (Rajpurkar et al., 2016), determining whether the context sentence contains the answer to the question), GLUE-QQP (determining whether a pair of questions are semantically equivalent), GLUE-RTE ((Dagan al., 2006; Bar Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009), detecting the entailment), and GLUE-SST2 ((Socher et al., 2013), predicting the sentiment of given sentence).
When finetuning the LLM with rsLoRA technique with rank r = 16 in Figure and r r 64 in Figure 3, apply the gradients from trainable parameters (i.e. every value and query matrix of the attention layers) to approximate influence functions. We run HYPERINE for 25 iterations and run LISSA for 10 iterations following the implementation of Kwon al. (2024). The total number of tunable parameters is 7.3M respectively for = 16, 64.
Moreover, We also experiment using the last layer's gradients of Roberta-lare the mislabeled datapoints. We only tune the last layer of the model on the corrupted training dataset, then compute the influence function based on the last layer's 's gradients. The results are shown in Figure 4, which indicates that the last layer's gradients can also be a candidate for computing the influence function.
detect
24

---

Published as a conference paper at COLM 2025
Comparisons between HYPERINE with GFIM and HYPERINE with FIM To explore using GFIM can lead to performance degradation, we compare HYPERINE with GFIM and HYPERINE with FIM. In this experiment, we set rank 8 since larger ranks (e.g. r = 16,32, would cause the Out-Of-Memory error in FIM. The results are shown in Figure where we do not observe the significantly worse performance in HYPERINF with GFIM, and it performs even better on some datasets than FIM, such as QQP and SS12.
Detection Rate #190
UCC4
QQP
HTE
5512
Dettcctinn Kate POS
Inspection Rata 904)
Irspaction Raba
Inspectan Raca
Figure 2: Mislabeled Data Detection across the GLUE Benchmark with rank r 16 for rsLoRA finetuning. HYPERINF significantly improve the detection rate (rt) according to the inspection rate (p) above all baselines, while LISSA performs barely better than the random guess. The dotted lines denote the detection rates from Random Guess and Oracle, which best possible accuracy at each inspection rate. For each method, we run the experiments with random seeds and report the detection rate with 95% confidence intervals.
Dereetion born noh
(yper Damait
Orack
032
R1T
Dertettonn Park 1E70
Instection Rate 60%)
Inspection Role p05)
Inspection Sate pist
Figure 3: Mislabeled data detection results on GLUE benchmark datasets with rank r = 64, #params 7.3M.
25

---

Published as a conference paper at COLM 2025
COLA
MBPC
214J
Detecton Tate mL/
Lowilnt "ppe
021
3012
Detect on Rare mi)
Inspection Rate 006)
Inspaction Rate p(s)
Inspection Rate pCl
Figure Mislabeled data detection results GLUE benchmark datasets, where influence function computed based on the last layer's gradients.
Derectos fare 0h1
Dasit Chei-rooet Lock
5572
Dertect on Rore 100
D063
Data Inspcctud
Figure 5: Mislabeled data detection results on GLUE benchmark datasets with rank 8.
26

---

Published as a conference paper at COLM 2025
E.1 Analysis of Complexity and Time Costs.
To understand the computation overheads incurred from different data attribution algo- rithms, report both time costs on CPU and one Nvidia A100 GPU according to 6 and on two datasets (COLA and MRPC) from the GLUE benchmark. Specifically, we only record the running time for computing the inverse Hessian vector product v G(0) with different LoRA ranks r = 1,2, 4, 8, 16.
We observe that the efficiency of three algorithms ranks largely differently between GPU and CPU. On CPU, DATAINF introduces least time overheads while HYPERÍNF incurs the most amount of extra time costs. In addition, the time costs from DATAINF and LISSA increase quadratically with LoRA rank r while HYPFRINE increase linearly (note that the y-axis is log scale). Alternatively, on one Nvidia A100 GPU, the time costs from all algorithms are almost constant across LoRA ranks, and HYPERINF costs least of time, followed by DATAINE. In comparison, LISSA requires (~ 4x) more time costs than HYPERINF and DATAINF.
Time cost (CPU) COLA (N=8551)
Time cost (CPU) en MRPC (N=3668)
Time cost (s)
10a
HypertNF
102
Datainf
LISSA
12
Rank
Rank
Figure 6: Runtime on CPU for approximating Hessian-vector product using different methods on GLUE-COLA and GLUE-MRPC datasets.
Time cost on COLA (N=8551)
Time cost on MRPC (N=3668)
400
150
Time (5)
HypentNF LISSA
100
200
75
50
100
25
12
Rank
Rank
Figure 7: Runtime on GPU for approximating Hessian-vector product using different methods on GLUE-COLA and GLUE-MRCC datasets. HYPERINF takes lowest time costs compared to other methods.
27

---

Published as a conference paper at COLM 2025
E.2 Correlation with Leave-One-Cropp-Out (LOGO) Scores.
The performance of a training data attribution (TDA) algorithm can be assessed by its ability to recover the true Leave-One-Out (LOO) score (Tukey, 1958) The LOO score of given datapoint X; is defined as the gap of validation losses of a model before and after removing the certain datapoint. To prevent the large computations incurred from retraining LLMs, (LOGO). Firstly, we rank all training datapoints according to assigned scores and split them equally into K groups from high to low scores (K 5 in our experiments), On each group of data C₁ we iteratively remove C and retraining the LLM on the remaining set of data Dtrain/C;. We define the LLM trained on the full training set 0g and the LLM retrained with removing Ci B/C; Then we measure the LOGO score as:
we evaluate the TDA algorithms with Lerve-On-Group--ee
LOGO(C) = L(0/c,, Doa) - L(0,, D
(25)
If C; contains high quality datapoints, excluding C; would hurt the model's performance and lead to an increment of validation loss. Therefore, the LOGO score proportional to the data quality within the group. In that case, we measure the rank correlation between the average influence score assigned to all groups and the corresponding LOGO scores. We report the spearman rank correlation scores on all four algorithms across six datasets in GLUE benchmark in Table The results demonstrate HYPERINF outperforms all the other baselines on the accuracy of data attribution.
Method (LoRA)
DATAINF
LISSA
TRACIN
HYPERINE
COLA
0.50
0.49
-0.99
0.70
MRPC
0.0
0.0
0.0
0.20
QNLI
-0.40
-0.30
-0.60
0.10
QQP
0.30
0.49
-0.30
0.70
RTE
0.60
0.60
0.40
1.00
SST2
-0.90
-0.30
-0.10
0.70
Table 6: Spearman Rank Correlation.
28

---

Published as a conference paper at COLM 2025
E.3 Illlstrations of Highly Influential Examples
For the mislabeled data detection task, we also provide the top-ranked data point (i.e., high-quality data, where its label correct ) and bottom-ranked data point (i.e., low-quality data, where its label is corrupted to be incorrect) from HYPERINF on MRPC task for better illustrations.
Top-ranked: ('sentence1': 'Microsoft which acquired Virtual PC from Connectix in February said a fix for the problem is not around the corner "sentence2': 'Virtual PC which Microsoft acquired from Connectix Corp. in February will not run on the G5 the company said .', 'label's : 9). Label 0 denotes the these two sentences are not semantic equivalent, whose label is correct, not corrupted.
Banked-ranked: {' ('sentence1': .. We acted because we saw the existing evidence in new light through the prism of Sept 11th sentence2': .. We acted because we saw the evidence in a dramatic new light through the prism of experience
11 The label should be 1, since these two sentences are semantic equivalent but the
'label': 0). label is corrupted to And HyperINN succeeds in detecting the wrong-labeled data point.
29

---

Published as a conference paper at COLM 2025
F
Data Selection for LLM Finetuning
Dataset Details. We run the experiments on four LLM reasoning tasks: QASC (a question- answering dataset with focus on sentence composition. It consists of 9, 980 8-way multiple- choice questions about grade school science) (Khot et al., 2020), HellaSwag (a challenging dataset for evaluating commonsense NLI) (Zellers et al., 2019), PIQA (a dataset introducing the task of physical commonsense reasoning) (Bisk et al., 2020) and LogiQA (is constructed from the logical comprehension problems from publically available questions of the National Civil Servants Examination of China) (Liu et al., 2020). For LogiQA, we the official validation set D in data selection and use labelled official test set for evaluation; for other three datasets, since the labels for the official test set are not available, we randomly split 20% from the official validation set as Dowl and use the rest 80% validation set as the held-out test set.
Implementation Details. For LoRA-finetuning we follow the same setting ment in Mislabeled Data Detection task while setting the rank r = 64. The hyperparameters are as the same as in VLM experiments (Table 7), while the Epoch number is to for fully-finetuninn and 5 for LoRA-finetuning across k = 5%, 20%, 40%. When selecting all datapoints (i.e. 100%), we finetune for only 1 epoch.
imple-
Evaluation Statistics. We present the detailed statistics of evaluation results in Table and Figure for LoRA- finetuinnn experiments, and Table and Figure for fully-finetuning experiments. HYPERINF significantly outperforms all baselines.
QASC
HellaSwag
PIQA
LogiQA
40
89.5
56
Accuracy
28
30
54
HyperiNF
89.0
20
LSSAA
26
TradNN
52
10
Random
88.5
0.1
0.2 Ratio
0.3
0.4
0.1
0.2
0.3 Ratio
0.4
0.1
0,2
0.3
0.4
0.1
0.2
0.3
0.4
Ratio
Ratio
Figure 8: Evaluation accuracy according to data selection ratio (k) for LLM LoRA-finetuning. HYPERINF greatly improves the reasoning accuracy above other baselines.
QASC
HellaSwag
PIQA
LogiQA
40
85
Accuracy
40
35
80
30
HyperrlNF
50
LiSSA
75
TracNN
fandom
10
70
0.1
0.2
0.3
0.4
0.1
0.2
0.3
0.4
0.1
0.2
0.3
0.4
0.1
0.2
0.3
0.4
Ratio
Ratio
Ratio
Ratio
Figure 9: Evaluation accuracy according to data selection ratio (k) for LLM fully-finetuning. In- fluence scores are computed based on the gradients of the last layer of LLM. HYPERINF shows significantly better performances above other baselines especially when k 5%.
30

---

Published as a conference paper at COLM 2025
G Data Selection for VLM Pretraining
G.1 Details of VLM Architecture and Training Strategy
Following LLaVa (Liu al., 2023c), we adopt the commonly used VLM architecture which consists of three components: a vision backbone Vg, a projector Fy and a language backbone LMg. Both the vision and language backbones are pre-trained, while the projector is randomly initialized and would be tuned through the alignment and instruct-tuning phases using multimodal data (Karamcheti et al., 2024; Liu et al., 2023c; Bai et al., 2023; Chen et al., 2023). We follow the auto-regressive training paradigm of vision-language models, where the images are tokenized into patches (i.e. visual tokens) to fit into the conventional training patterns of language models. Specifically, each datapoint in multimodal instruct- tuning dataset can be represented as a tuple (Nimg Xtext). We get a sequence of embeddings of the image patches through the vision backbone Pimg = VØ (Ximg) then feed it into the projector to obtain the transformed features fimg Fu (Pimg) Meanwhile, we have the embeddings from textual tokens as etext LMg (Xtext). We then concatenate the features from both modalities together to conduct next-token predictions. In our experiments, we apply CLIP ViT-Large (Radford et al., 2021) with patch size of 14 and input resolution of 336px as the vision backbone and L ama2-7B (Touvron et al., 2023) as the language backbone. For the projector Fig, initialize two-layer GELU-MLP (Hendrycks & Gimpel, 2023). Along the suggested setting from Karamcheti et al. (2024), we freeze the vision backbone V throughout the entire training process while only tuning the projector Fi and the language backbone LMg-
Specifically, we utilize the Prismatic- VLM framework (Karamcheti et al., 2024) to train the VLM. We use 6xA100 80G GPUs to train the model, and the hyperparameters are set as Table 7.
Table Hyperparameters setting for training VLM
Hyperparameters
Values
Epoch
Optimizer
AdamW 2e-5
Learning Rate
Weight Decay
0.1
Max Grad Norm
1.0
Warmup Ratio
0.03
Batch Size per GPU
16
Scheduler
Warmup & Cosine Decay
G.2 Details of VLM Dataset
Instruct-tuning Dataset. We follow the work of Karamcheti et al. (2024) and this dataset contains 665K multimodal instruct tuning examples Liu et al. (2023b) has identified a set of "trigger prompts" for each dataset in the mixture, to induce more capabilities of VLM. The datasets are sourced as follows, where we removed ShareGPT (language-only) in our experiments. We split it into a training dataset and validation dataset as 8 2 ratio.
LlaVa Synthetic Data (158K): A synthetically generated dataset of conversations, fine-grained descriptions, and question-answering data from Liu et al. (2023c), built by prompting GPT-4 (OpenAl et al., 2024) with image captions and object bounding boxes from COCO (Lin et al., 2014).
Standard VQA Data (224K): A combination of visual question answering data sourced from the training sets of VQAv2 (general question answering) (Goyal et al., 2017), GQA
7 httpps: peettp::/ggee //github.b....... www.cooc//////o com/TRI-MM -AAP/TRRIMM//AA
tab-mmmtbbrrddeeeooo file
8tt can be downloaded following the instructions of httt::://ggihuu.ccc//ttt////////
com
31

---

Published as a conference paper at COLM 2025
(spatial and compositional reasoning) (Hudson & Manning, 2019), OK-VQA (reason- ing requiring external knowledge) (Marino et al., 2019), and OCR-VQA (reasoning over text/logos in images) (Mishra et al., 2019). LLaVa v1.5 defines the following trigger prompt: " "(Question)? Answer the question using a single word or phrase."
Multiple Choice VQA Data (50K). Multiple choice visual question answer- ing data sourced from A-OKVQA (requires diverse external knowledge) (Schwenk et al., 2022). LLaVa v1.5 defines the following trigger prompt: " (Question)? A. (Option A) B. (Option B)... Answer with the option's letter from the given choices directly."
Captioning Data (22K). Images and captions sourced from TextCaps (images with text/logos) (Sidorov al., 2020). LLaVa v1.5 defines the following trigger prompt: "Provide one-sentence caption for the provided image."
Referring Expression Data (116K). Referring expression grounding (bounding box predic- tion) and region captioning data sourced from RefCOCO (Kazemzadeh et al., 2014; Yu et al., 2016) and Visual Genome (Krishna et al., 2016). For bounding box prediction (lo- calization), the model needs to generate normalized bounding box coordinates (as nat- ural language string). For the localization task, LL v1.5 defines the following trigger prompt: " "(Referring Expression) Provide the bounding box coordinates of the regionthis sentence describes."
For the inverse task (region caption), LLaVa v1.5 defines separate trigger prompt: "Provide the bounding box coordinate of the region this sentence describes." "
G.3
Data Selection after Cross-Modal Alignment With Projector and LoRA of Language Backbone
Details of Cross-Modal Alignment. We keep the same hyperparameter setting as in Table 7 and adopt LoRA to the language backbone. We keep the same LoRA setting in the LLM Affettnnuuning... In the alignment phase, we tune the projector and LoRA layers while keeping other parts frozen. We use the Vision-Language Alignment dataset (Karamcheti et al., 2024), which consists of 558K (image, caption) pairs, where the caption is sentence description of the corresponding image. The images are sourced from LAION (Schuhmann et al., 2021), Conceptual Captions (Sharma et al., 2018) and SBU Captions (Ordonez et al., 2011). Considering the limited computation resources, we randomly select 5% datapoints from the alignment dataset for the alignment phase. We leave the larger-scale experiments to future work.
Details of the Instruct-tuning. Because of the limited computation resources, we constrain our experiments on 10% of instruct-tuning training dataset used in G.2. We compute the influence function based on the gradients from both Project and LoRA layers, then select == 20%, 40% datapoints using various influence function-based methods from the 10% training subset, which is equivalent to 0.5%, 2%, 4% of the original 665K instruct-tuning dataset. In this experiment, we also finetune the projector and LoRA layers of the language backbone and keep other parts frozen.
G.4 VLM Pretraining Before Cross-Modal Alignment
Setup. Karamcheti al. (2024) illustrated from extensive empirical experiments that only applying instruct-tuning can achieve comparable performant pretrained VLMs as the conventional two-phase training (cross-modal aligument then instruct-tuninng) for LLaVa (Liu et al., 2023c). Thus, we hereby skip the aligmment phase in LLaVa (Liu et al., 2023c) and aim to select the most beneficial multimodal instruct-tuning datapoints for more efficient VLM pretraining (instruct-tuning not suitable for computing influence function, we use the gradient of the last layer of the pretrained language backbone for HYPERINF and all baselines, to select the datapoints. In this experiment, we compute all instruct-tuning training datapoint's influence score of each method, then select the top-k% (k 20%, 40%, 80%) subset with the lowest scores. During
only). Since the projector is randomly initialized which is
32

---

Published as a conference paper at COLM 2025
instruct tuning of this experiment, tune the projector and the whole language backbone while keeping the vision backbone frozen.
Results. We present the evaluation accuracies on four multimodal downstream tasks in Table 8. Notably, when selecting k 20% of datapoints, HYPERINF improves the accuracy in average by 7 7.20% above DATAINF, 8.37% above LISSA and 9.11% above TRACIN. However, we also note that when the selection ratio gets larger (k 40%), the performance of other baselines will approach HYPERINE, since the impact from approximation errors on the data ranking is mitigated. Meanwhile, we observe that the random selection is a very strong baseline for all tasks, where only HYPERINE has a small improvement above the random baseline (0.25%) in average accuracy while all the other methods cause large performance degradation (> 5%). We hypothesize that using pretrained LLM backbone without leveraging cross-modal alignment information may lead to sub-optimal results.
Evaluation Statistics. We present detailed statistics for downstream evaluations in Table 8 and Figure 10. HYPERINF greatly improves the accuracies across all tasks above the other data selection baselines, while the random selection is strong baseline. When selecting 20% subset, HYPERINT the only method that could outperform random selection according to average accuracy.
Skip alignment in training, not data selection. (Karamcheti et al., 2024) demonstrated only apply Instruction-tuning on VLM can achieve comparable performance two-phase training (Liu et al., 2023c). To investigate whether this applies to data selection, we applied HyPERINF, DATAINE, LISSA, and TRACIN before alignment. Since projector gradients are randomly initialized prior to alignment, we only used gradients from the last transformer block in the language backbone to compute influence scores. As shown G.4, while HYPERINF still provides slight improvement (0.25 1%) over random baselines, the other three methods suffer significant degradation (2 5% 4) in accuracy. We hypothesize that the alignment phase crucial for learning connections between language and vision feature Spaces-informatte essential for effective VLM pretraining data selection. Therefore, we recommend applying data selection after the alignment phase.
Table 8: Downstream evaluation accuracies (%) from VLM instruct- tunitttuunngggg data selection experiments (before cross-modal alignment). The best results are Bolded and the second-best are Underlined. The gradient from the last layer the language backbone is used to compute approximated scores. HYPERINF could outperform the Random baseline while the other methods fail when selection ratios small. The (L) indicates the improvement emeet (degradation) compared to the Random baseline. Methods with 5% accuracy degradation marked in Red.
Method (4%)
Random
DATAINF
LISSA
TRACIN
HYPERINE
20%
71.30
66.91
66.20
65.33
70.40
VQAv2
40%
74.84
75.35
75.92
75.84
75.27
60%
76.29
75.35
76.99
76.95
76.89
20%
55.92
53.29
52.23
51.03
57.97
GQA
40%
59.83
60.95
62,41
61.76
61.63
60%
61.49
62.97
63.11
62.62
63.35
20%
86.11
86.04
85.52
85.04
85.66
POPE
40%
86.58
85,98
86.39
86.52
86.91
60%
87.00
86.63
86.40
86.99
86.92
20%
36.20
15,50
13.10
12.70
36.50
TextVQA
40%
45.00
45.60
44.90
44.90
45.70
60%
47 47.60
49.40
48,90
49.20
49.20
20%
62.38
55. 55.43
54 54.26(8.12.)
53.52 53.52(8.96L)
62.63
Average
40%
66.56
66.97 (0.411)
(6.17)0.41))
67 (7.25
67.40
67. 67.38pp.211
60%
68.09
68.59 (0.50)
68.85
68. (8.94
69.09
33

---

Published as a conference paper at COLM 2025
VQAv2
GQA
POPE
TextVQA
50
75
Accuracy
86.5
40
Datalnf
30
70
HyperiKkF
55
85.5
TraciNN
Random
65
85.0
0.2
0.4
0.6
0.2
0.4
0.6
0.2
0.4
0.6
0.2
0.4
0.6
Ratio
Ratio
Ratio
Ratio
Figure 10: Downstream evaluation for VLM instruct-tuning data selection (before cross-modal alignment). HYPERINF benefits the most when selecting small subset k 20%, from its accurate approximation influence function. With increasing, the performance other baselines approach HYPERINE, since the impact from approximation errors is mitigated. Random selection is strong baseline for all data selection methods.
34

---

Published as a conference paper at COLM 2025
H
Comparison between Matrix Inversion Algorithms
Implementation Details. In this section, we compare the efficiency of computing inverse of matrices between Schulz's method and other commonly used methods', including Gaussian Elimination, Conjugate Gradient, Generalized Minimal Residual method (GMRES) and Faster Gaussian Elimination (i.e. torch. inverse). For the iterative methods, we all set the number of iterations
for fair comparisons. We follow the same step in Section. 4 to construct the invertible matrix M, and set the dimension of the matrix in different scales: {16, 64, 256, 1024, 40%6 and N 12800. We use the Frobenius Norm to measure the error between the approximated and true inverse, where set the Gaussian Elimination as the ground truth. addition to the error comparison, we also compare the time cost of each method in terms of efficiency aspect. We run the experiments with 3 random seeds and report the average and standard deviation of time costs. All the experiments are done with single A100 GPU.
Results. The comparisons of error and time cost are shown in Table 9 and Table 10 as well as Figure 11. Schulz achieves a similar error margin as FGE, which is better than CG and GMRES in most cases. Furthermore, Schulz also has the lowest time cost generally in different dimension settings even when = 4096, while other methods observe a signifi- cant increase in running time ranks become larger(especially for Gaussian Elimination, Conjugate Gradient and GMRES). This illustrates the efficiency and stability of HYPERINF since Schulz's method is the main part of our method.
Table 9: Error comparisons among different methods for computing the inverse of the matrix. CG, and FGE denote the Conjugate Gradient and Faster Gaussian Elimination respectively, We reimplemented all the algorithms in torch if the original implementation does not support GPÚ acceleration.
Matrix Dim
CG
FGE
GMRES
Schulz
16
3.5e-10 112e-10
3.0e-1 11
131e-12
1.3e-10
142-11
4.2e-11 11 15.1e-12
64
9.7e-10 +52--11
8.7e-11
48.6-12
1.6e-10
+1.6-11
1.4e-10
+3.9-122
256
9.9e-9 +366-10
3.9c-10
#1.1e111
8.9c-10
#1.3e-10
5.4c-10
#1.3e-11
1024
1.2e-8 153e-10
2.1e-9 -18e-11
3.7e-9 138e-11
2.5e-9 13.1e11
4096
1.2e-7 +51e-10
2.1e-8 =19e-10
1.5e-7 +75e-10
2.7e-8 +200-10
Table 10: Time cost (s) comparisons among different methods for computing the inverse of the matrix. GE, CG and FGE denote the Gaussian Elimination, Conjugate Gradient and Faster Gaussian Elimination respectively. We reimplemented all the algorithms in torch if the original implementation does not support GPU acceleration.
Matrix Dim
GE
CG
FGE
GMRES
Schulz
16
0.04 +0.2
0.11 +0.05
0.02+a.03
0.41 +0.02
0.002 +o.02
64
0.31 10.02
0.4310.3
0.01
2.2710.17
0.0008
100001
256
2.55
2.37
0.001 -0,000
-.001-a.000
12.7+0.11
0.002
1024
23.7+a10
14.6+ 0.06
0.007
77.1 -0.44
0.002
4096
313.81222
107.9_s.1
0.0710.009
581.6
0.001
35

---

Published as a conference paper at COLM 2025
10h
Goudeee
PGZ Bur Surchiieeree
Schutt
GPRILS Lordh
GARES (edbyy
10
00
Time
264
1621
4266
Varra Simension
Figure 11: Time cost comparisons among different methods for computing the inverse of the matrix. Schulz presents superior efficiency than other methods.
36

---

Published as a conference paper at COLM 2025
I Supplement Results of Convergence Test on Matrix Inversion
We follow the same setting as in section and construct matrices M = Ñ E s/8 + A1 e
Raxd
To study the convergence with various data distribution and initialization condition, we report the results with s; and a vectors drawn from 5 difference distributions:
Each element of si and v are drawn from Standard Normal Distribution: ( 1)
Each element 9; and v are drawn from Normal Distribution: (0.5,1)
Each element of 6; and v are drawn from Normal Distribution: (O,
Each element of s; and
drawn from Normal Distribution: (0.5,55)
Each element of and drawn from Uniform Distribution: U(0,1)
We also include the Neumann Series (which is the same method of LiSSA) and Successive
Over Relaxation (SOR) methods compare. For SOR, the iteration shown as:
x(k+1)= (D wL) (woll (1 w))) woDDwwL)
(26)
where D, I., U denote the diagonal, lower and upper triangular parts of M. a is a hyperpa- rameter, when w > it overrelaxation, and when as it is underrelaxation. We choose 0.5, 1.5 for experiments. To measure the error for all methods, we use the Frobenius norm of the matrix | Ql(r.
The results are shown as Figure 12, Figure 13, Figure 14, Figure 15, and Figure 16. HYPERINF with Schulz's 's algorithm demonstrates remarkable stability and convergence performance, which is robust with various data distribution and initial conditions. LISSA only converges in a few circumstances, indicating it's sensitive to the initial condition and matrix distributions. For SOR, only when the data distribution is from NN 1) (see Figure 12 and Figure 13) it can converge in limited circumstances.
Results.
37

---

Published as a conference paper at COLM 2025
HyperiNF N=200
HyperINF 8=800
HyperINF N=6400
Hyper/NF N
dim
250
5250
d0m-
0000
dim-4696
3069
2003
3003
2308
12
20
Iteration
LISSA N=20D
LISSA N-800
USSA N-6400
LISAA N-12800
10-
Log of Error
16'
16
d1m
167
dim-2048
dimm 1024
30"
507
10m
30
20
33
30
46
33
20
Iteration
Iteration
Iteration
Iteration
Datainf N=200
Dotainf N=800
Datalnf N-6400
Datainf N= N=12800
2363
6050
6001
3463
4530
2300
4350
3480
3550
2030
1363
2208
2530
2033
2000
3003
4080
2000
2000
3909
4003
1300
2000
3000
4363
1000
2000
3000
6000
Dimension on
Dimension
Dimension
Dimenslon d
Neumann N=200
Neumann N=800
Neumann N=6400
Neumann N=12800
10-
Lag of Error
1675
1677
16th
dim-512
167
10
1024
10/
10-4
100
dimm4096
10
10
20
30
33
33
20
Length
Length
Series Longth
Longth
200
SDR N-800
SOR N-6400
N-12800
dim-512
d0m-2048
40m-
10"
-2.54 -0.2 0.30
6.4
80
53 75 136 225
23 20
46
23 20
20
Iteration
Iteration
Iteration
Iteration
Figure 12: Convergence test of HYPERINE, LISSA, DATAINF, Neumann Series, and SOR (w 0.5). We construct M N 1 + and apply various methods to approximate the inverse Hessian- vector product M M where E Rd, e Id are randomly generated, each element is from the Standard Normal Distribution (0 1). Only HYPERINF For LISSA, it does converge some cases (e-g. 6400, dim 512), but would diverge when dim larger. SOR only converges when N is large and dim is small.
converge to low error rate in all cases.
38

---

Published as a conference paper at COLM 2025
HyperiNF N=200
HyperINF N=800
HyperINF N=6400
Hyper/NF N=12800
0003
dim
6268
5003
5260
4063
dim-4696
d0m-
4309
130
2003
370
3003
3000
12
20
Iteration
LISSA N=200
LISSA N-800
USSA N-640D
LISA N-12800
16.-
Log of Error
16"
164
1443
10"
18-
d1m=512
304
187
dim-2048
dimm 1024
307
107
10m
53
15
23
30
20
33
30
46
33
20
Iteration
Iteration
Iteration
Iteration
Datainf N=200
Dotainf N=800
Datalnf N-6400
Datainf N=12800
2000
5800
3453
2768
5830
5830
2100
3890
2308
2003
2000
3003
4000
2000
2000
3909
4003
1300
2000
3000
4369
3000
2000
3000
6000
Dimension on
Dimension
Dimension
Dimenslon d
Neumann N=200
Neumann N=800
Noumann N=6400
Neumann N=12800
16.9
Log of Error
1673
1671
16711
18
dim=512
107
18
1024
30"
10
50
10
20
30
33
33
20
30
Length
Length
Series Longth
Longth
N- 200
SDR N-800
SOR N-6400
N-12800
dim-512
30/4
3577
d0m-2048
40m-
2012
107h
-2.54 -0.2 0.30
6.4
20
46
23 20 20
Iteration
Iteration
Iteration
Iteration
Figure 13: Convergence test of HYPERINE, LISSA, DATAINF, Neumann Series, and SOR (w 1.5). We construct M N 1 +AI and apply various methods to approximate the inverse Hessian- vector product M M where E Rd e Id are randomly generated, each element is from the Standard Normal Distribution (0 1). Only HYPERINF For LISSA, it does converge some cases (e-g. 6400, dim 512), but would diverge when dim larger. SOR only converges when N is large and dim is small.
converge to low error rate in all cases.
39

---

Published as a conference paper at COLM 2025
HyperiNF 200
HyperiNF N=800
HyperiNF N-640D
HyperINF N 12800
dmm-12
dimm 024
dmm-2048
4003
dmm40066
Iteration
Iteration
Iteration
iteration
LISSA N-200
LISSA N BOD
LISSA N-6400
LISSA N N- 12800
10.0
23th
35th
Log of Etror
1444
16
12
23
334
1024
dm-2048
155
dm-4090
Iteration
Iteration Datainf N-800
Iteration Datainf 4-6000
Iteration
Datainf N-200
Dataiin N-12800
4033
400
2403
300
4030
4002
2008
Dimensior
Dimension Neumann N-800
Dimension Neumann N-6400
Dimension
Neumann N-200
Neumann N- 12800
15.-
20th
15t0
Log of Error
1571
dmmn512
1575
dmm11044
dmm22488
109
dmm4406
Series Length
Series Length
Series Length
Series Length
SOR N-200
SOR N-800
SOR N-6400
SOR N-12800
dm-512
dm-1024
dim-0095
Log of
Iteration
Iteration
Iteration
Iteration
Figure 14: Convergence test of HYPERINE, LISSA, DATAINE, Neumann Series, and SOR (w 1.5). construct M N !
and apply various methods to approximate the inverse Hessian- vector product v, where s; E R² & Rd are randomly generated, each element from th Normal Distribution A (0.55 Only HYPERINE can converge to a low error rate in all cases. For other methods, they all diverge. For SOR, it has the nan issue.
40

---

Published as a conference paper at COLM 2025
HyperiNF 2-200
HyperiNF N-800
HyperiNF N- 6400
HyperiNF N 12800
4440
dmm-522
dimm 024
dmm=2048
dmm4006
15600
16600
Iterattio
Teeation
Iteration
Iteratioo
USSA N--00
USSSA Ni800
USSA N 6400
USSA N 128000
3077
1951
107
Emor
2600
20
337
30-
2014
237
dmm11024
101
18
dm-2048
dm-4409
101
38
Iteration
Ieeraton Datalnf N-800
Iteration Datalnf N-6100
Iteration
Datalnf N-200
Datainf N-12800
22500
14200
1430
25800
1230
19000
22530
20300
0057
12000
25300
35032
11580
Dimension d Neumann N-200
Dimension Neumann N-800
Dimension Neumann N-6400
Dimension d
Neumann N- 1-800
367
194
10a
10--
2013
102
4mmn512
Log
dmm11024
38
dmm22088
dm-4096
107
10
10'
Series Length
Series Length
Series Length
Series Length
SOR N-200
SOR N-800
SOR N-6400
SOR N-12800
dm-512
10
dm-1024
Log of Erro
dim-0095
Itoration
keration
Iteration
Iteration
Figure 15: Convergence test of HYPERINE, LISSA, DATAINE, Neumann Series, and SOR (w 1.5). construct M N 1 !
and apply various methods to approximate the inverse Hessian- vector product v, where s; E Rd, E R4 are randomly generated, each element is from the Normal Distribution (0,55) 5). Only HYPERINF can converge to à low error rate in all cases. For other methods, they all diverge. For SOR, it has the nan issue.
41

---

Published as a conference paper at COLM 2025
HypertNF
HyporiNF
HypertNF
HyperiNF
232
150
dim=2048
4m-4056
Itorattion
Iboratton
Reration
Boration
Nm000
US
Log of Enorr
337
13"
4mm4020
Iteration
Iteration Datainf N=:00
Reration Datainf N4
Ber alion
Datainf N=200
Datalin N 12-0000
1000
340
1200
200
Dimension
Dimension Neumann N=000
Dimension Neumann N=6400
Dimension d
Neu mann N=200
Neumann N=80000
13.-
39"
13"
Log E370r
16"
16a
19
Length
Length
Length
Langth
SOR N-200
SOR N=800
SOR N-6400
SDR N=12800
0.22 6.00 122
1.44 1.37 022
24 -3.0 1.2 CS6
-000
Figure 16: Convergence test HYPERINE, LISSA, DATAINF, Neumann Series, and SOR (w 1.5). We construct M N 1 S AI and apply various methods to approximate the inverse Hessian- vector product M v, where s; & Rd, & Rd are randomly generated, each element is from the Uniform Distribution (0, 1). Only HYPERINF can converge to a low error rate in all cases. For other methods, they all diverge. For SOR, it has the nan issue.
42

---

Published as a conference paper at COLM 2025
J Discussion and Limitations on FIM and GFIM approximation in Influence Function Computation
J11 Limitations of FIM Approximation of Hessian Matrix
While the Fisher Information Matrix (FIM) have been widely applied to approximate the Hessian matrix (Bartlett, 1953; Kwon et al., 2024), we recognize that some infeasible conditions required by Equation 3 cannot be met in realistic LLM training cases, which might cause discrepancies and undesirable downstream effects. Firstly, Equation 3 only stands when the model is nearly converged, which can hardly be achieved when train LLMs; Besides, Equation requires that the labels y are drawn from the distribution ply|x, [ty]x,0) 0). While the ground-truth labels normally used y in influence function computation. From the optimization point of view, using FIM to approximate second-order gradients or curvature during training could lead sub-optimal optimization outcomes, wuch as adverse distortion of the gradient field (Kunstner et al., 2020). For more detailed and complete studies of FIM and hessian matrices, we refer the readers (Kunstner et al., 2020).
J.2 Limitations of GFIM Approximation of FIM
In Theorem 3.1, we make the idealized assumption that each column in the gradient matrix g is independently and identically distributed (i..d.) following a distribution with zero-mean. However, we demonstrate that this assumption may not be strictly valid in realistic cases of large langauge model training.
According to Figure 17, we visualize both the fisher information matrix (FIM, vec(g) vec(g)") and expended generalized fisher information matrix (GFIM, 1, @ {gg") of gradient matrices from LoRA finetuning on the MRPC dataset.
In Figure 18, we constructed a 16 x 1000 matrix by sampling each column from a standard guassian distribution with zero-mean and one-variance independently and identically. We then plot the FIM and expended GFIM matrices of the given matrix.
In practice, FIM and GFIM show some differences, especially with randomness and complex dynamics during LLM training. However, it does not impact the empirical performance of our method according to the improvement from our comprehensive experiments. How to derive more accurate low-rank approximation of Hessian matrices within tractable computations an important and compelling research topic. We will leave for future work.
43

---

Published as a conference paper at COLM 2025
Diferess Sclmoan 3TV sndfMV
10:00
1090
1282
10:00
2005
1807
5000
1WB WIKE
2000
14:00
2006
000 8000 1U80E 5480
39.0
Figure 17: GFIM and FIM of Gradient Matrices from LoRA fine-tuning (r=16)
Lefferenca FMM
2000
2000
0000
1000
1200
3200
Figure 18: GFIM and FIM Matrices sampled from Standard Gaussian Distribution
44

---

Published as a conference paper at COLM 2025
J.3 Linear Independence of Matrix Columns
In realistic LLM training, it is hard to justify the i.i.d. assumption made in Theorem 3.1. However, we provide the empirical evidence that each column in the gradient matrices are linear independent with each other. Specifically, the rank of the gradient matrix should be equal the number of columns, i.e, the LoRA rank in low-rank fine-tuning.
We hereby compute the rank of each gradient matrix across all training data points from MRPC dataset and present the distribution of matrices ranks in Figure 19 and Figure 20. With r=8 and r=16, most of (> 90%) gradient matrices are with full column ranks, which shows that Theorem 3.1 stands in real low-rank tuning cases, In addition, we also compute the difference between GFIM and FIM in the above same setting (r 16 in this experiment).
Rank Distribution
300000
Frequency
200000
100000
50000
Rank
Figure 19: Rank Distribution of Gradient Matrices with r=8.
Rank Distribution
300000
Frequency
250000
1550000
100000
8
12
34
16
Rank
Figure Rank Distribution of Gradient Matrices with r=16. Rank distribution of gradient matrices on MRPC. More than 90% matrices are with full column rank, which justifies our linear dependent assumption in Theorem 3.1.
45


Reviews:
[{"review_id": "i2yFyBxj4E", "reviewer": "Reviewer_fWxU", "summary": "The authors focus on addressing the computational costs associated with data attribution by replacing the Hessian with a generalized Fisher Information Matrix. The paper then shows through several experiments (mislabel detection, data selection, multimodal fine-tuning) to back up the stability and performance of HyperINF.\n\nQuality and clarity: Overall the paper is generally structured well, and I can understand the core contributions the authors intend to make. I appreciate the work the authors made to produce experimental results and to benchmark previous methods.\nOriginality and significance: I am concerned there may be limited novelty due to overlap with previous literature and lack of evidence to strongly disentangle it both in theory and experimentation.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors present and show experimental results for another method to approximate the Hessian. This could provide another interesting tool in the data influence toolbox for researchers in this field.", "reasons_to_reject": "The paper outlines how it differentiates from previous work, but I think there isn't enough separation on the theoretical or experimental side. The paper aims to supplement the limitations of DataInf's O(d^2) approximation error, but it's not clear that HyperINF is much better (eg. dense fine-tuning only occurs on the final layer, and it's not clear it is the right experimental setting since d is related to rank of weight updates). Additionally, the substitution of Sherman-Morrison with Schultz method may actually detract from a desirable trait of DataInf -- namely that it replaces an iterative method with a closed-form solution.", "questions_to_authors": "1. Are there confidence intervals for the performance of methods in Table 3 and Table 4? Without these, it's hard to understand what the percentage improvements translate to (and the significance of the results).\n2. LoRA was done with r=64, but does this represent a common LoRA implementation? (eg. the original LoRA paper goes from 1->64 rank) Can you justify this decision or provide results for lower rank? Similarly, dense fine-tuning is only applied to the last layer -- should this experiment be applying full-fine tuning of the same layers that are using LoRA approximation?\n\nI will consider moving my review up if the authors can better explain how HyperINF overcomes limitations in datainf's approximation errors with respect to the rank of the weight updates.", "extra_scores": {}}, {"review_id": "Ldl4kHy8q8", "reviewer": "Reviewer_ZA5B", "summary": "This paper is well-written and easy for readers to follow. It proposes an original and novel method for efficiently and accurately approximating influence functions, an important topic for us to understand model interpretability and data quality. The results show high effectiveness and the theoretical analysis shows clear efficiency improvement compared to existing methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Influence function is a principled way to evaluate training data's effect on model performance but its challenge of high computational costs of its matrix inversion has limited the application in LLMs. The Schultz method-based approximation in this paper is theoretically sound and practically effective.\n* The authors conducted comprehensive evaluations on large-scale VLM and LLM models, showing the scalability of the proposed method, even on dense finetuning. The results are convincing compared to existing methods.", "reasons_to_reject": "* The experimental results shown in section 5 are indirect. It would be better if the author can show some direct result. For example, the test loss after funetuned with and without a subset of examples, or some qualitative results on the detected \"highly influencing\" examples, like Figures 1 & 2 in Koh & Liang. Although the results can be very noisy, they will give readers a better sense on how well the proposed method's approximation actually is.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tJ2c8PrlXH", "reviewer": "Reviewer_fxsH", "summary": "The proposed HyperINF efficiently approximates influence functions using Schulz’s method and GFIM, enabling accurate data attribution for large models with low memory cost and strong empirical results across tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "HyperINF introduces the use of Schulz’s hyperpower method combined with the Generalized Fisher Information Matrix (GFIM) to provide highly accurate and computationally efficient influence function approximation. \n\nAcross diverse and challenging benchmarks—including LLM/VLM fine-tuning and mislabeled data detection\n\n HyperINF shows GPU-accelerated convergence, constant-time memory scaling, and low-rank efficiency", "reasons_to_reject": "GFIM-based low-rank Hessian approximations rely on i.i.d. zero-mean gradient assumptions, which may not always hold in real-world settings", "questions_to_authors": "Can HYPERINF be adapted to settings where only predictions or losses (not gradients) are accessible, such as API-based LLMs?", "extra_scores": {}}, {"review_id": "EaDBnBoY0P", "reviewer": "Reviewer_KW2V", "summary": "This paper utilizes influence functions for estimating importance of training data points. The goal of proposed method (HyperINF) is data selection for finetuning LLMs and VLMs. Experiments include mislabled data detection task with RoBERTa-large, Llama2-7B finetuning on four reasoning datasets, and VLM (CLIP ViT-large, Llama2-7B). On the mislabled data detection, the proposed method significantly outperforms the baselines. On the finetuning data selection tasks (LLM, VLM), the proposed method provides competitive performance.\n\nHyperINF approximates the Hessian using generalized Fisher Information Matrix and uses Schulz's method for matrix inverse approximation. Compared to baselines (LISSA, DataInf), the paper shows strong convergence guarantees at higher dimensions.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method provides strong results on the mislabeled data detection task, and competitive performance on data selection for LLM finetuning (LoRA and dense) and VLM finetuning.\n- HyperINF provides stronger convergence guarantees at higher dimensions compared to baselines.\n- The paper is very well written, and covers a lot of additional results in the Appendix.", "reasons_to_reject": "- The proposed method (HyperINF) performs well but at a significant compute and memory cost compared to some of the baselines (e.g., DataInf). Notably, the performance of DataInf is often comparable to HyperINF. I think the paper can benefit from a discussion about efficiency, and its trade-off with performance.\n- The results for LLM and VLM finetuning were reported using a single 7B model. It's unclear how the proposed method (HyperINF) scales to other model sizes. Since the experiments are based on finetuning dataset, I believe it might be feasible to run experiments with additional model sizes.\n- There is considerable variance in relative performance between HyperINF and baselines across finetuning datasets (Table 3). Additional discussion might be needed. Interestingly, the results are stronger for data selection on VLM data selection.", "questions_to_authors": "Minor comments:\n- Section 5.3 title incorrectly states the data selection for VLM 'pretraining', but the results are reported for selection of instruction-finetuning data. The abstract and introduction accurately reflect the experiments.\n- Paper misses out on a relevent prior work (Choe et al., 2024). I understand the focus of this work is on use of influence functions for LLM finetuning, whereas Choe et al., is more focused on attribution to pretraining data. However, there is considerable overlap in methodology.\n- The paper provides three citations for the same paper (Grosse et al., 2023)\n\nReferences\n- Choe et al., 2024. What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions. https://arxiv.org/abs/2405.13954.", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this paper the authors propose a memory-efficient approach to data attribution by approximating influence functions using a generalized Fisher information matrix and Shell’s method. They evaluate the approach across a variety of practical benchmarks, including data selection for LLM fine-tuning and mislabeled data detection, where the approach performs competitively or out-performs baselines. 

Reviewers noted that the paper is well-written and demonstrated strong results through a comprehensive evaluation, as well as stronger convergence guarantees than other approaches. Reviewers raised concerns about the generalization of the approach to other model scales (experiments limited to 7B LLMs), or in real-world settings where the zero-mean gradient assumption used to justify approximating the Hessian with the GFIM may not be realistic. However, reviewers agreed that the paper was worthy of acceptance despite these limitations.
Decision: Accept