First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence Estimation
Authors: Dmytro Vitel
Anshuman Chhabra
Venue: ICLR 2026 Poster (2026)
Keywords: Influence function
Data valuation
Model Debugging
Detrimental Sample Detection
Primary area: interpretability and explainable AI

Abstract: Identifying how training samples influence/impact Large Language Model (LLM) decision-making is essential for effectively interpreting model decisions and auditing large-scale datasets. Current training sample influence estimation methods (also known as influence functions) undertake this goal by utilizing information flow through the model via its first-order and higher-order gradient terms. However, owing to the large model sizes of today consisting of billions of parameters, these influence computations are often restricted to some subset of model layers to ensure computational feasibility. Prior seminal work by Yeh et al. (2022) in assessing which layers are best suited for computing language data influence concluded that the first (embedding) layers are the most informative for this purpose, using a hypothesis based on influence scores canceling out (i.e., the cancellation effect). In this work, we propose theoretical and empirical evidence demonstrating how the cancellation effect is unreliable, and that middle attention layers are better estimators for influence. Furthermore, we address the broader challenge of aggregating influence scores across layers, and showcase how alternatives to standard averaging (such as ranking and vote-based methods) can lead to significantly improved performance. Finally, we propose better methods for evaluating influence score efficacy in LLMs without undertaking model retraining, and propose a new metric known as the Noise Detection Rate (NDR) that exhibits strong predictive capability compared to the cancellation effect. Through extensive experiments across LLMs of varying types and scales, we concretely determine that the first (layers) are not necessarily better than the last (layers) for LLM influence estimation, contrasting with prior knowledge in the field.

Published as a conference paper at ICLR 2026
First is Not Really Better Than Last: EVALUATING LAYER CHOICE AND AGGREGATION STRATEGIES IN LAN- GUAGE MODEL DATA INFLUENCE ESTIMATION
Dmytro Vitel & Anshuman Chhabra
Bellini College of Artificial Intelligence, Cybersecurity, and Computing, University of South Florida
(dvitel, anshumancc Besshummnne@us..ee.m @usffuu..eeee.
ABSTRACT
Identifying how training samples influence/impact Large Language Model (LLM) decision-making is essential for effectively interpreting model decisions and au- diting large-scale datasets. Current training sample influence estimation methods (also known as influence fumctions) undertake this goal by utilizing information flow through the model via its first-order and higher-order gradient terms. However, owing to the large model sizes of today consisting of billions of parameters, these influence computations are often restricted some subset of model layers to ensure computational feasibility. Prior seminal work by Yeh et al. (2022) in assessing which layers are best suited for computing language data influence concluded that the first (embedding) layers are the most informative for this purpose, using hypothesis based on influence scores canceling out (i.e., the cancellation effect). this work, we propose theoretical and empirical evidence demonstrating how the cancellation effect unreliable, and that middle attention layers are better estima- tors for influence. Furthermore, we address the broader challenge of aggregating influence scores across layers, and showcase how alternatives to standard averag- ing (such as ranking and vote-based methods) can lead significantly improved performance. Finally, we propose better methods for evaluating influence score efficacy in LLMs without undertaking model retraining, and propose a new metric known as the Noise Detection Rate (NDR) that exhibits strong predictive capability compared to the cancellation effect. Through extensive experiments across L.MSs of varying types and scales, concretely determine that the first (layers) are not necessarily better than the last (layers) for LLM influence estimation, contrasting with prior knowledge in the field.
INTRODUCTION
Large Language Models (LLMs) have demonstrated stellar performance tasks across a number
of applications and domains (Street et al.) 2024 Mittelstädt et al. 2024 Marco et al. 2025). Despite
these advancements, current state-of-the-art models still exhibit suboptimal behavior complex
reasoning tasks (Jiang et al. (025). hallucinate responses and facts (Cleti and Jano 2024). and
can make biased and unfair decisions (Gallegos et al.) 2024 Peters and Chin-Yee 2025). To
improve the trust and safety of LLMs, it is imperative to better interpret and understand model decision-maken makinn (Singh et al. 2024).
Recently, data attribution and influence methods (Hammoudeh and Lowd. 2024 have shown great promise in interpreting L.M behavior from the perspective of the training data (Grosse al. 2023 Chhabra et al. 2025). These approaches seek to detect noisy, anomalous, out-of-distribution, or problematic training samples and conceptually link them model output performance (Pleiss et al. 2020; Yang et al. 2024 Jiang et al.) )). Even highly curated datasets can contain detrimental samples, often introduced unknowingly through human error or surreptitiously by malicious adver- saries (Ekambaram et al. 2017). More specifically, influenc functions (Pruthi et al., (2000) Sui et al.) 2021 Kwon et al. 2024) employ gradient- baseed analysis to help developers understand how models' decisions are influenced by training data. Despite their clear benefits, fundamental issue in influence computation is the computational overhead of utilizing the entire gradient space from all the layers of the LLM, which, for modern models, amount to billions of parameters.

---

Published as a conference paper at ICLR 2026
A common workaround (Figure I when employing influence functions in LLMs is to restrict the gradient in- put to only certain layers Yeh et al. 2022 Chhabra et al. 2025) However, there is little consensus on what the best layers are for influence estima- tion, and it is not known whether the output head, attention layers, or word embedding layers are better suited for this purpose. This is partly because very little work has been undertaken to analyze which layers are more use-
training
layer
layer
LORNA
valur
LORA
fine fine-tuning
detrimentaal samples
cuary
Validation set
SCOTES
Figure The influence estimation pipeline for LLMs.
ful for influence analysis. More specifically, prior seminal work by Yeh al. (2022) analyzes layer choice computing language model data influence and finds that the first few (embedding) layers most beneficial for influence estimation. However, these conclusions are derived from a strong assumption about the observed effectiveness of the layers being used, which might not hold in practice, requiring deeper investigation into the effect of layers. Additionally, the work only studies one influence method and smaller-sizer language models such as RoBERTa (Liu al.] 2019), whereas we undertake our study across various LLM parameter sizes and recent models, thereby capturing general patterns of influence estimation as models scale.
discuss subsequently in our paper,
Moreover, influence functions that utilize gradients from multiple layers generally aggregate their contributions through averaging across parameter subsets. However, this averaging can obscure important distinctions by allowing compensatory effects between layers, potentially masking the true contribution of individual training samples. In this work, we also explore and propose alternative strategies for combining influence estimates computed independently across different layers, aiming to preserve layer-specific insights and improve the interpretability of influence attribution.
Finally, assessing new influence-based attribution methods involves measuring how accurately they identify detrimental training samples. In practice, this evaluation is typically performed by removing the least influential samples, fine-tuning the model on the filtered dataset, and then observing the impact downstream task performance. However, this process is computationally expensive, partic- ularly when influence measured across different layers or parameter subsets of model. As result, it becomes valuable identify reliable extrinsic indicator one that be computed efficiently and correlates well with the final task accuracy. Such metric can guide the development and selection of influence-basee based attribution methods without requiring exhaustive fine-tuning experiments. To this end, we also propose novel metric (the Noise Detection Rate) and show how it can be used as reliable proxy indicator of downstream influence estimation performance.
Contributions. In sum, our contributions are as follows:
We evaluate the cancellation effect metric proposed by Yeh et al. (2022) for identifying layers most suitable for influence estimation. Our theoretical analysis and experiments demonstrate that, when filtering large numbers of samples, the cancellation effect is an unreliable indicator.
We quantify the impact of detrimental training samples on language model behavior and identify the model layers most informative for influence estimation. We conduct extensive experiments using noisy versions of the GLUE benchmark in prior influence work (Kwon et al.) 2024
Chhabra et al. 2025) and employ several LLMS such as Llama-3.2, Qwen-2.5, and Mistral 7B.
We also introduce novel strategies for aggregating influence scores across layers, and demonstrate that they frequently outperform the standard averaging method used in current functions.
We propose and evaluate the validity of noise distribution- based proxy measures: Noise Detection Rate (NDR) metric and Area Under the Curve (AUC). as reliable indicators of downstream influence estimation performance, with the goal of minimizing reliance on exhaustive fine-tuning for assessing influence estimation accuracy.
2 RELATED WORKS
Our work falls within the broader area of interpretable machine learning, particularly focusing gradient- methoo estimations of training data importance (Koh and Liang 20077; Yeh et al.) 2018
2019 Jia al. 2019 Pruthi al. 2020 Khanna al. 2018 Sui et al.) 2021 In madieet-basesd
2

---

Published as a conference paper at ICLR 2026
models trained via empirical risk minimization, influence functions offer a practical means to estimate the impact of individual training samples without performing costly leave-one-out retraining. For deep learning. the foundational work (Koh and Liang, 2017) introduced Taylor-series approximation combined with the LiSSA optimization algorithm (Agarwal et al. 2017) to make influence estimation tractable. Subsequent efforts, such as Representer Point (eh et al. (008) and Hydra (Chen et al.) 2021), aimed to improve these estimates, though their focus remained primarily on vision tasks. More recent advancements, including DataInf (Kwon et al.) (202)) Arnoldi iteration (Schioppa et al.) 2022), and Kronecker- factored curvature approximations (Grosse et al. 2023), have scaled influence estimation to large generative language models like LLMs. Simpler methods have also emerged, using raw gradients as proxy for influence (Pruthi et al. [2020] Charpiat et al. 2019), often enhanced with ensemble-based strategies (Bae et al. 2024| Kim et al. 2023). Additionally, self-influence 10m24Kimmm computed solely within the training set - has been shown to be effective for identifying influential
samples (Bejan et al. 2023; Thakkar ct al. 2023). Influence functions have proven valuable across
diverse applications, including classification (Koh and Liang
2017: Chhabra et al. 2025 2024).
generative modeling (Koon et al. [024]] Schioppa et al.
2022
Grosse et al. 2023). active learning
(Liu et al. 202T), in-context learning (Nguyen and Wong 2023
S.et al. 2024 Askari al.
2025a
and layer quality estimation (Askari et al. 2025b).
Gradient compensation in feature-attributed gradients has been studied previously (Liu al. 2020
Kapishnikov et al.] (202T) In data attribution, influence functions have been used to detect spurious training artifacts and improve performance (Han al. [2020; Pezeshkpour al. 2022 Askari et al. 2025a). Prior work (Hammoudeh and Lowd 2024| Ych et al. (222)) highlights that dot- dot-product-bast methods and high gradient cancellation may reduce the discriminative power of influence scores.
We extend this line of work by empirically and theoretically investigating whether such cancellation impairs the detection of mislabeled data.
It is also interesting to note that prior work has studied the applicability of influence functions in LLMs, with varying results. Clearly, while influence functions have been used successfully across several L.M tasks and applications Wang et al. (2023); /Zhang et al. (2025) y Askari et al. (2025)); Xia et al. (024), recent work Li et al. (025)) finds that Hessian-Free, Datainf, and LiSSA perform poorly on tasks such as harmful data detection, class attribution, and backdoor identification, due to iHVP approximation errors, uncertain convergence, and weak correlation between parameter changes and model behavior. Their experiments use default influence-functioo aggregation across all layers. In contrast, we extend these insights by identifying specific layers and aggregation strategies that achieve performance capable of exceeding the default settings in Li eet al. 2025). As part of future work, we believe indetifying optimal layers for influence analysis can significantly improve the efficacy and fidelity of influence functions LLMs, as observable in our main paper results.
implementations with mean
Moreover, given our focus on where influence scores are strongest and how individual valuations can be effectively combined, we relate our analysis to knowledge-editing methods that similarly perform localization and aggregation. ROME (Meng et al., 2022) and R-ROME (Gupta et al.) (024a) use causal fracing to identify the MLP layer encoding fact and apply a rank-one edit. MEMIT (Meng) let al., 2023) generalizes this via per-layer causal-effect scores and top-k distributed edits, while EMMET (Gupta et al.) 2024b) refines aggregation to reduce redundancy. In contrast, we use gradient- based influence to locate layers most affected by beneficial or harmful training samples and, consistent with KE studies, find the strongest discriminative signals in middle layers.
3 PRELIMINARIES AND NOTATION
3.1 INFLUENCE FUNCTIONS
Given a training sample ā E X, model weights θ, and a validation sample x' E X', the influence is defined as follows for the utility function f.
1(2, xe,x''0
Σ Vvf(t( FeX"
He (
(I)
X
The optimization step A0, in the direction of the vector given by the term under the parentheses decreases f on X'. The second term, H VViã, θ) (H Hessian matrix) defines the optimization step AQ, that would minimize the loss on the sample T.
3

---

Published as a conference paper at ICLR 2026
The loss optimization some training samples could be misaligned with the optimization of Therefore, fitting these samples would increase f. We refer to such samples as detrimental for the utility under consideration. This definition corresponds to the previously used definition in (Koh and Liang, 2017 Kwon et al. 2024, Chhabra et al. 2025). In practice, it is not feasible to compute on large models due to the time complexity of the Hessian inversion (cubic time in the exact case). Therefore, influence is estimated approximately. The TracIn method (Pruthi et al. 2020) replaces the Hessian vector product H. H1 Vi(ā, @) with the gradient Vi +). Cosine similarity between the utiliiyy term and H. = Vi ④) serves as another first-order influence estimation approximation. DataInf (Koon al. 2024) a recently proposed second-orde method based on swapping the order of matrix inversion and averaging across training samples in H. Its enror decreases when @ is small, which makes this method suitable for fine-tuning with LoRA (Hu et al. 2022 Dettmers et al. (023).
For crossyy loss, the Hessian is the block diagonal matrix of gradient second moments IXI-'E Vi(a, ()V(" (z, θ), one block for one layer L in the neural network. Thus, is an aggregation of influence values across validation samples and model layers, defined as follows:
1 +
Ff,,F', (1) s. I (3, ,0 Oi) = Ve, f (æ', (.)H(( F'EX'ELL
Σ
(2)
X
l' F', θi) are tensors collected on different layers and for different influence functions. We employ TracIn, Cosine, and DataInf in experiments (Vl, Ve, (1, Oi)), analytically defined as follows:
TracIu(2, x', Ot) v/ Vl2.
(3)
Cosine(7,5',0)) = VCH. Vl2
(4)
DataInf (2, - EEEX (XXIIIA) E
SEX VIVVV
(5)
V42
To address the question of which layers are more effective at detecting detrimental samples, we also consider two influence functions proposed by Yeh et al. (2022). Both of them, TracInwe and TracIn 10 wer operate on word embeddings. The first, TracInwes considers only the weights of the tokens (including special tokens) in both the training and the validation samples:
al(F', 0.) a( 01)
TracIn (F, Owe)=
Σ
(6)
00,
00,
tERNE' w
The second, TracIn 10 we additionally filters out the weights E 1,,00 0 Owe keeping only the top-10 10 that have gighest-by-magninee gradients for training sample F.
3.2 CANCELLATION EFFECT
In their study of layer choice effects in influence estimation, Ych et al. (2022) consider the compen- sation of gradients as contributed by training samples. They argue that if the sum of gradients w.r.t a particular weight 0 becomes very small, then the optimizer would not change 0 significantly because of them. However, the magnitudes of the summed training gradients could be very large and hence, the 0-contributed influence score could also be large. The authors further state that 0, the weight with a high cancellation effect, should be ignored in influence estimation because it reduces the model's 's dis- crimination power. To this end, they propose metric C' to estimate the cancellation effect layer W and present the results where smaller C' values lead to better model confidence (Ych et al. 2022):
Σn Vw((z, W)I
C(W)-
AW]
C' can be computed throughout the fine-tuning procedure, considering every checkpoint individually. Accumulating the magnitudes of atomic gradient updates per sample requires size-one batching during training, which is computationally expensive and infeasible for current models. We thus adopt the original formulation estimate the cancellation single checkpoint in evaluation mode.
4

---

Published as a conference paper at ICLR 2026
3.3 PROPOSED INFLUENCE AGGREGATION APPROACHES
As is evident, Eq. averagess influence scores across all layers and validation samples. This can cause influence compensation, where opposing contributions cancel out, or dominance, where large scores overshadow others. In our work, we aim to investigate alternative aggregation strategies that can be employed instead to mitigate these effects. Generally, influence attribution can be abstracted as:
[ X', θτ) Ax (1'(, (.))
(8)
where A is an aggregation operator (e.g.. taking the average across samples/layers) We thus propose novel aggregation approaches as follows:
Ranking. Every validation sample and layer ranks training samples according to their influence. These ranks are summed, eliminating the domination of individual influences with high magnitude. Additionally, our proposed Rank method ignores incorrectly predicted validation samples by the selected checkpoint:
Rank(T') Σ EIC < (,,,''
(9)
D'EX",IELDEN
where is the indicator function, and X" denotes the correctly predicted validation samples.
Positional voting. To avoid the domination of very low very high ranks in averaging, we propose method based on voting. Every validation sample and layer assigns i votes to the least influential training datum, k - 1 to the next one, etc., until the number of votes reaches zero. We then pick k equal to the number of training samples for filtering as potentially detrimental. As with Rank, Vote method considers only correctly predicted validation samples by the selected checkpoint:
Vote. (o'e ( =
E I'X
max Σ ((1 (,))) < "'','','',0) ,0,))).))..
(10)
ØEX
4 RESEARCH QUESTIONS
We now formalize the research questions (RQs) we seek to study in this paper. To bridge the gaps identified above regarding the role of layers in LLM influence analysis, the impact of layer-wist aggregation strategies in influence functions, and whether influence functions can be evaluation through novel external metries, without requiring costly fine-tuning, propose the following RQs:
Can the gradient cancellation effect serve as a reliable predictor of layer contribution in influ- ence estimation? ? In Section [.1 we propose both theoretical and empirical evidence to answer RQ1, showing that high cancellation within a layer does not necessarily imply poor attribution performance.
[QQ1].
[RQ2]. Which model layers yield the most effective influence scores for detecting detrimental samples? ? Addressing this question (Section 5.2) serves two practical goals: (1) improving training data filtering methods, and (2) revealing which parts of the model These efforts are motivated by the lack of general consensus on layer choice in influence estimation.
sensitive such samples.
[RQ3]. Can alternative aggregation strategies improve influence estimation performance compared to traditional averaging? Moving beyond influence score averaging across layers, we introduce two novel aggregation methods (Section 3.3): Rank and Vote, for combining individual influence
scores. In Section 5.3 we evaluate their impact on model accuracy following sample filtering.
[RQ4]. How reliably can the detrimental sample distribution measures predict the downstream performance of influence-based data filtering methods? In Section 5.44 we study whether our novel proposed metrics: Noise Detection Rate (NDR) and Area Under the Curve (AUC), can serve useful metrics for assessing the performance of influence functions, without requiring expensive fine-tuning.
5 ANALYSIS AND EXPERIMENTAL RESULTS
Datasets. We evaluate detrimental sample filtering across eight GLUE datasets (Wang et al. 2018): QNLI, MRPC, QQP, SST-2, CoLAA, MNLI, RTE, and STS-B, These benchmarks cover a range of
5

---

Published as a conference paper at ICLR 2026
natural language understanding tasks, including sentence similarity, paraphrase detection, sentiment analysis, linguistic acceptabilitty, and natural language inference. Together, they provide a diverse and widely used testbed for assessing model performance and the impact of data quality.
Models. The experiments are conducted on several LLM types: RoBERTa-Large (Liu et al. 2019). LLaMA-3.2 1B (Touvron et al. 2023), Qwen-2.5 1.5B (Qwen et al. 2025), and Mistral 7B (Jiang et al. 2023). These models represent various widely used transformer based language models ranging from millions to several billions of parameters, offering a comprehensive view of influence behavior across model sizes and architectures.
Methodology and Protocol. Similar prior work (Kwon et al.
2024 Chhabra al.
2025 we
employ the five-stage pipeline detailed further in Appendix A (I) Inject synthetic noise into training data and initialize from compressed embeddings. (2) Fine-tune on noisy data, selecting checkpoints with the lowest validation loss (Pruthi et al. 2020), which we verify outperforms final-epoch or accuracy- baarac selection. (3) Compute influence values across all tunable layers. (4) Partition the model into embeddings (WE), attention groups (four splits), and classífication head (CL), aggregating influence per training sample. (5) Remove 30% least influential samples and retrain, evaluating effectiveness by the best test accuracy. Each configuration is repeated over 10 seeds for robustness.
Furthermore, to robustly compare configurations, we compare them pairwise across multiple runs. Configuration outperforms B if it performs better for the same dataset, checkpoint, and random seed. We also report the counts of configuration A being better more times than some predefined ranking threshold (e.g. 75%) compared to other configurations. This results in a score matrix showing which configurations consistently outperform others. From this matrix, we identify Pareto fronts and assess which influence methods perform better.
5.1 [RQ1] VERIFICATION OF ASSUMPTIONS ABOUT CANCELLATION EFFECT
The cancellation effect arises when opposing gradient con- tributions cause small net parameter updates (AW) (Eq. 7 while the cancellation metric C(W) remains high. Intu- itively, the key issue arising in Eq. 7 is that C not additive across parameter subsets. For example, although many gra- dients pass through the classification head (CL), frequent- token embeddings in WE also accumulate large but opposing updates. Since C' aggregates changes via norms, such cancel-
Table I: Training gradient cancella-
tions for various LLMs (1 no can- cellation, 0o -> extreme cancellation).
Layer
Median
WE
22
oc
40.3
00-05
11.
10"
0.1
Mistral 7B Lienn-3.2 Robettd
06-11
105
14.3
12-17
94
12.5
ou
0.1
18:23
85
IL1
1.5
10"
0.2
lations are obscured, limiting its reliability as an indicator.
CL
11.1
L66
0.1
WE
Table D presents the cancellation effect measured across LLMs, datasets, and seeds. The embeddings WE indeed have the smallest values. The median value of C' reflects the cancellation of an individual parameter, predominantly observed within a corresponding layer group. Value C = denotes the smallest cancellation when all the gradients are codirectional (e.g. majority of WE parameters). However, there WE. weights with infinite cancellation scores that have or more collected gradients from training samples. Clearly, the results confirm that layer group cancellation C "smoothes" the gradient compensation over large number of parameters and hides those weights where the individual parameter gradient cancellation is high. Yeh et al. (202)) showed that influence estimates from layers with lower C' better capture expected prediction changes for ground-truth classes when top-k opponents are filtered out for given sample. In contrast, we extend this analysis to the validation set as a whole, filtering opponents across many samples and measuring downstream performance via task accuracy. Results demonstrate only weak no correlation between C' and performance (Table [ column p: Spearman correlation).
04-07
00-43
8.4
11.8
7.7
1.7
1.2
10
DO
-0.1
-0.2
08:11
12-15
40.17
5.3
10"
4.1
CL
1.1
2.5
10$
WE
1.5
1.1
00-17
17.7
1:35
17
1.6
0.0
08-15
16.4
18.6
2.2
16-23
15.6
24-31
15.7 1
15.5
1.8
00
CL
20.5 191
11.7
3.8
10*
0.1
Next, we provide theoretical evidence that reinforces the aforementioned issues with the cancellation effect and demonstrate how it is an unreliable estimator of influence performance (see Appendix G for the proof).
Theorem 5.1 (Cancellation Can Improve Influence Estimation.). Let X be training set. Consider:
1. samples 61,52 E X, where 3n is noisy and Ti is clean;

---

Published as a conference paper at ICLR 2026
2. parameter vectors and such that their cancellation scores satisfy C(0) C(w) with
((w) + oo for {ō1, Ø2);
3. the TracIn influence scores Lo (based on 0 alone) and سي (based jointly on 8,w);
4. influence score distance between テ and T2 w.r.t. weights è and validation samples X':
( X') [7 X', ) [ x', 0)]-
Then there exists a validation point 73 such that: New (T1, T2 ka) > (01, T2 Tz);
i.c the separation between noisy and clean samples is strictly larger under law, showing that contrary the claim of Yeh et 2022), the inclusion of high cancellation weights can improve influence.
Remarks on RQI Findings. Our empirical results demonstrate an absent-to-went teekk correlation between the cancellation effect and the accuracy of models retrained on datasets filtered by corre- sponding layers. Further, our theoretical analysis provides a counterexample. demonstrating that weights with high cancellation can, in fact, improve influence estimation.
5.2 [RQ2] LAYER EFFECTIVENESS IN INFLUENCE FUNCTION ANALYSIS
Next, owing to the limitations of the cancellation effect as previously discussed, we now seek to analyze which layers are better suited for influence estimation in language models. We compare layers based on how well they can detect detrimental samples in accordance with the framework outlined at the beginning of the section. In general, our results are only aligned with (Yeh et al.) 2022) regarding the classification head CT performance. We observe that CL highly susceptible to influential noise across models and methods. Dataset filtering with scores from CL improves downstream accuracy less than other layers (Figure
At the same time, WE is not the best choice of layer for every case scenario. While TracIn applied on the smaller model RoBERTa-Large demonstrates best accuracies when WE is used, in general, the first second attention layer groups are better options- both in terms of performance and computational complexity, when larger models are used. Methods TracIn and TracIn improves upon Tract with WE, but requires more clock time in practice (Section J. For other influence functions, we also observe the best performance the first and second attention layer groups. Appendix contains details on the layer ranking per model.
Figure 2 demonstrates the best test set accuracy variation (mean and Q1-Q3 quartile range) after filtering with influence scores for Mistral 7B. Configurations have statistically significant differences. The best layers outperform the worst ones (generally, CL for most tasks and methods) by 10-15%. Similar observations for other models are in provided in Appendix B due to space constraints.
QNLI
MRPC
SST2
QQP
90-
50
75-
70-
COLA
MNLJ
RTE
STSB
75-
70-
70
Datalin
Tracht
Cooiee
Danatf
Trachn
Cosine
Dotaln
Trxcho
Cosine
Daninf
Tracin
Cosine
TracInw
TracIn.0 10
WE
00-07
08
16.23
24.31
CL.
Figure 2: Best test accuracy of Mistral 7B after 30% filtering (averaged over runs). Early and middle attention layers yield the strongest influence method performance on most tasks. Differences are statistically significant (Friedman p-value: 1e-142).
RoBERTa-Large For this comparatively smaller model, filtering performance peaks in later attention layers. Datalnf achieves the strongest result on group (layers 18-23) with a 0.70 win rate, outperforming other configurations. Cosine on group (layers 12-17, 0.56 win rate) and TracIn
7

---

Published as a conference paper at ICLR 2026
the embedding layer (0.51) are the closest competitors. The two variants TracIn we and TracIn 10 yield nearly identical scores (0.47) but are consistently dominated by TracIn on WE. Interestingly, COLA accuracy trends (see Figure 9 surpassed by influence-based filtering that retains a fraction of noisy samples.
Appendix B reveal that the oracle setting Full 1 is occasionally
Llama-3.2 1B. For Llama, influence functions perform worst among all settings: none of the configurations surpass the uniform removal baseline (Random), even when a large fraction of mislabeled samples discarded. The strongest outcomes are obtained with DataInf on group (layers 04-07, 0.64 win rate) and group (layers 00-03, 0.58), comparable to Random (0.64). Accuracy trajectories (Figure [O] Appendix B) further show that Random outperforms influence methods on QQP, COLA, and STSB in later epochs. Despite this, noise distribution analysis (Figure 2T Appendix D reveals non-uniform patterns, with mislabeled samples concentrating at both extremes of the influence range, describing substantial fraction of detrimental samples as influential for the task goal. We provide additional discussion on Llama results in Appendix (M]
Qwen-2.5 1.5B. The strongest results are achieved by Cosine on group 2 (layers 07-13) and by DataInf on both groups I (layers 00-06) and 2, all ranked top- 1. TracIn follows as the runner-up, with its best performance group. Accuracy trends (Figure 8 Appendix B further show that middle layers (07-13) consistently outperform other layers across GLUE tasks and methods.
group 2, while TracIt produces comparable outcomes on the same
Mistral 7B. Top-1 configurations constitute DataInf and TracIn. DataInf achieves the strongest results on layers 08-15 (0.71 win rate), 00-07 (0.69), and 16-23 23 (0.51). TracIn performs best on layers 00-07, closely matched by TracIn 10 which improves over TracIn on WE. Cosine on layers 00-07 and 08-15 ranks as runner-up. Accuracy trends (Figure 12, Appendix B reveal task-specific preferences: Cosine (08-15) excels on SST2 & STSB, Datalnf (08-15) on QNLI & RTE, and TracIn on COLA.
Remarks on RQ2 Findings. Across models and methods, we observe a consistent pattern in layer- level performance. With mean-score aggregation, the classification head (CL) shows limited ability to discriminate noise across all influence methods. DataInf and Cosine frequently outperform TracIn, particularly on early-middle attention layers of deeper models, suggesting that these layers encode representations that are most informative for noise filtering. Comprehensive rankings across all configurations are further provided in Appendix (Tables 4 5 6). Additionally, we verify and reconfirm our findings on autoregressive datasets in Appendix
5.3
[RQ3] EXPLORING BETTER STRATEGIES TO COMBINE INFLUENCE SCORES
Next, we aim to study whether our proposed aggregation strategies (Section 3.3 can result in better influence function performance. We first analyze the correlation of training-sample influence scores across layers and methods (shown in Appendix D Figure I3)
revealing three layer groups: early, middle, and late. Within a group. layers assign similar scores: for example, TracIn variations on WE consistently identify the same noisy samples, and late layers show strong agreement across all methods. Correlations between groups are weak, indicating that layers often compensate for one another. Across methods, Datalnf closely aligns with TracIn on later layers, while Cosine shows low agreement, providing a complementary view of importance attribution. These patterns motivate the underlying ideas behind our proposed aggregation strategies, Rank and Vote, that reconcile layer- and method-level discrepancies. Figure 3 fillusttrate the performance differences relative to mean aggregation for Mistral 7B (results for other models show similar trends and are provided in Appendix H).
Voting substantially improves filtering for TracIn and DataInf on Mistral, with no statistically signif- icant losses across datasets, though CoLA remains the most challenging benchmark. Comparable gains are observed for TracIn on Llama and RoBERTa (Appendix HH), with occasional drops on CoLAA, QQP. Moreover, performance improvement is observed for NLI tasks, SST-2, and STS-B. For DataInf on Llama, Rank slightly outperforms Vote, whereas in most other settings Vote remains superior.
Utilizing influence functions with Vote on Mistral leads to significant improvements and promotes outlier confi gurations (layer- inffuene method combinations) to higher positions in the global ranking (Appendix F T)) For instance, TracIn CL and TracIn !0 now share the top rank and thus become non-dominant. On RoBERTa, the rank of TracIn CL improves from 10 (Mean) to (Vote). On Mistral, the TracIn CL layer rank rises from 12 (Mean) (Vote).
8

---

Published as a conference paper at ICLR 2026
QNLL
UUF
Teals
10220
Coass
Trahs
Figure 3: Mistral 7B: Accuracy improvements (% change) of Rank and Vote relative to mean aggregation, 10 runs per layer group. Voting consistently improves accuracy for Datalnf and TracIn across most datasets and layers, while degrading performance for Cosine. Statistically significant differences are shown opaque colors (Wilcoxon test, p < 0.1 and mean change > 1%).
TracIn
Cosine
DataInf
Value B
60
00
Layer
20
Layer
20
Layer
Mixe 8
100
50
100)
Mote
100
Figure 4: Filtered detrimental samples with LoRA Value module for Qwen-2.5 1.5B as function of the number of votes &s (positional voting, Equation 10) and the selected model attention layer. The heatmaps on the axes show where performance is maximized.
The positional voting scheme introduces the hyperparameter number of votes each validation sample assigns training samples. We also study how aggregation performance varies as a function of k. Figure shows the percentage of filtered detrimental samples across different values of & and model layers for Qwen-2.5 1.5B with LoRA Value B module, which provides the strongest overall performance. For both DataInf and TracIn, performance is maximized for k [10, 50). In contrast, the Cosine-hased method attains local minimum around k 50 and reaches its maximum at larger k. Similar trends are observed for other LoRA modules (see Appendix [ for more details).
Equation 10 representing the
Remarks on RQ3 Findings. Our results demonstrate that alternative aggregation strategies can significantly improve the effectiveness of influence estimation. In particular, Vote consistently outperforms mean aggregation across multiple datasets and models, with notable gains such as an increase in the win rate of DataInf (layers 00-07) 0.84. These findings establish that aggregation is critical factor in influence analysis. While Vote offers simple and effective alternative, future work could explore efficient unsupervised or mujti-ooeective aggregation methods A exploit the influence signals and further enhance filtering performance.
5.4 [RQ4] IDENTIPYING MEASURES TO ESTIMATE DOWNSTREAM PERFORMANCE
We now investigate whether utilize external metric proxy to evaluate influence function methods, thereby foregoing expensive retraining/fine tunnin based evaluation. this end, we propose two metrics. The Noise Detection Rate (NDR), quantifies the proportion of mislabeled samples

---

Published as a conference paper at ICLR 2026
Datalní
Tracnn
Cosine
Layer-wise NDR.
1LT WE
15 Layer
15
Tayen
Value
methods
Figure 5: Layer-wise NDR (%) across layers of Mistral 7B.
ranked among the top-k% least inftuential, i.e., the fraction of noise filtered prior to final fine-tuning. In contrast, the Area Under the Curve (AUC) metric captures the overall skew of noise across the influence ranking, indicating whether mislabeled samples concentrate at the top of the list or are more uniformly spread. We validate the hypothesis that NDR and AUC can serve reliable proxy metries for influence estimation by measuring the metric correlations to the actual ground truth performance observed after fine-tuning on filtered datasets. Table 2 reports the correlation of NDR and AUC with the actual ground-truth downstream performance. Overall, we observe moderate-to-stee statistically significant positive correlations. NDR at 30% generally exhibits higher values than AUC. However, values in the 0.5-0.7 range indicate that these metrics not consistently reliable across all configurations. Vote aggregation has stronger correlations of 0.9, suggesting that combining influence scores appropriately can enhance proxy predictiveness.
Figure 5 further illustrates the proportion of noise filtered by each layer of Mistral 7B. Across models and influence functions, the Value B LoRA modules consistently capture the largest fraction of noisy samples, with peak filtering ob- served in the middle attention layers for larger models and later layers for smaller models (see Appendix E for additional results). Additional sample-influence ranking variation analysis in Appendix K indicates that average noise influ- ence attains its minimum at layers with the high- est noise-detection efficiency.
Table 2: Spearman correlation (p) of training gra- dient cancellation (C), noise detection rate (NDR), and NDR-AUC with final task accuracy after filter- ing, across influence aggregation methods. High correlations are shown in bold and denotes non- statistically significant results,
Inf.dame
Rubasted ang
NDR
AUC
Llameck 18
NDR
ADC
Misral
NDR
AUC
Danter
0.2
07
u.5
0.5
211
3.5
Truelngs
0.3
066
0.2*
n.6
0.5
0.14
0.3
Traclah
0,3
0.6
0.0*
0.5
0.5
auge
0.5
Tracln
0.0*
0.4
0.3
0,1"
0.6
0.5
0.0
95
Cosine
022
0.7
06
0."
0.5
0.5
0.1
0.6
Danlar
0.3
057
1.3
-01
0.66
102
0.5
Traenng
0.3
0.6
115
-0.1"
0.2
0.2*
Du0e
0.2
Traclnh
0,3
0.5
01.
0.2"
0.1
0.0*
0.1*
0.2*
0.1*
Tractn
0.2
0.5
0..
0.1"
0.5
0.5
0.1*
9.5
Remarks on RQ4 Findings. The cancellation effect shows little correlation with downstream performance and fails to reliably indicate influential layers or weights. In contrast, NDR serves to be useful proxy, particularly under Vote aggregation, though correlations are generally moderate-to-hipe across layers, tasks, and models. The proposed AUC metric exhibits similar trends. Together, these results suggest that proxy metrics incorporating both the quantity and quality of filtered anomalics could offer more reliable indicators for influence function evaluation, highlighting promising direction for future research.
Cosine
0.2
0.7
0.7
0.1%
0.6
0,8
0.1
0,6
0.6
Dandar
0.3
0.8
1L3
2011
0.6
0.5
Tieen
-0.4
-0.1"
0.5
0.5
-0.1
Tracln
0.4
0.7
0.5
0.11
0.5
0.5
0.0*
0,5
Tracln
0.2
0.7
0.1
0.6
0.6
9.1
0,5
Cosine
0.2
0.7
0.6
0.1
0.2
0.2
0.09
CONCLUSION
Our paper conducts multi-faceted analysis to better understand how influence functions can be effectively leveraged for data-centric learning large language models. Firstly, our work challenges prior assumptions in past work about the most informative layers for influence estimation in LLMs and demonstrates that middle attention layers often provide more reliable signals than the first embedding layers. Secondly, we introduce improved layer-wise influence score aggregation strategies that move beyond the standard averaging approach, and attain significant performance improvements in influence analysis. Finally, propose external evaluation metrics, such as the Noise Detection Rate (NDR), to offer a more robust framework for evaluating and interpreting influence scores without undertaking costly retraining for influence function evaluation. These findings advance the understanding of training data influence large language models and provide practical tools for more accurate and efficient model auditing.
10

---

Published as a conference paper at ICLR 2026
7 REPRODUCIBILITY STATEMENT
We have open-sourced our code and implementation for our experiments: https:/ggihub..om/diieee infl, To support reproducibility, multiple runs were executed with fixed random seeds specified the accompanying shell scripts, and determinism was enabled in the underlying pytorch framework. All experiments were conducted on an Ubuntu server with 8x NVIDIA A A100s (40GB VRAM/GPU).
REFERENCES
Chih-Kuse Ych, Ankur Taly, Mukund Sundararajan, Frederick Liu, and Pradeep Ravikumar. First is better than last for language data influence. In Proceedings of the 36th International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 0181713871083
Winnie Street, John Oliver Siy, Geoff Keeling, Adrien Baranes, Benjamin Barnett, Michael McKibben, Tatenda Kanyere, Alison Lentz, Blaise Aguera Arcas, and Robin I. M. Dunbar. Llms achieve adult human performance on higher-order theory of mind tasks, 2024. URL httts:://ax.ttt org/ 20tg/bb/240
18870
Justin M. Mittelstädt, Julia Maier, Panja Goerke, Frank Zinn, and Michael Hermes. Large language models can outperform humans in social situational judgments. Scientific Reports, 14(1):27449
Nov 2024. ISSN 2045-2322. doi: 10.1038/s41598- 00008/441598-02--990009 10. 1 038 4 024- 79048-0)
URL
Guillermo Marco, Luz Rello, and Julio Gonzalo. Small language models can outperform humans in short creative writing: study comparing slms with humans and Ilms, 2025. URL https //arxiv. wwg. abs 2409. 11547
Botian Jiang, Lei Li, Xiaonan Li, Zhaowei Li, Xiachong Feng. Lingpeng Kong, Qi Liu, and Xipeng Qiu. Understanding the role of LLMs in multimodal evaluation benchmarks, 2025. URL https
//openreview net fww.fforumooooo
Meade Cleti and Pete Jano. Hallucinations in Ilms: Types, causes, and approaches for enhanced
reliability, Oct 2024. URL osf. iotttj3t//jjvv
Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Der- noncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. Bias and fairness in large language models: A survey. Computational Linguisticy, 109-11199- 2024. ISSN 0891-2017. doi:
10.1162/coli URL/https://dbi.cooo httttp:::/abi.oor//110111.1. 1112/co611cl111 00524
Uwe Peters and Benjamin Chin-Yee. Generalization bias in large language model summarization of scientific research, 2025. URL http:/tss//aarrv..rrr/aaa///////
00025
Chandan Singh, Jeevana Priya Inala, Michel Galley, Rich Caruana, and Jianfeng Gao. Rethinking interpretabiliiyy in the era of large language models. ArXiv, abs/2402.01761. 2024. URI https
7/api www.mmeattt.... ntt cscho aar www.oo////oooo.. Ctrr//rrpussti TD: 260412574222
Zayd Hammoudeh and Daniel Lowd. Training data influence analysis and estimation: a sur- vey. Machine Learning, 1136):2351-2400; May 2024. ISSN 1573-0565. doi: 10.1007/ 1007/ ott.:///..........00000--------.
s10994-003-00449-77 |tt.://pes//aaaoooo.oo000//00000
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilé Lukošiūté, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions, 2023. httpttpsss/aaii..r.r..
abs/2308.030 03296
Anshuman Chhabra, Bo Li. Jian Chen. Prasant Mohapatra, and Hongfu Liu. Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Models. ICML, 2025.
11

---

Published as a conference paper at ICLR 2026
Geoff Pleiss,
Tianyi Zhang, Ethan Elenberg, and Kilian Q. Weinberger. Identifying mislabeled data using the area under the margin ranking. In Proceedings of the 34th International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Jingkang Yang. Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distriibttion de- tection: A survey. International Journal of Computer Vision, 132(12):56355-666,, Dec 2024. ISSN 1405- doi: 10. s 611263-024-02117
URL URt.ttpss//0o...o2//00..000
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael Mozer. Characterizing structural regularities of labeled data in overparameterized models, 2021. URL https:://axa....ottt
abs/ 200//0022..0002
Rajmadhan Ekambaram, Dmitry B. Goldgof, and Lawrence O. Hall. Finding label noise examples in large scale datasets. In 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 1420-24244 2017. doi: 10.1 1011100//..20117822299000
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. In Proceedings of the 34th International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Yi Sui, Ga Wu, and Scott Sanner. Representer point selection via local jacobian expansion for post-hoc classifier explanation of deep neural networks and ensemble models. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 23347-23358. Curran Asso- ciates, Inc., 2021. URL http:/ttppp//pprrceeing.g...se reur http://srrrrrrrooo//////ee/eeeee
Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in loRA-tunee LLMs and diffusion models. In The Tweifth International Conference on Learning Representations, 2024. URL httLttttt////peneeeeee
net. wwtp::eeeeeeeemioeeeooee mett:
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyamov. Roberta: A robustly optimized bert pretraining approach. 2019. URL /tttp::/aaxxiv.org///a///
11692
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, JMLR.org, 2017.
page
Chih-Kuan Yeh, Joon Sik Kim, Ian E.H. Yen, and Pradeep Ravikumar. Representer point selection for explaining deep neural networks. In Proceedings of the 32nd International Conference Neural Information Processing Systems, page 9311 1-9221, Red Hook, NY, USA, 2018. Curran Associates Inc.
Chih-Kuan Yeh, Cheng-Yi Yu Hsieh, Arun Sai Suggala, David I. Inouye, and Pradeep Ravikumar, On the (inifidelity and sensitivity of explanations. Curran Associates Inc., Red Hook, NY, USA, 2019.
Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J. Spanos. Towards efficient data valuation based on the shapley value. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of the Eivety-Seconde International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 1167-1176. PMLR, 16-18 Apr 2019. URL www.:smml...rrrrr htw.:eeeeiinn..l....pppres////ss
html
Rajiv Khanna, Been Kim, Joydeep Ghosh, and Oluwasanmi Koyejo. Interpreting black box predic- tions using fisher kernels. In International Conference Artificial Intelligence and Statistics, 2018. URL https::: / api. s semant inwaanittstoee wwwphho/la...rrooooooosos.o.e :trp::uusD:::00::.. 53085397
Naman Agarwal, Brian Bullins, and Elad Hazan. Second-order stochastic optimization for machine learning in linear time. J. Mach. Learn. Res,, 18(1):414844188 January 2017. ISSN 1532-4435.
12

---

Published as a conference paper at ICLR 2026
Yuanyuan Chen, Boyang Albert Li, Han Yu, Pengcheng Wu, and Chunyan Miao. Hydra: Hyper- gradient data relevance analysis for interpreting deep neural networks. In AAAI Conference on Artificial Intelligence, 2021. URL https: www.ssseeaannttioeeeeeee
231 802386
Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up influence functions. Proceedings of the AAAI Conference on Artificial Intelligence, 36(8):8177-88886 Jun. 2022. doi: 10.1609/aaii atticle/view/20075
URttttts:://ojs aaai. org.i..og//inee wtwpg////iee..oeee ottp:///..jj..aaaa.aoooorarraeep pht./AeI
Guillaume Charpiat, Nicolas Girard, Loris Felardos, and Yuliya Tarabalka. Input similarity from the neural network perspective. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL https: httpt:pp:pppp.cciiiig.....as//s/ htwp::eeddiinng....rue. neurips ccppppe.cpppppppeppp fileffie/..ee Les/
pdÉ.
Juhan Bae, Wu Lin, Jonathan Lorraine, and Roger Baker Grosse. Training data attribution via approximate unrolling. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL httpttt::///peeerriiiiiie
www/teetooorroeeo d= 3NaqGg92KZ
Sung Yub Kim, Kyungsu Kim, and Eunho Yang. Gex: A flexible method for ap- proximating influence via geometric ensemble. In A. Oh, T. Naumann, A. Glober- son, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Infor- mation Processing Systems, volume 36, pages 5888-5911. Curran Associates, Inc., 2023. URL httpss//pprocceeinggs http://oeeeeeeeeeeeeeeeeee..o//. wwtppprr.cppppppppppppppp
Paper-
Irina Bejan, Artem Sokolov, and Katja Filippova. Make every example count: On the stability and utility of self-influence for learning from noisy NLP datasets. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 10107-10121 Singapore, December 2023. Associa- tion for Computational Linguistics. doi: 10. 00.18853/v1//000 120p2/2...mmmmmmam.222222..
URL https
//acl ogy. org. 2023. emn p ma i n. 6257
Megh Thakkar, Tolga Bolukbasi, Sriram Ganapathy, Shikhar Vashishth, Sarath Chandar, and Partha Talukdar. Self-influence guided data reweighting for language model pre-training. In Houda Bouamor, Juan Pino. and Kalika Bali, editors, Proceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, pages 9033-2044) Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10118865/1/2223. http:/222....mmmmam..... emnlp-main.125. URL
httpp:://sclanthology org. 2023. emnlp ma in. 125/
Anshuman Chhabra, Peizhao Li, Prasant Mohapatra, and Hongfu Liu. What Data Benefits My Classifier? Enhancing Model Performance and Interpretability through Influence-Batt Data Selection. In /CLR, 2024.
Zhuoming Liu, Hao Ding, Huaping Zhong, Weijia Li, Jifeng Dai, and Conghui He. Influence selection for active learning. In 202/ IEEE/CVF International Conference on Computer Vision (ICCV), pages 9254-9263, 2021. doi: 10. 199109999449992222001 1.0914.
Tai Nguyen and Eric Wong. In-context example selection with influences, 2023. URL https /arxiv.erg/ab///322.11111
Vinay M. S., Minh-Hao Van, and Xintao Wu. In-context learning demonstration selection via influence analysis, 2024. URL htt.ttss//aaraii.rr//as///00.
Hadi Askari, Shivanshu Gupta, Terry Tong, Fei Wang, Anshuman Chhabra, and Muhao Chen.
Unraveling indirect in-context learning using influence functions, 2025a. htthss::::////// //arxiv/ org/ h/tg//bs/220 2501. 01473
Hadi Askari, Shivanshu Gupta, Fei Wang, Anshuman Chhabra, and Muhao Chen. LayerIF: Estimating layer quality for large language models using influence functions. NeurlPS, 2025b.
13

---

Published as a conference paper at ICLR 2026
Frederick Liu, Amir Najmi, and Mukund Sundararajan. The penalty imposed by ablated data augmentation, 2020. URL
org/ abs/ 04769
Andrei Kapishnikov, Ben Wedin, Besim Namik Avci, Michael Terry, Subhashini Venugopalan, and Tolga Bolukbasi. Guided integrated gradients: An adaptive path method for removing noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pages 5050-5058, 2021. http::t::/// arxiv org/ abs 2106. 09788
Xiaochuang Han, Byron C. Wallace, and Yulia Tsvetkov. Explaining black box predictions and unveiling data artifacts through influence functions. Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics. pages 5553-5563, Online, July 2020. Association for Computational Linguistics. doi: 10. 18653/v 1/2020. 1r20000cclmmaii44900 492..492. URL 2020 acl-minnmmain. 492/
Pouya Pezeshkpour, Sarthak Jain, Sameer Singh, and Byron Wallace. Combining feature and instance attribution to detect artifacts. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 2022, pages 1934-1946, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. (0.18863//1/2000 findings-acl. 153. URL Siltts::/aclantho.le wtgp:oyy.rggg///// findings acl. 153/
Xinyi Wang, Antonis Antoniades, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, and William Yang Wang. Generalization vs memorization: Tracing language models' capabilities back to pretraining data. International Conference on Learning Representations, 2025.
Han Zhang, Zhuo Zhang, Yi Zhang, Yuanzhao Zhai, Hanyang Peng, Yu Lei, Yue Yu, Hui Wang. Bin Liang, Lin Gui, et al. Correcting large language model behavior via influence function. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 14477-14485, 2025.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. In International Conference on Machine Learning, pages
PMLR, 2024.
Zhe Li, Wei Zhao, Yige Li, and Jun Sun. Do influence functions work on large language models? In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Findings of the Association for Computational Linguistics: EMNLP 2025, pages 14367-14382, Suzhou, China, November 2025. Association for Computational Linguistics, ISBN 979-8-89176- 335-7. doi: 24.18653/v1/222. htt.:/...................a 2025. fi sindings--ennl pp..pp........... 775/
URL http::::///aaaattthhoooo org/
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associa- tions in gpt. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 978171387 1088.
Akshat Gupta, Sidharth Baskaran, and Gopala Anumanchipalli. Rebuilding ROME: Resolving model collapse during sequential model editing. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen, editors, Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pages 1178-21744, Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 10.18653/w1/2024
http:///222...mmmmmmee www.eemminnnn..mmmmmmmmmmm 1210. URL https:// aclanthology www.://4...022222mmm_.... main. ma.n.1210...........1..
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in transformer, 2023. URL httpptss///xxxi..ooo http://w....r.///////// 07229
Akshat Gupta, Dev Sajnani, and Gopala Anumanchipalli. A unified framework for model editing, In
Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for
Computational Linguistics: EMNLP 2024, pages 15403-15418. Miami, Florida. USA, November 2024b. Association for Computational Linguistics. doi: 10. 10.18653/v1/2022 1/2024 nd
URL wttpt:tt///// pesttss//aclln antho ogy wtwttholooy..org///000 wtw.:iiiiinnnniiinnnnnnnn ngs- 1p. 903/
14

---

Published as a conference paper at ICLR 2026
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. International Conference Learning Representations, 2022. htRp:tts://// Seattpss//perreriwwwnee///e
id=nZeVKeeFYf9
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora:
efficient finetuning of quantized llms. In Proceedings of the 37th International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2023. Curran Associates Inc.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupata, and Afra Alishahi, editors, Proceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. 10.18663//1/W11854444 URL.
54466
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cris tian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melissa Hall Melanie Kambadur, Sharan Narang, Aur'elien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/2307.0922. 2023. URL htttps:: 259950998
semant i icscholas org/ Corg/orrpsss
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL
wwwp://////aaooooooooooo 15115
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL htthtt::: //arxiv
org/abs/2310. 06825
15

---

Published as a conference paper at ICLR 2026
APPENDIX
A EXPERIMENTAL SETUP DETAILS
This section describes the five-stage experimental pipeline from Section 5
in detail.
Stage 1. We define the anomaly detection task on GLUE datasets with injected label noise. For influence computation, we select up to 4500 training samples, 500 validation samples, and 500 test samples for reporting downstream performance, with 10 random seeds controlling different selections. For binary classification tasks, 20% of training labels are flipped uniformly random. MNL.I is binarized by keeping only entailment (0) and neutral (1) classes, and STSB scores 0-5 are converted to 0/1 using threshold 3; noise is added similarly.
Checkpoints preparation adds LoRA modules (Dettmers et al. 2023 to query and value projections in every attention layer, compresses embeddings to the tokens participating in the GLUE dataset splits. The embedding compression greatly speeds up influence computations based on gradients.
RoBERTa-Large has 355M parameters (1.5M trainable), 24 attention layers, and a classification head with dense projection. Llama-3.2 IB is multilingual, with 16 attention layers, attached LoRA modules, and a tunable classification head (0.5M trainable parameters); embedding compression reduces parameters for influence estimation. Qwen-2.5 1.5B has 28 attention layers, and Mistral 7B has 32. Word embeddings (WE) remain frozen during training, but gradients are computed for influence estimation. Learning rates are: Roberta 3e-4, Llama le-4, Qwen 3e-4 (Ie-3 for RTE), and Mistral 5e-5; all models use 16-bit float weights.
Stage 2. Models are fine-tuned on noisy datasets for 10 epochs from the initial checkpoint. Validation loss and accuracy are tracked to select the checkpoint for influence estimation. Following (Pruthi al. 2020) and preliminary experiments, we use the checkpoint with the lowest validation loss in the following stages.
Stage 3. We split the neural network layers into groups: embeddings WE, classification head CL, and 4 groups with an equal number of internal attention layers. The groups in the results are numbered from 1 to 4. For instance, group of RoBERTa-Larg includes the first to sixth attention layers, also denoted 00-05. Group 2 has 06-11, group 3 layers 1 12-17, and group 4 - 18-23. Similar grouping is done for other models.
The scoring routine computes attributed values from the collected influence tensors of layer groups for a particular aggregation method under consideration (default is averaging). This stage also estimates other indicator metrics such as noise detection rate (based on knowledge of the flipped label), noise histograms, and cancellation effect.
Stage 4.
Training samples are ordered by their influence scores, and the lowest 30% are removed. The filtered datasets are fine-tuned using the same hyperparameters as Stage 2, starting from the Stage 1 checkpoint. Method performance is evaluated via test-set accuracy on samples unseen during training or influence computation.
Stage 5.
We apply the following techniques to optimize influence computation.
Compression. To reduce computation, we replace the full embedding layer with weights correspond- ing only to tokens present in the task dataset. While sub-views may be preferable in production, this approach is effective in benchmarking, especially for multilingual models with many unused tokens.
Batching. Equations 5 depend dot-product VIT Vlg. To maintain the gradients for n train- ing samples, k validation samples, and m parameters, TracIn and Cosine requires ((nkm)) DataInf 0 + n²m) of memory. In one iteration, we pick n; training and kn validation samples s.t. (1i,kim gradients fit the available GPU memory, and compute ((1 k1) influence scores between sample pairs. Staying in (mnkkm) limit requires the iteration through all pair of [n/n1] training and [k/k]] validation batches.
Inner loop has to recompute the gradients because of the memory limit. For A100 80GB GPU, 4500 training and 500 validation samples, influence values on Mistral embeddings are computed in 20 batches our experiments. It is still very expensive for Daf because also depends on Vig Vly.
16

---

Published as a conference paper at ICLR 2026
B BEST TEST SET ACCURACY DISTRIBUTIONS AND TRENDS
This section presents the best test set accuracy distribution after filtering for RoBERTa (Figure 6p. Llama (Figure 7) and Qwen (Figure [). Tracin indeed has decreasing performance trend from early later CL layers across models. However, Datalnf and Cosine frequently outperform TracIn on early or middle attention layers for bigger models, and later attention layers for the smaller Roberta.
MRPC
SST2
QOP
60-
COLA
MNLI
RTE
STSB
70-
30-
60-
75-
Datadat
Trxds
Cusine
Datalaf
Toalo
Cosine
Dutabal
Truda
Chiine
Datila
Trxda
Conin:
TracIn..
TracIn t0
06-11
12-17
18-23
Figure 6: Roberta-Large best set accuracy after 30% filtering over 10 runs.
QNII
QQP
35
70
COLA
MNLI
RTE
STSB
80-
12-
70-
75
70-
Untalni
Trnckt
Datainf
Tractn
Costne
Dutalri
Thxcit
Coene
Dabain
Tractn
Conine
TracIn..
TracIn 10
04-07
08-11
12-15
Figure Llama-3.2 IB best test set accuracy after 30% filtering over 10 runs.
QNLI
MRPC
SST2
QQP
50
60-
COLA
MNLI
RTF
STSR
60-
50-
66-
70-
Uataini
Trnckt
Cooone
Daiatn
Tracto
Costne
Dotalri
Coene
Dabiii
Traccnn
Conine
Tracinw
Tracin]0 10
WE
00-066
07-13
14-20
21-27
CL
Figure 8: Qwen-2.5 1.5B best test set accuracy after 30% filtering over 10 runs.
17

---

Published as a conference paper at ICLR 2026
The following trends presented for the best configurations, influence method, and layer group for Roberta (Figure 99, Llama (Figure I0), Qwen (Figure II and Mistral (Figure Accuracy trends measured on the test set, not used for training or influence computation.
ONU
MRIC.
OOP
75
30
OLA
MNLI
ITB
STSH
50-
Epoch
Epoch
Epoch
Full
Tradon WE
Cosise, 12-17
Donatad, 18-23
Randon
Figure 9; Roberta-Large accuracy trend on test set after 30% filtering.
QNU
MIIC:
OUP
90-
Accin19.55
COLA
MNLT
RTB
STSD
80
800
60-
60.
10
Fpond
Fpoch
Full
Tradds, 04-07
Cunine, 04-07
Datalff 04-07
Randon
Figure 10: Llama-3.2 1B accuracy trend on test set after 30% filtering.
ONLE
MRFC
S572
80
80-
10
Accincy.5
COLA
MNLT
RTE
STSB
50
60-
2
10
Epoch
Epoch
Epoch
Epoch
Fuil
Tractt, o7
Cosnn, 07 13
Dacaind, 07 13
Random
Figure 11: Qwen-2.5 1.5B accuracy trend on test set after 30% filtering.
ONLE
MBRC
90
800
70-
60
Accuney.S
CII.A
MNIT
RTF
NTSH
70-
40
2
10
10
Equass
Epoch
Pull
TeacIn
WB
Cosine. 08 15
Datainf 08 15
Rundon
Figure 12: Mistral 7B accuracy trend on test set after 30% filtering.
18

---

Published as a conference paper at ICLR 2026
C LAYER RANKING PER MODEL
Table Paodeepreeenttt the results of layer performance ranking based on the measured best test set accuracy after sample filtering, where 30% of the least influential samples according to each layer are discarded and subsequent retraining is performed. In the table, the model weight groups are presented as rows ordered starting from Tracinges 10 TracInge (i.e., selection from WE), followed by the full WE layer, attention layer groups, and the CL layer. Attention groups are shown in the same order they appear in the model, but their size varies with the model.
Table 3: Layer ranks (embed WE, attn 1 1-4, head CT)) and win rates (in parentheses) across models and influence functions (threshold=0.5), '_' denotes N/A. More specifically, Datalnf is infeasible to run on WE and TracIn specific WE variations Tl. Most performant layers are shown in bold.
Roberta- Large
Llama-3.2
Qwen-2.3 1.3B
Mistral 7B
Layers
DataInf
TracIn
Cosine
Datalnf
TracIn
Cosine
DutaInf
TracIn
Cosine
DataInf
TracIn
Cosine
The Thee
T (.49)
(.56)
(.49)
(.58)
2 (448)
(.52)
1 (.51)
2 (.50)
WE
1 (.54)
(411)
3 ((44)
(42)
(.40) (.19)
(.50)
(.58)
(.29)
(.50)
(.35)
((33)
((54)
1 (.53)
(.49)
((42)
2(455)
2(53)
(.61)
(.60)
2
4 (.30)
5 (.40)
(.40)
(.58)
1 (.54)
1 (.53)
1 (448)
(.57) (.60)
1 (.60)
(.48)
2 (.55)
3
2 (.40)
(.44)
(.56)
(.42)
(.38)
(45)
(449)
(.48) (.61)
(.39)
(.29)
(.37)
1 (.55)
(336)
(554)
(.23)
(.22)
(.28)
(.31)
(.30)
2(40)
(.20)
(.20)
(.23)
CL
(.36)
((17)
(.15)
4 (.18)
6.17)
(.15)
(.16)
5.222)
(.18)
(.20)
(.20)
5 (,09)
Here, the ranking given across layers. many cases, first or second attention groups happen to be most performant, which makes them a good selection for detrimental sample detection.
D INFLUENCE SCORE CORRELATIONS BETWEEN LAYERS AND METHODS
Figure 1 presents computed correlations between training-sample attributed scores of different layers and influence methods. TracIn splits the model into three groups of early, middle, and later layers based their score agreement, i.c., the strength of correlation. Datalnf has high correlation with TracInf on later layers such as CL. This layer split suggests that different parameters could capture different detrimental samples, and good aggregation of scores is viable.
Roberta Large
Llama- 3.2
Mistral 7B
sv es
DEDCDE 1506
22
22
uz 02 us 9247
02
02
E2
03
16
3303
0-05 87
10 62 BE
22
160
22 04
0404
Q- 05465666000
B5 03
Besazer a
2402
02 2242
23
22
02
12
05 02
02
82
150581
040606
2407
060
wit 21 ai
2747
4262
12
06
07
0303
850584
82
056577700 oz
05 1583
Codnee
Trainn
Danlat
Cosine
Tracnn
Danaial
Cosite
Trael
Docain
Figure 13: Influence scores correlation for 3 models, methods, and layers.
19

---

Published as a conference paper at ICLR 2026
E DETECTED NOISE IN 30% LEAST INFLUENTIAL SAMPLES
The following results present the noise detection across layers for Roberta-Large (Figure 14p. Llama- 3.2 IB (Figure [5). Qwen-2.5 1.5B (Figure [6], Mistral 7B (Figure 17). The consequent conclusions are (1) LoRA Value B is the best in capturing noise across models; (2) layers with high NDR also have high downstream performance; (3) spike NDR moves from later attention layers for shallow models to the early-middle layers for deeper models.
Datalaf
Trachn
Cosine
Laver-wise NDR,
Layer
Layer
Quary
Query
Figure 14: Roberta-Large NDR for 30% threshold across modules and layers.
Datainf
Tracla
Cosine
Layer-wise NDD.
Layers
Layax
Query A
Query
Value
methods
Figure 15: Llama-3.2 1B NDR for 30% threshold across modules and layers.
Layyr--ase NDR. 15
Datainf
Tractn
Cosine
Query A
Query
Valce
Value
methods
Figure 16: Qwen-2.5 1.5B NDR for 30% threshold across modules and layers.
DataInf
Traefn
Cosine
Layer- wise NDR.
1220 WE
10
Layers
Query A
Query
Valee A
Value 8
WE methods
Figure 17: Mistral 7B NDR for 30% threshold across modules and layers.
Next, from the following result we can conclude that Value B on network middle layers captures the highest noise levels and is good choice for influence estimation.
20

---

Published as a conference paper at ICLR 2026
F BEST TEST SET ACCURACY AFTER FILTERING
The following tables 6 present the ranking of filtering methods based on different influence scoring and aggregation for models under consideration.
Table 4: Soberta-Large best test set accuracy on GLUE after 30% filtering with influence scores. Methods are ordered from best worst with average ranking across 10 runs per setup. The highest accuracy means are highlighted.
Mahhod
App
Layer
Win
QNET
MRFC
COLA
MNLT
RTB
Full
0.52 2012
89.5 195=112 12
86.0 2.5
$4.7 0.0
85.2 = 1.5
85.0 1.2
$7.3 2.0
76.1
5.0
8500 = 1.5
Cusine
Kank
18:25
0.66 2: 0.15
87.9
= 19
77.
1.6
$1.2
1 1.05
79.9
1 4.5
79.6
1.3
56.3 2.
62.5
70
66.9
1: 1.3
Datalof
Mean
18-25
0.86 d: 0.16
87.4
= 2.1
78.1
#:2.1:2.11
94.4
0.6
80.0
8: 3.1
K3.3 2 1.2
85.5 2:2.
60.5
5.6
86.7
+: 1.5
Datator
Mesn
all
01.66 0.16
86.6 2.5
77.7
1.3
94.3
0.9
79.5
288
83.5
1.2
84.1 1:20
67.3 6.5
86.6
1.7
Datalof
Vate
18-25
0.65 0.16
87.7
77.5
1.8
80.1
79.1
16.1 +-2
Datalff
87.6
1.6
77.7 1:7
TracIa
Vate
00-05
0.61 0.16
86.4
1.9
79.9
2.1
78.6 4.1
76.6
63.5
86.9
Datalof
Vate
00-05
0.59 0.16
87.0 2.1
79.6
2.26
$4.2.:02
78.5 2: 4.1
77.1 1.6
85.1 1:28
63.6
6.0
86.7 = 1.3
Traelinge
Vore
WE
0.58 0.17
86.9 1.9
78.9
2.2
94.1 0.0
78.4 = 3.8
77.2 2.1
K5.4 2:2.1
(AA4 5.7
86.4
= 1.2
Cosine
Kank
12:17
0,55 + 0.18
87.8 2.
75.7 1.9
94.0
77.1 8 4.3
2..6
85.9
2:2.
63.8 7.1
86.5 = 1.6
Tracla
Vate
18:25
0.62
73.3
0.7
80.0
1.
86.0 1.7
62.7 64
$6.
Traela
0.61
9422.
79.1
85.8
Traela
Vote
06-11
0.58
87.0
78.8 3.0
94.1
78.9
85.3 2.26
87.0
Tracla
Vate
12-17
0.58
87.4 1.9
7644 1.8
94.0 1.0
78.2
K9
77.7 8:
86.4
63.6
86.6
Cosine
Vate
18-23
0.57 0.17
87,9 1.8
76.0 2.0
94.1
0.3
79.0
2 5.4
78.3 1.7
$5.7
1:22
61.9
7.0
85.6 1.6
Daialnf
Vite
12-17
0.56 1: 0.18
86.9
2111
76.7
2.3
94.0
0.9
78.1
4 4.5
77.9 2.5
$6.2 2:2
64.0
57
86.5 1.5
Cosine
Kank
all
0.56 :: 0.17
87.5 =
75.2
2.1
94.2 1.0
77.8
5 5.3
77.7 8 1.5
26.0
63.7 7.1
86.5 1.5
Traela
Vote
0.55
87,9
1.7
75.6 L.8
0.9
78.1
77.0
86,1 2:22
$6.0
Datalof
Meus
2.5
78.9
K3.4
Tradia
0.53
86.5 14
78.1 1.5
78.2 4.1
76.6
1.6
Traciait
Vate
WE
0.29 = 0.16
86,9 =
75.8 1.9
78.5 2 4.3
76.1
23.9
037
53
$6.7
Dataoof
Vaic
06-11
0.55 0.16
87.0
= 2.3
76.0
1.4
94.0 0.9
79.0 + X.
76.5 + 1.9
K5.5 4-2.
62.3
65
86.9 1.8
Datalof
Rank
18-25
0.53 = 0.18
87.5 1.9
76.5
1.4
93.8 1.1
79.7 + 4.1
7.
85.5 2.5
62.1 56
864 14
DataInf
Rank
all
0.52
87 2.4
76.4 1.9
94.4 0.5
77.0
1.
1.9
63.2
859
Cosine
1.7
Traela
Mean
WE
0.17
81.3 2.7
3.3
78.4 2.5
1:30
61.1
Datalaf
Kank
12:17
0.19
86.2 2.6
75.9 L.B
94.5 0.9
26.1
1.
63.2 600
2
Dutalof
Meas
00:05
0.17
85.1
3.0
76.3 3.4
0.1
19.0
+ 3.0
8142
$2.1 4:22
63.0
8.1
85.8
1.6
Cusine
Mean
12-17
0.55 0.16
86.7 22
7..0
11
76.1
2 3.6
2..4
$4.1 2:22
61.7
62
87.0
= 15
Cosine
Mess
18-25
01.54 = 0.16
88.0 19
75.8 2:7
94.2 1.0
77.4 + 3.5
77.6 1.6
85.1 2:2.
61 54
86.5 = 1.1
Datalof
Vote
87.8 1.6
76.3 2.5
78.5
80.0 7.70
Danalnf
Mesn
0.52
Tradia
1.6
76.6 2.4
78.0
$5.3
61.0
Traein
Mean
00:05
0,45 0515
84.4 34
77.
4.42
93.7 1.10
78.3
2.6
80.4 4.43
62.6
61.9
63
K5 4.2
Trioln
Kank
all
0.49 0.17
87.1 20
$4.6 1.1
75.1
+ 1-2
623 SS
850 1N
Datalnf
Mean
06 11
0.49 :: 0.15
83.9 322
76.8 2.7
90.9 0.9
77.5
a 2.
62.4 1.4
S1.1 2.5
66.6 68
85.7 244
Tracin
Menn
WE
0,46
85.0 2.4
94,1
78.3
$1.6
62.7
84.9
Traeln...
WF
Cosine
0015
87.6 211
04
158
16. 1.S
Cosine
Vote
12-17
0.47 0.17
86.3 2.5
75.3 2.1
94.2 0.9
74.5 2 4.3
77.9 2.0
$6.0 1:23
6244 5.
86.5 1 1.5
Tracia
Rank
12-17
0:42 a: 0.17
81.7 = 11.2
76. 2:1
94.3 1.0
76.0 + N.
77.6
1.8
84.7 4:233
63.3 87
2.
Random
0.40 :: 0.15
84.3 19
71.2 2.1
93.6 1.1
28.5
à 2.4
2.
$1.62222
$7.8
26
2.
Tracla
Mean
12-17
0:40=
82.2
76.1 3.0
$1.0
76,9
SL.
65.1 K3
R2.5 4.00
Traeta
Rank
WE
77.1
83.8
1.5
634 53
85.5
Cosine
Meas
0.39
85.1
73-2 2.8
93.7 $4.111 L.D
75.0
76.7
3.3
67.5
Cosine
Rank
CL
0,45 0.16
87.5 1.7
713
1.0
17.5 2.33
76.6 1.8
$5.3 4:2.
60.1 54
86. 2.08
Tracint"
Rank
WE
0.37 0.15
85.7 =
70.9 1.1
94.1
0.9
75.1 1
77.7 2.
$3.9
1.5
63.1
53
85.3 1.7
Datainf
Rank
06- 11
0,36 0.15
86.2
=
2.56
94,2 A 1.2
23.8
8 4.5
7.29
83.1 & 1.6
6366 73
85.6 1.3
Cosine
Kank
01
0.39 0.15
86.5
1.7
7.1
$4.0
14.6
16.0
$4.2
=
1.5
Tracin
86.0
211
73.9 3.36
73.8
1.3
63.5
Cosine
Menn
06-11
(1.40 0.14
85
TA
71611.111
93.7
15.8
N4
SA77 T
654
T3
66.3 2 1.
Tracla
Mean
06-11 11
10
0.38 0.13
82.4
3.6
73.5
2.0
93.3 1.5
17.2
2 3.6
N2.7 4 1.4
S0.2 3.3.2
64.9
8.9
84.5
2.9
Dotainf
Rank
00-05
10
0.37
= 0.14
85.5
23
70.6
2: 1.3
91..3
0.
74,6
2 4.7
77.5
1.6
$4.0
2.3
63.0
58
85.4
Tracla...
Rank
WE
10
0.36
= 0.15
85.8 1.
71.7
2: 1.8
93.9 0.9
23.9
+ 5.6
84.1 1.6
63/6
67
85.2
1.5
Cosine
Mesn
00-05
10
0.36 0.14
85.
2.6
71.
2:4 7.4
93.8 0.
74.6
488
77.7 2.0
84.4
63.5
50
84.5 2.0
Cosine
WF
85.4
780
83.6
Cusine
86.0
72.3
2.7
74.4
77.7
62.9
Tracla
Menn
18-23
0.34 0.12
83.1 4/7
76.5 1.7
1.5
75.2 2 2.5
80.7 1.
80.7 1:30
56.5
21
Cosine
Rank
00:05
10
0.33 0.14
85.7
= 2.6
702 L.1
94.3 0.
24.3
5.7
7
53.7 199
61.4 6.0
849
Cosine
Vate
CI.
0.35 = 0.15
87.0
= 2.5
703 + 1.1
94.1 0.7
73.8
+ 4.6
75.3
1.8
K5.4 2-23
58.8
55
85.7
1.2
Cosine
Vote
06-11
10
0.33 = 0.15
86.1
= 1.9
71.
94.1 1.0
72,5
8 4.8
76.7 1.7
84.3
1.7
614 60
85.7 1.8
Cosine
Vite
00-05
10
0.33 0.15
85.6 2.4
70.7
94.3 0.6
73,9
80.8
63
8533
Traeta
Rank
0.31
85.8
70.5
Danalff
0.29 0.15
81,9
1.22
75.0
60.8
Traela
Meut
0.29 0.12
79.8
90.7 3.0
25,9 29
79.1
799
600=
82
Traela
Mean
CL
10
0.21 0.10
80.2 422
25
90.0
24,6 24
78.2 1.7
79.8 1:4
58.7 5.1
2.0
Traela
Rank
CI.
10
0.11
79.1
4.0
75.4 1.9
87.9
+ 3.1
76.1 + 26
76.8
+.0
79.1
59.2
44
83.2 +
1.8
Cosine
Mess
CL.
10
0.17=
83.7
28
7.
92.5
1.7
25.2
7673
1.9
60.7
45
$1.8 13
Rows are ordered according to mean rank from the best method to the worst. Two baselines are Random (dropping 30% uniformly) and Full (removing all noise (20%) and 10% of random clean samples). A good filtering method should outperform the Random, the lower bound, and approach the Full, upper bound. The tables highlight the best bas-blsel methods for each GLUE dataset. AUC measure is specified for the NDR curve; high value indicates skew of noise towards the beginning of the influence range.
21

---

Published as a conference paper at ICLR 2026
Table 5: Llama-3.2 1B best test set accuracy on GLUE after 30% filtering with influence scores. Methods ordered from best to worst with average ranking across 10 runs per setup. The highest accuracy means are highlighted. In contrast to Roberta and Mistral, influence methods demonstrate poor performance of noise detection compared to Random baseline. Llama noise distribution on figure 21 confirms that a lot of mislabeled training samples become influential.
Matrod
Layer
Kank
Win Rale
QNLT
MRFC
SS12
QUP
COLA
MNTT
RTE
STSB
Full
0112
2.
84.4 14
834 8.
820
4..9
Cosine
Kank
01-07
D365 D.IS
77.5
1:3
71.3
2.
75.8
20
17.3
27
614
M19
+ 1.
Random
0.64 0.15
76.7 5.1
71.6 2.5
89.5
2.0
80.2 2.1
80.5
2:9
73.3
2.7
56.3 1.9
82. 0 2.5
Dataln
Rank
04-07
0.64 0.15
76.9 14
70.7
0 1.3
90.8 1.3
73.0
76.0 =2
76.9
61.
2
Dutalof
Mcan
04-02
26.5
70.9
1:64
78.1=
63.0 29
Traela
Rank
04-07
766+14
706+1. 1.0
75.7
809+ 614+47
Datalaf
Rask
00403
0.16
77.7 25
73.2
7
613
Datalaf
Mean
00-03
0.58 0.16
76.2 2:4
1.
84.5 4: 10.1
76.5
78.5 14
73,4 33
$9.9 3.6
1:
Traelagr
Vire
WF
0.58 0.17
7..
+-.
80.5
3.1
75.2
41
78.2
0S
75.4 14
$8.0
Tracla"
Vore
WE
0.58 0.17
7..9
72.0
90.0 3.2
75.7
4.1
76.3 18
75.3 311
583 3.9
7
Cosine
Rank
0.57 0.16
77.5
70.3 0.9
9.11
72.3
75.6
77.0
618
7.
Cosine
4-22
61.5
Datalnf
05-07
0.18
75.5 22
74.8
39
39
Tracln
Vote
00-03
0.56 0.18
73665.1
72.4
2.5
90.1 3.30
74,4
77.0
188
72,9 3.4
580
77.7
Damala
Rank
all
0.56 + 0.17
712 3.1
70.6
2: 1.0
40.1
22.3
75.5 = L.I
16.9 3.0
6.7
73.5 1.6
73.5
Tracla
Rank
all
0.55 0.16
77.7
3.4
70.3 1.1
90.8122
71.9
75.2 15
75.0 7..4
24
Datalof
Vore
03-03
0.54 0.18
73.3
5.6
2.0
90.2 2.5
74.2 4.1
76.8
3.4
58.5 4040
474
Tracta
WB
0.18
71
58.
Datalar
0.17
76.61
76.4
Cosine
Mean
01-07
0.5
TTA 3.
100 1.0
751
14.5
61
16
Cosine
Rank
00-03
0.58 0.15
78.0
2.4
70.0 #: 1.0
89.6 +: 4.0
12.6 33
75.9 = 211
26.1 227
61.5 4.
2.7
Cosine
Meon
00-03
0.58 0.15
78.2 8: 1.4
70.3
1.2
90.1
1:2.7
72.1
3M
76.0 = 235
74.5 8
62.0 1.9
7000 35
Tracla
Rank
03-03
0.57 0.16
77.7 + 2.4
70.0
1000+10 L.1
88.6 5.5
73.3 3.1
75.9 211
75.6 2.7
$9.0 610++47
783+
Trxca
Vote
05-07
0.55 0.18
74.5
71.0 1.0
90.8
4.
75
74.4 3.0
Tradln"
Mean
WE
0.15
4:74
72.6
Datalaf
Mean
0.50 0.16
44
+.25
$1.7
75.2
79.1
72.3 3.0
61
Cusine
Mean
08-11
0:19 0.16
2.8
4: 1.3
1.
72.8
28
75.1 = 1.A
71.4 3.5
60.0 2.7
71. 2
DoiaIní
Vite
all
0-46 + 0.18
72.1 4.60
71.0
1.4
49.4
72.8
4 4.0
75.2 = 1.8
72.0
& 3.6
$8.4
7666
Cisine
Vore
04-07
0.54 10IS
WT.
1.0
Y0.Y 27
74.9
2.0
7670
X9
40
Tracla
Moan
04-07
0.14
75.6
70.0
$1.0
78.4
73.1
4.0
612
5.
Tradin.e
WB
0.16
70.7
73.4
Traela
Kank
0.17
75.8 4.7
0.8
72.1 2.
74.8
76.0
72.119
Cosine
Mean
0:49 0.16
77.3 3.6
0..5
92.3
72.5
75.-
70.9 2.8
50.9
72.7+1
Danaof
Mean
08-11
0.49 0.15
74.4 5.4
6.9+22.
82.1 + 12.7
74.9 28
78.6 - 22
72.3 3.7
+8
4.8
Datainí
Vote
08-11
0:46 0.18
72.8
5.6
70.9 1.5
89.9 3.1
72.9
75.0= 1.4
72.3 4.4
S8.1 4.66
Tradin
Vote
0.42 0.18
718 5.
71 1.3
89.7 1.
72.3 39
75.0
71.9
$7.9 4:44
75. 3.4
Tracaa
Vote
12:15
0.17
70.5
712
$8.0
12.0
74.4
70.5
43.1
2.
Tradaa
Mean
0.L4
75.6
44
Tradn...
Mean
WE
0.51 0.15
7:.0
71 1.9
79.1 120
74.8
79.5 1.8
72.9 4.4
$7.9 4.4
19
Cosine
Mean
WB
0.48 0.16
76.0 4.4
70.7
4: 1.5
72.6 3.1
74.9 2.
72.8 5.6
602 4.6
Tracla
Vote
08-11
0:42 0.18
72.0
70.8
1.4
90.2 3:0
11.8 36
74.9
1A
72.0 4:
$7.3 4.6
33
Damaaf
Rank
CL
0.40 0.17
7.12
71.1 1.4
$5.1 6.1
74.7 1.3
74.7 3.3
59.1
2.1
Dutalnf
Vote
12:15
70.9 71
30
70.9
44
2.
Cusine
W15
0.50 0.15
3.66
22
12.6.:43
76.2-26
74.1
Cosine
00-03
0.16
09
25
75.4
6066
Traclato
Rank
wB
0.49 0.16
7022 1.3
$8.7 4.1
12.4 2.1
75.0 22
74.1 3.4
58.7 3.8
2.0
Dotalní"
Rank
12-15
0:43 0.166
7.7
713 1.4
$5.9
9.4
75.1
1.0
74.3
58.6
2.1
Tracin
Mean
WE
0,45 1014
75.7
10
76.3 133
79.0
28
72.3
49
4.7
Cosine
Rank
WB
0.16
71.2
58.6
4.
Traela
Rank
WB
0.16
74
69.8 1.3
72.1
74.8
73.7
5922
Datatar
Vore
0.16
2.3
71.8 3.5
73.9
70.6 4-7
740+
Cosine
Kank
12:15
0.1
75
0..
88L1 6.22
22
75.5
1.6
15.0
59.5
Traela
Rank
12:15
0.36 0.15
75.1 4.9
71.0 1.4
$2.9 + 12.
70.7 2.0
74.7 1.1
73.6 4.3
57.5
70.3 22
Cosine
Menn
12-15
0.35 0.16
7
70.6
0.9
90.6 1.0
70.7 188
75.0
69.7 t: 2.6
59.3
3.0
+
Tradaa
Mean
NOT T.2
70.6
79.7
Cosine
0:42 DTS
YT.7
742
16.1 2.5
RIV
Cosine
Vose
all
0.41 0.14
78.8
3.1
70.3 0.9
$8.5 3.0
69.7 2.3
74.4 = 1.4
75.8 3.0
58.8
4.4
70.8 1 2.5
Cusine
Rank
CL
0.40 1 0.13
75.4
4.4
70.2 0.8
$7.7 6.3
71.6 322
73.8 1.5
76.1 2.7
60.7
4.5
67.7
1.8
Cosine
Vore
12:15
0.33 0.14
712
3.6
0.22
36.6
4.6
68.9 166
74.2
1.2
74.9
56.5
23
Cosine
Vote
CT.
0.32 0.14
76.2 5.4
70.1
8677 1:43
69.5 26
71.8
75.6 2.4
$7.5 1.4
2.
Datalof
Mean
12-15
0.14
7066
692
69.8
606
Tradin
0.15
70.2
Datalar
Mean
0.78 0.5
66.7 5.58
4-4
$1.9
72.8 45
7644 23
68.4
1:12
Cosine
Mean
CL
0.23 0.12
70. 4.9
2.2
192
20
73.6=15
69.3 24
20
6,
Traein
Rank
CL
0.22 0.12
70.7 T9
7.62
12.
20
73
7
15
20
Tracla
Mean
12:15
0.22 0.11
66.5 61
0..5
77.1 109
72.0
23
66.5 38
26
29
Trada
Meun
0.17 0.09
65.2
6.
$3.1599
69.9
65.6
23
092131
Trnela
Mean
0.16
65.
65.6
64
4.1
68
22

---

Published as a conference paper at ICLR 2026
Table 6: Mistral 7B best test set accuracy on GLUE after 30% filtering with influence scores. Methods are ordered from best to worst with average ranking across 10 runs per setup. The highest accuracy means are highlighted.
Marrod
Age
Tayer
Win Rate
UNTT
MRFC
C
MNT
RTE
SYSB
Full
0336 0.12
k7.1 1.5
172210 1.2
953
1.1
84.2 2 21
83.2 =
82.1 = 2x
6000 35
78.3 2 32
Datatof
Vate
00-07
0.84 0:22
K2.9
1/7
74.2
39.3 2.3
80.7
80.3 2:3
80.6
79.0
3.0
66.6 1.9
77.2
Traca
Vote
00-07
0.83 0.23
74.1
14.1::24 2.7
80.1
78.5
28
66.6
76.7 2.1
Datalof
08-15
2.
72.5
90.2 2.3
79.4 30
66.6
+
Datatf
Vore
0.30
71.0
76.6
67
78.5=11
Traela
Vate
081 0:19
84.4
1:.
90.7
1.9
77.3 24
76.4
79.4
29
67.
Traela
Vore
08-15
0.80 0.22
83.8 +: 1.6
+.
90.1 2.0
78.4
23
76.5
18
79.4
30
668
2.
Traciast
Vore
WB
0.78 0.25
82.1
188
2.
2:25
79.0
= 26
80.2 19
77.5
25
66.7
3:
Tracin
Vote
24-31
0.76 0:25
83.3
18
1.
9..6.11..
75.3
28
75.8
79.7
31
66.9
3.
Trneta
0:33
K38
79.3
Datala
0.24
79.5
Dataaf
Vote
24-31
0:24
836
2.
$8.9 4.2.1
74.7 2.8
75.4 122
79.4 2.66
4.
Trieln.r
Viee
Wi:
0.76 0:1h
K2 1.
T8K
2.
MIDI 177
17.5 KM
29
Tracla
Vote
WE
0.76 0.27
82.2
1.6
2.5
$7.4 1.9
78.5
29
79.3 2.2
77.5 3.2
66.0 3.
76.4 3.5
Trxcn
Vote
16-23
0.73 0:22
83.6
1.8
1.00
90.5
75.8
75.6
79.6
67.1
77.9 a 28
Datalnf
0.26
83.4 1.8
1.
2:11
79.1
68.9
Datalni
Mean
0.24
11.4
76.6
16.1
Tradin
Mean
00-07
0.59 0:27
80.4 1.7
22.0
76.4 37
77.0 25
75.7 2.29
63.8 477
73.8 4.
Traela!
Meon
WE
0.57 0:25
79.7
244
72.1 2.1
82.1 1.26
77.3 22
78.9 1.9
75.3 K3
62.5 1.8
Dataloi
Mcan
00-07
0.57 0.25
80.7
2.1
71.9
83.2 3.2
16.7
76.9
1.9
76.4 2 3.0
61. 3.7
3.
Dutalnf
Mean
0.53 0.26
80.2 4.5
70.6
75.0
73.1
76.1
75.5
64.7
4.9
Cosine
Rank:
0.25
74.0
75.6
2.
Datalaf
Kank
0.26
$2.5 1.5
71
74.2
63
22
Datalaf
Runk
00-07
0:37
80.1
7.
72.3 31
75.3
75.7
3.0
6A.1 61.1
72
Danaao
Rask
all
0.44 0.25
79.2 + 1.8
7.9
81. 7.3
72.2 + 18
75.5 = LA
72.8
3.7
64.2 2.0
22
Datafff
Meon
16-21
0.43 0:24
77.9 4.7
70.2
69.3 9.4
72.3
1.6
76.4
1.8
75.5
6 X.
66 4:00
71.1
Mean
WF
DX1 1:35
3.3
76.1 S1
78.J
14.3
144
623
Traela
WB
0.25
79.4 1.9
713+11
80.4 7.1
4.
Tracln
Mean
0:22
78.8
72.7
Traele..
Rank
WF
0:76
78.
1.
72.5
776
73.6 2.5
62.9 18
73
Cosine
Moan
00-07
0:48 0.25
79.2
3.4
71.61. 1.3
83.7 4.99
74.6 1 34
75.4 = 18
74,8
4.0
63.1
3.0
Tracln""
Rank
WE
0:48 0.27
79.1
2 2.4
11 1.5
82.8
2.29
73.3 211
76.7
255
74.1
25
62.5
16
Cosino
Rank
03-07
0.48 0:37
79.6
2.6
84.1 1:4.
72.0
75.2 222
75.7 2.
634 A3
2.
Tradla
Rank
05-15
0.47 0.27
80.9
1.
74.5
73,9
35
43
Tradia
WE
0.26
78.8
71
76.0=
74.4
Tracla
Kank
00-07
0.27
29
71.
3.7
72.5 33
74.9
14.9
6.
720
Cosine
Rank
WB
0.43 0.26
79.4 2.8
7..7
3-.3
70,9 20
76.2 2A
73.7 2.0
63
33
Traela
Rank
All
D41 0.25
79.1 2-1
71.++14
K3.D
71.7
19
75.4 1.5
71.9 +3
63. 33
20
Cusine
Rank
all
0:40 a 0.24
$1A
1.
83.5 8.1
69.6
1 19
73.8
1.6
76.0 2
64.8
21
Cosine
Rank
16-23
0.36 0:28
60.4
7000
74.0
69.1
73.8
75.7 4.9
674 1:77
Mean
WE
750
14.1
Random
0.25
2.5
3.2
73.2
Cosine
Vote
00-07
0.39 0:24
784 3.4
7.00
77.1 4:4.1
71.7.
75.1-117
74.9 2.1
61
Cosine
Mean
08-15
0:44 1 0.33
MLA4
RKV
14
7122
71.2-1 13
72.1 2:444
750
Cosine
Vore
wB
0.38 0.25
78.7
+ 2.2
70.5 + 1.4
76.3 3.1
21.6 3.6
76.3 2 25
75.6
62. 37
7.
Datainf
Rank
16-23
0.37 0224
77.0 2.3
70.0 112
72.4 2:00
75.5
72.1
63.9
22
08:15
Datatar
Rank
0:30
74.7 2:0
72.3
9.6
71.9 19
76.0-
70.4 4:2
64.3
2.
Tracln
Rank
16-23
0.34 0:22
7..0
7.99
72.3 9.1
7
75.3
1.8
70.4 1:4.1
62.9
25
Dumaaf
Rank
CL
0.29 0.19
74.4 3.0
71.9
00.8 5.8
71.9 19
75.2 1.6
68.3
3.1
6422
02
Cosine
Mean
all
0,28 021
7.2
20.3
85.9 4.5
70,8 19
73.4=16 1.6
70.1 2.3
6.33
6.77
Trainn
Kank
24:31
1.31
33
7.
68.1
11
15.5
39
33
22
Traela
19
Tracla
Mean
16-21
0.14 0.33
54
750
44
Domaaf
Mean
24.31
0.51 0.20
75.9 4.2
69.1 4.8
6..6
71.4 22
75.9 - 1.6
70.8 2.8
62.6
4.0
2.
Doialat
Mean
CL
0:20
26.1
4.7
6.
12
75.8= 1.6
69.7 3.77
62:2 3.9
09:0
Cosine
Rank
24-31
0.26 0.19
78.5
1.9
69.9 1.7
64.3 10.4
69.1 19
73.0
73.1 4:77
66.0 1.2
6.
Cosine
16-23
0:30
76.0
4:00
97.3
69.9
670
Traela
Mean
0.19
710
74.1
75.3=
GKA
61
3.
Cosine
Vue
0:19
79.7 1.7
70.1 1.8
65.7 8:79
66.9 24
72.8
74.7 3S
62
6
Triela
Mean
24.11
15
0.27
0:30
750
1.5.
JO.1
70.7 3.1
15.4
= 1.3
NLAI
2.6
41
3.
Cosine
Rask
C1
13
0.23 0.18
2.6
09.2 1.1
66. 10.1
68.5 20
72.6
1.
72.2 4.9
65.5
66.3
Tracta
Mean
72.1
4:30
2 13
11
68.4 4.0
61224:88
3.
Cosine
2.
74.5
64
Cosine
0.14
74
692
64
18
708-
74.1
619
250:26
Cosine
Mean
24-31
0.16 116:100 0.15
699
1.0
688
14
72.8
22
67.7 a 30
13.
64
Cosine
Voe
24-11
0.15 0.13
782 + 1.8
69 1.6
59.1
4.56
67.+22
71.9-16 16
73.6
61
65
Cusine
Mean
CL
14
0.08
70
57.1 09
100
68.5 13
15.2=15
62
62
23

---

Published as a conference paper at ICLR 2026
G PROOF OF CANCELLATION EFFECT THEOREM
Theorem G.1 (Cancellation Can Improve Influence Estimation.). Let X training set. Con- sider:
two samples ø1, E X, where #1 is noisy and T2 clean;
2. two parameter vectors and w such that their cancellation scores satisfy C(0) < C(w) with ((w) + oo for (
3. the TracIn influence scores le (based on 0 alone) and Lew (based jointly on 0,w);
4. influence score distance between Iι and T2 w.r.t. weights @ and validation samples X': Agl(ā1, 82 X') ( )) - 2 X', 0)].
Then there exists a validation point ズ3 such that:
Ao [11 82 Rs) > Ale (z1, 32 Øz);
Le. the separation between noisy and clean samples strictly larger under Io thereby showing that contrary to the claim of (Yeh et al. 2022), the inclusion of weights with high cancellation can improve influence estimates.
Proof. Suppose, for contradiction, that no such validation point exists. Then for all Ø3.
Aoo S Aoo
Let
a, (E 0)
Ol(2,, W)
b, Ow
00
Cancellation assumptions. Since ((w) + the gradients at nearly cancel:
b2 < O,
161 + bel
= ++
For 0, cancellation is finite:
0102
aa + a2l
Influence relation. From the TracIn definition and the cancellation assumptions, without loss of generality, one obtains
Alo = (az(ai 02)1, ATo =
03
01 az
= |0z(ai a2))l(
b2
by
by
bi
b2
a1
02
a3
C(w) >> C(0)
bj
b2
a1
02
ATo >> ATo I
E
bg
a1 02
(a + 02
a3
Contradiction. By assumption Nlow < Alg, which requires
<. 0.
|a + 02l
Since E > 0, this inequality forces azbs
Thus the assumption can only hold if all validation points ã yield anti-aligned gradients under and w. Such universal anti-alignment is impossible, because and typically share co-directional gradients on at least some validation points. This yields contradiction.
Therefore, there must exist validation point 33 such that
ATo ATo,
completing the proof.
Remark. This shows that contrary to the claim of (Yeh et al.
2022). inclusion of weights with high cancellation can improve discrimination in influence scoring, although the improvement depends the choice of validation sample. Indeed, there are also cases where Alg < Alg. so the proposition in (Yeh et al. 2022) cannot be concluded in full generality.
24

---

Published as a conference paper at ICLR 2026
H RANKING AND VOTING ON ROBERTA AND LLAMA
The figures 18| 19 show the change of Roberta-Large and Llama-3.2 1B performance when we filter the training set with rank and vote aggregations across different influence methods. Mistral 7B has the most benefit of these methods (see figure 3. Transparent distributions correspond to statistically insignificant (Wilcoxon p p-value 0.1, as we have 10 runs only) or less than percent change.
ONLI
SST2
O0P
MNLI
treh
101
Cooe
Trxcs
Figure 18: Roberta-Large accuracy change % with ranking and voting compared to mean aggregation. Voting has the most positive effect on the TracIn influence method, greatly boosting the performance of the last layers (CI.) scores. COLA is the "toughest" dataset (accuracy drop across layers and methods). The mean aggregation is still better here.
ONLI
SST2
OOP
COLA
MNLI
RTE
STSB
Teach
CHete
Coasc
Tracet..
Truds
Figure 19: Llama-3.2 IB accuracy change % with ranking and voting compared to mean aggregation. SST2 and MNLI are greatly boosted, while COLA and QQP demonstrate drops.
For NLI tasks, both Ranking and Voting yield consistent improvements, while on CoLA and QQP we observe a performance decline.
25

---

Published as a conference paper at ICLR 2026
1 NOISE INFLUENCE DISTRIBUTION ON MODEL LAYERS
This section presents noise distribution histograms for methods and layers. Every chart contains 10 quantiles, 450 samples each, depicting the relative noise count in the corresponding influence value range. The first and last quantiles represent the least and most influential samples, respectively.
Trel
Trachng
WE:
00465
66-11
12-17
18-21
ALC-66
A3C-
AUC-31
ADC-89
AUC-37
Duaid Trahs
QNLI
AUC-64
ALC-II
AUC-64
ALC-64
AUC-69
AUC-73
ALC-77
AUC
AUC-
AUC-84
AUC-J8
AUC-
AEC=669
ALK 76
A1CC=
AU
MIRIC
AUC
AUC-701
AUC+,70
AUC-
AUC+33
ADC+33
AUC-
OHin Deald Taalo
ALC+654
AUC
AUC+90
AUC-96
SST2
AUC
AUC
6UC=
ALC=AS
AUC 80
AUC=90
ALC=
AUC=
AUC=89 85
AUC=SS
9UC=
AUC=95
AUCHUM
AUC=
ALC-
AUC-09
AUC-77
AUC--R
AUC
PPP
AUC=661
6LC-6l
AUC-62 62
ACC+ 5k
ADC
AUC-72
ALC+
AUC-65
Trado
ACC=.70
AUCh=76
AUC-
AUC= 30
Coins
AUC-59
AUC-57
AUC-75
AUC-79
AUC-
Daild
COLA
AUC-55
ALC-57
ALC-57
A9C-59
AUC-78
7I
ALC= 50
AIK=
AUC=
ALC- 83
AU
Oniic
ALC=A5I
AU
AUCC=74
ALC--R
AUD
Dadd
MINI
59
ALC-59
62C-
AUC-67
AUC-63
Taals Onnn Deald Tado
Latt
AUC--72
AUC-33
AUC-
ALC-90)
AUC
62C
ADC+55
AUC-S7
AUCC 56
SUC=S
AUC= S7
ALC=
ALC
59C=
ALCS S0
AUC 45
A3C+
AUC
A4C+
ADC-90
AUC-31
Dadel Trade Coaie
63C=66
63
AUC-75
AUC-
AUC-20 201
ALC-AR
RUC-R
AUC- 39
AUC-122
Figure 20: Roberta-Large noise distribution across mean mean-aggregaiie aggregated influence score range for influence methods, datasets, and selected layers. High bars represent bigger percentage of noise in the corresponding quantile. The noise is usually the first quantiles (least influential), as expected. Methods have the least discriminative power on the RTE dataset. TracIn frequently has high noise in the last quantile across layers.
An influence function is not useful for anomaly detection it is closer to a uniform distribution. For selected methods, most mislabeled samples are concentrated at the beginning. However, some layers
26

---

Published as a conference paper at ICLR 2026
and methods have high influential noise level. For instance, TracIn demonstrates relatively high spikes on both sides across configurations. TracIn on WE and CL has a high percentage of influential noise. Generally, performant attribution should strive minimize the noise entropy, producing unimodal but not uniform distributions.
The following figures 20
present how mislabeled samples spread in the influence scores. In the experiments, we discard the first two quantiles.
21
22
Thelt
Tracind
WE
00-05
04-07
08-11
12:15
AUC=
ALC
AUC=A7
6L
61
Dunii
ONL
AUC-65
AEC+609 50
AUC
AUC+777
ADC
AUCCUTT
OHIn Deard Takl
AUC>66
AUC-U7
AUC-A1
ALC-00
AUC-
ALC-
AUC-69
AUC-71
ALC-
AUC
AUC-75
AUC-77
AUC-78
AUC-
AUC
ALCC.63
AUCC-76
AUC--77
ADC=
AUC=66
Draid Trado
AUC- 60
ALC-
AUC- 60
ACC-666
AUX 72
AUC-74
ALC=6f
AUC- 56
AUC-
AUC
AUD
Cosre
AUC-06
AUC-71
AUC-A9
ADC-66
AUC-67
Dadel
AUC-A1
ALC-10
AUC-
Tuats Ondrc
Ialll
AUC-71 71
ALC=
AUK
AUC=777
ALC= 76
AUCC=76
ALCCAA9
AUC
AUC
ALC-RI
AU
COLA
AUC-62
ALC+67
6UC-66
ALC*699
72C
AUC-
AUC-
AUC- 77
ALCC-76
AUC
AUC+
ATC+-0
AUC-10
ALCo.51
AUC
AUC-A6
ADC
AUC-61
MNLI
59C=
A
AUC 60
ALC
AUC
AUC=AS
ALC M
AUCEST
ALCH.5 59
AUC
59C
ADC S3
AUC-ST
AUC-
57
AUC
AU
59 59
ALC--00
AUC-AV
AUC-AI
ALC- 40
AUC-60
Cous
ALC-A3
AUC-66
AUC-AI
MLC-8
AUC-57
Daslel Tosh Oodee
STSB
59C-59 59
ALC-9
59C-59
ALC-A2
AUC-66
AUC-AI
ADC-56
AUC-55
AUC-61 61
ALC--66
AUC-AS 63
ALC- 6l
Figure 21: Llama-3.2 IB noise distribution across mean aggregated influence score range for influence methods, datasets, and selected layers. Similar to Roberta-Large (figure in many cases the noise in concentrated in the first quantile. However, Llama-3.2 IB scores demonstrate less discriminative power, effectively spreading the noise more across the range. In other words, noise has more influence on Llama model.
27

---

Published as a conference paper at ICLR 2026
Tracloer
Trackoy
WE
00407
16-25
2431
ALC=
AUCS 76
AUCs,77
ALCE
A
Duald Tracta
ONLI
AUC-662
ADCH
AUC-65
ALC+,70
AUCH799
AUC-71
AUC
AUC-211
ALCC+73
AUCH64
AUC+.81
AUC
ALC=M0
AUCSTT
AUC=AM
AUCE
AUC
Duald
MRIC
mmil
Iaad
AUC-75
AUC-75
AUC-788
AUC-76
AUCH
AUC--4
Onin Driid
AUC=71
SST2
AUC-61
ALC=
AUC--2 62
ALC-
AUC
AUC=557
ADC-
AUC
AUC-65
CCC-66
AUC-
AUC-83
ALC-
AUC
ALC-Al
AUC-
AUC-A9
ADC-66
AUC
Deald
bail
Meall
QQP
6LC=6l
AUC-- 62
ALC=669
ALK
AUC- 64
AUC=
AUC
Tracto
AUD
Colle Dadii
A
ALK 7
AUC--7
ALC M6
AUDS
COLA
AUC+
AUC-75
AUC-80
AUC+-76
AUC+80
AUC+
ADC+ 51
AUC
ALC=
AUC
AUC= 64
AUC=
AUD=
Duahi
MINII
AUC=57
S
S0C=
ALC=
AUC
AUC=A2
ALCS-
AUC
AUC-65
AUC-
AUC-72
AUC-AR
AUC*
AUC
ALCC
bull
ALCC 56
ALC-
AUC--S 55
57
AUC
AUC- 59
ADC
Trafa
ALC-56
AUC-30 50
AUC-A1
AUC-,99
AUC-3T
Drahi Trabo
59
AUC- 68
ALC- 68
AUC-65
ALC-
AUC
Figure 22: Mistral 7B noise distribution across mean-aggregatee influence score range for influence methods, datasets, and selected layers. As with Llama (figure 21). we observe that noise has more influence on deeper Mistral. For instance, even for the Datalnf method, we observe a high concentration of mislabeled samples in the last quantiles. Cosine similarity has noise spike in middle of influence range instead. The middle layers the Mistral are less susceptible to the influence of noise across methods.
28

---

Published as a conference paper at ICLR 2026
J INFLUENTIAL DATA IDENTIHICATION ON AUTOREGRESSIVE DATASETS
Datasets. To further validate the RQ2 (first is not better than last), we conduct the experiment on the autoregressive datasets introduced in Kwon et al. (024): Grammar, Math, and Math (With Reasoning). Unlike the settings used in the main text, where detrimental samples were synthetically injected, these datasets contain semantically related instruction categories. In this context, desirable behavior is that training instructions exert stronger influence on test samples from the same semantic category. In this way, we assess how well influence scores recover meaningful relational structure rather than merely detecting artificial noise. Such scenarios are highly relevant for practical applications, including knowledge probing, debugging data pipelines, and detecting harmful misaligned training examples.
Additionally, we extend our experiments to incorporate more recent gradient- and activation-based influence estimation techniques. We report the following results for Outlier Gradient Chhabra et al. (2025), Kronfluence (EKFAC)
New methods.
et al. 2023), and RepSim Li et al. (2025). We detect gradient outliers by scoring training samples with OneClassSVN using an RBF kernel on a per-semantic- ser-smmaiii-category basis. EKFAC scoring combines forward and backward signals to estimate influence. RepSim is based purely on hidden representations measured after each attention layer either on last sample token averaged across all tokens of the sample (mean).
Models. The results are provided for Qwen2.5-1.5B (Ir=3e-4) and Mistral-7B-v0.3 (Ir=le-3).
Metrics. We use the evaluation metrics, AUC and Recall, from the original work Kwon et al. (2024). Higher values indicate bettser alignment between the sample ordering by the influence technique and the actual semantic category of the test samples.
Conclusions. Figures 23
24 present the AUC and Recall across layers for Qwen, and Mistral. We observe that RQ2 conclusion still holds for new datasets.
26 for 25
First, early layers consistently exhibit weaker performance in AUC and Recall for both models across methods. Similarly, the last attention layers show drop in AUC-Recall in the majority of cases. Unfortunately, the best- best-performing layers vary with model-dattset-methhd, but generally, we observe the performance maximum for 20-24 layers.
Second, for both Qwen and Mistral, the LoRA module (attached to value projections) delivers the strongest performance across most configurations. Our practical recommendation from consistent observations across all evaluated models and datasets is to rely on influence scores computed on these modules. These weights yield the highest NDR, AUC, Recall, and downstream task performance.
29

---

Published as a conference paper at ICLR 2026
Layer,
Layer,
Layes,
Train
Coaine
Dataled
Ordler Groakent
RepSim
Davy
Qway
Volue1
RepSin Jet
Reptin Mon
Figure 23: Qwen 2.5 1.5 AUC on influence datasets.
Sentence Recall Laver
Much Recall vs Laver S
Math Renson Recall Laver, S
Thas
Cosine
Datiaf
Outlier Grudlent
RopSim
QAXy
Quty
Valee
Velue
BeySin Lini
RapSin Moa
Figure 24: Qwen 2.5 1.5B Recall on influence datasets.
30

---

Published as a conference paper at ICLR 2026
Layer,
Layer
Layer,
Coaine
Ordias Groskent
ReySim
Quir
Query
Vi A
Valuo
Repdinn Jast
Repfin Mon
Figure 25: Mitral 7B AUC on influence datasets.
Layer.
Layer.
Layer,
Tocin
Cosine
FEEREBRANC B BAARRAAAAN
Daraid
Oullie Gradent
DRFAC
RepSiin
Qtary
Valle
Yaloo
Hagiim Last
----
Reptini Man
Figure 26: Mistral 7B Recall on influence datasets.
31

---

Published as a conference paper at ICLR 2026
K SAMPLE INFLUENCE VARIATION ACROSS MODEL LAYERS
We provide the following visualization of the sample influence ranking trends as additional interpreta- tion of why middle layers have better performance. Figures 27 and 28 present these ranking, where highest rank means highest influence score, for Roberta-Large and Qwen-2.5 1.5B, respectively, and three "kinds" of samples:
Top Ranked Noise Sample- the noisy sample with the highest influence across layers;
Avg Noise the averaged rank of all noisy samples per layer;
Avg Benign the averaged rank of "clean" samples per layer.
Trnehn
Casine
DataInf
ONLI
700
1000
100
1000
MRCC
1000
100
SS12
8
O0P
100
COLA
2000
2000
2000
400
400
4000
MNLI
L BAAABRA
RITE
1000
1000
2000
STSB
Benign
Top Ranked Sample
Figure 27: Roberta-Large sample influence rank variation with layer. Mean and 95% confidence intervals are presented for 10 random seeds.
Conclusion. We observe that the average noise influence rank drops with layer, i.e. noise becomes less influential. At the same time, the benign samples maintain the influence rank almost at same level. For Roberta-Large. the biggest difference between average ranks is observed at the last influence layers. This corresponds to the downstream performance maximum (Figure 6 where the best accuracy is also observed on later attention layers. Similarly, for Qwen-2.5 1.5B, the noise average rank has minimum on layers 15-20, where the model has also the best downstream performance (Table
32

---

Published as a conference paper at ICLR 2026
At the same time, for Qwen, in many cases, the top ranked noisy sample maintain high values of influence when the average noise rank has minimum, signifying the presence of the noise outliers that may affect the performance.
Tracnn
Cosine
Datalof
1000
1000
1000
ONLI
3000
100
2000
15
15
16
3040
200
MRCCC
21660
SS12
4000
4000
4000
1000
4000
100
1000
COLA
200
2000
R
H
8 71
15
8
71
100
MNLI
15
65
1000
S1SB
2004
200
2000
Avg Benign
Avg Noiee Rank
Top Ranked Noise Sample
Figure 28: Qwen-2.5 1.5B sample influence rank variation with layer. Mean and 95% confidence intervals are presented for 10 random seeds.
33

---

Published as a conference paper at ICLR 2026
L THE EFFECT OF VARYING THE NUMBER OF VOTES IN POSITIONAL VOTING
Formula TO contains the hyperparameter k, the number of votes that each test sample assigns to training samples. The reasonable question is how the noise detection rate changes with and what would correspond to the best performance? The following figure 22 presents the NDR for in range [10, 100) with step 10, for different layers and per NN module of Qwen-2.5 1.5B. The NDR value averaged across datasets and random seeds.
TracIn
Cosine
Datainf
Value B
40
15
15
-0
Layer
10
100
100
&
100
Query B
Layer
30
20
Lay
Wee 26
90
Wire
111
Value A
50
50
40
20
Layer
100
N.
100
2.
100
Query A
44
10
1
45
60
90
100
10
10
Late
Wire d.
40
60
80
10
Figure 29: Qwen-2.5 1.5B NDR variation for different Vote i and layer number.
Conclusion. NDR values show a clear dependence the choice of k. The range E [10,50] yields the strongest performance for most settings. For DataInf and TracIn, NDR decreases monotonically beyond this range. suggesting reduced discriminative power at larger subset sizes. Cosine shows the opposite pattern, reaching its minimum at & z 50 and improving for larger k. We use A = 30 (filtering threshold) in all main experiments. Determining how the optimal scales with the level of injected noise remains an open question.
34

---

Published as a conference paper at ICLR 2026
M LLAMA-3.2 1B INFLUENCE SCORE PERFORMANCE DISCUSSION
In Section 5.2 we compare the training data filtering by influence attribution with random baseline. We found, that particularly for LLaMA-3.2 IB, the influence functions fail outperform random filtering. In this section, we verify if the best by NDR module (Figure 15 LoRA B attached to value projection, outperforms the baseline.
The artifacts observed for -L.MA-3.2-- IB likely arise from its reduced depth (16 attention layers) relative to larger models best-performing layers across architectures indicates that reliable influence attribution requires not only well-chosen probes but also effective layer localization. This layer dependence parallels findings in other domains where the success of model interventions hinges on identifying the most relevant layers.
24, Qwen-2.5 1B 28, Mistral 7B 32)). The variation in
The following table presents the accuracy on GLUE datasets after influence filtering by Value B score of 8th and 9th attention layers (best according to NDR). Results include Mean, Rank, and Vote aggregations.
Table 7: Llama-3.2 1B downstream performance of 8th and 9th attention layer influence scores measured for LoRA modules B that are attached to value projections.
Method
Agg
Layer
Rank
ONLI
MRPC
SST2
QQP
COLA
MNLI
RTE
STSB
Full
1.20
1 0.30
10=00
00
15 1.9
10:00
10:00
10000
2.T=
10
6.0
Cosine
Vote
v-b-y
6.80 1 4.10
6.
8.
12.8 2.2
5.123
4.6 13
7.3
6.6
331
Cosine
Vote
vib-8
6.90 + 4.40
5.2
11.2
12.0
2.
6.4
5.2
7
2,8 1.0
Datalaf
Vote
v.b-8
9.20
+3.1+49
7.1
9.
80
6.4
11.6+34
10.6
7.0
Random
9.40
6.00
833
10.6
13.3 5.1
38122
682
14.5
14.4 14.4-66
36
Datalaf
Vote
vib-9
9.50 4: 5.20
5.5 4..0
8.8
4.
$.2
15.0
110
9.
Tracin
Vote
v.b.8
9.90
3.
7.66666
4.6
9.4
2.
+1.8
11.2
8.
Cosine
Rank
v-b-8
10.40 =: 5.10
6.6
+
8.1
+.22
25
4.
922
1.7
Cosine
Rank
v-b-9
10.60 + 5,00
8.8 4.4
6.8 4.1
12
12.4 +.2
7.6++41
84-47
+..4
Traeln
Cosine
Mean
v-b-8
10.70 =: 5.20
5.6
45.0
5.3
3.3
100
9.5
74
9.2
Datalnf
Mean
v-b-8
11.10 + 5.60
4..4+44.
+2.6
13.6-61
6.9
8.8 +52
12.6
9.1-67
1188
Cosine
622
Datalaf
Rank
v-b-8
11.70 1: 4,60
4.4
4..4
10.6
16.4 1.6
162 2.5
7.
13
10.1 3.4
Tracnn
Runk
v-b-8
12.70 = 480
10.7
1222
12.0=38
17.0
16.4
7.0
14.0=33
1,.6
Thaen
16.6=34
80
54
10.4660
14.6
Datalaf
Rank
v-b-9
12,80
4,50
10.8 4.9
11.7
124-49
1.71
158
8.4
12.6500
15.2 130
Datalaf
Mean
vib-9
13.10
= 3.60
6..6
12
16.0=
4.
10.6 4.9
16.2
13.2
16.0
Traela
v-b-9
122-59
1722
15.2
Tracln
Mean
v-b.9
14.00 = 5,30
+5.6
11.2
1522
12.6
13.0
178
16.0
17.0
We observe that Cosine scores (gradient similarities) outperform Random with Vote aggregation (k 20) selected modules. The differences are statistically significant. Wilcoxon test p-value is 0.0072 for configuration (Cosine, Vote, v-b-9) and Random; p-value is 0.0085 for (Cosine, Vote, v-b-8) and Random. At the same time, (DataInf, Vote, v-b-8) and Random have the p-value 0.47, signifying that the null hypothesis of their similarity cannot be rejected. At the same time, Mean aggregation cannot outperform the baseline even the best NDR layers.
Conclusion. With the new Vote aggregation strategy, influence scoring outperforms the baselines on locations that perform best according to NDR. Our practical recommendation for LLaMA-3.2-11 and consistently across the evaluated models, is to rely on influence scores computed on the LoRA value projection modules. These weights yield strong NDR performance also on autoregressive datasets (Appendix J).
35


Reviews:
[{"review_id": "KJAIY0jwsw", "reviewer": "Reviewer_xbbS", "summary": "The paper studies which LLM layers are informative for sample-level influence and proposes simple cross-layer aggregation (Rank/Vote) to combine per-layer scores. It further uses no-retrain proxies (e.g., NDR/AUC) to pre-screen configurations across models and tasks.", "questions": "Same with weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Addresses a practical question (which layers to use and how to aggregate) with simple, general aggregation operators (Rank/Vote).\n- Provides broad empirical evaluation across models/tasks and includes no-retrain proxies (NDR/AUC) that are useful in practice.", "weaknesses": "- **Related-work positioning should be strengthened.** Add a concise paragraph clarifying scope vs. **knowledge editing** (e.g., ROME, MEND, MEMIT):  Explicitly discuss the differences and connections in terms of **“where” (layers/locations)** and **“how” (locality vs. cross-layer aggregation)**. **No new experiments are required**—a brief positioning and citations will suffice.\n\n- **Novelty is relatively simple.** (Aggregation is straightforward; theory is light.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NCfb9uwGPd", "reviewer": "Reviewer_UsFs", "summary": "This paper shows that middle attention layers, not the first embedding layers, are most effective for estimating data influence in LLMs. It proposes new aggregation methods and a Noise Detection Rate metric, achieving better and more reliable influence estimation across multiple models and datasets.", "questions": "How sensitive is the positional voting approach to k?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is clearly written and well organized, making it pleasant and easy to follow.\n\n2. The work challenges the established conclusion [1] that the embedding layer is the most informative for LLM data influence estimation. The authors provide both theoretical and empirical analyses showing that the \"gradient cancellation effect\" can be unreliable in practice, and offer a counterexample (Theorem 5.1). This contributes a fresh perspective and theoretical insight to the field of LLM interpretability.\n\n3. The motivation is clearly presented and supported by experiments. The proposed method is conceptually simple, easy to implement, and practically useful.\n\n4. The experimental setup is comprehensive, covering multiple datasets and LLM architectures. It also evaluates several influence estimation methods, which strengthens the empirical analysis.\n\n5. The authors have provided open-source code to facilitate reproducibility, which is highly appreciated.\n\n```\n[1] Yeh, Chih-Kuan, et al. \"First is better than last for language data influence.\" Advances in Neural Information Processing Systems 35 (2022): 32285-32298.\n```", "weaknesses": "1. The current baselines are mostly classical. Comparing with more recent influence estimation approaches would make the study more comprehensive and convincing.\n\n2. Some recent studies [2] restrict gradient computation to specific layers for efficiency reasons. Computing influence across all layers in large LLMs could be computationally expensive. A discussion or quantitative analysis of the computational cost of the proposed approach would strengthen the paper.\n\n3. The current analysis primarily reports overall performance metrics. Including visualizations that illustrate how influence rankings of individual samples differ across layers would provide valuable interpretability.\n\n4. The intuition or mechanism explaining why middle layers perform better remains somewhat underexplored. A deeper discussion would be helpful.\n\n5. The paper could also benefit from a more detailed analysis of how architectural differences between LLMs affect the observed influence patterns and performance differences.\n\n```\n[2] Chhabra, Anshuman, et al. \"Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Models.\" International Conference on Machine Learning (2025).\n```", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s3wtxfKJbZ", "reviewer": "Reviewer_B3C8", "summary": "This paper challenges the assumption that embedding layers are best for computing influence functions in LLMs, showing that middle attention layers often perform better and that the \"cancellation effect\" metric is unreliable. The authors propose novel aggregation methods (Rank and Vote) that outperform standard averaging, and introduce proxy metrics (Noise Detection Rate and AUC) to evaluate influence functions without costly retraining, demonstrating through experiments on GLUE benchmarks across multiple LLMs that their approaches significantly improve detection of detrimental training samples.", "questions": "1. How do the proposed methods perform on real-world noise beyond synthetic uniform label flipping?\n\n2. Why do influence functions fail on LLaMA-3.2 1B, and how to select layers given different LLM architectures?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper provides both theoretical (Theorem 5.1) and empirical evidence challenging prior assumptions about optimal layers for influence estimation.\n\n2. The paper introduces well-motivated aggregation strategies (Rank and Vote) that outperform standard averaging, revealing layer-specific behaviors and improving influence estimation performance.", "weaknesses": "**1. Limited Evaluation Setting**: The experiments rely solely on synthetically injected label noise (20% uniform flipping) on GLUE benchmarks, which may not reflect real-world data quality issues.\n\n**2. Inconsistent Results Across Models**: The findings show notable inconsistencies, particularly for LLaMA-3.2 1B where influence functions fail to outperform random filtering, and the best-performing layers vary across models, suggesting the conclusions may not generalize.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iSEcha4VBT", "reviewer": "Reviewer_z9e9", "summary": "In this work, the authors study one particular problem in influence function approximation, i.e., which layer to focus on. They invalidated the existing claim that the first layer is the best and proposed some method to select the layer (based on two newly proposed metrics NDR and AUC). Multiple empirical studies are conducted to answer 4 research questions.", "questions": "How do you justify the relevance of your work in the presence of the recent observation made in \"Do influence functions work for LLMs\"?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "On the positive side, I enjoy reading the draft for the following reasons. \n\nFirst, the problem of identifying influential samples is an interesting and important one and in fact existing approaches are less than satisfactory. \n\nSecond, some of the results are interesting and relevant, such as those for answering RQ1 and RQ2 (the empirical part).", "weaknesses": "On the less positive side, the draft can be improved from the following aspects.\n\nFirst, the authors seem to be unaware of some recent work questioning the usefulness of the influence function for LLMs (“Do Influence Functions Work on Large Language Models?”). If we were to trust the empirical results (which is not unlike some of your observations) and reasoning in that paper, there is in fact little point in patching this existing influence function estimation method. Kindly discuss. \n\nSecond, some of the empirical results and conjectures are rushed and may not be reliable from what I read, given the many variables for such studies, such as the data distribution and the optimization algorithm and so on. \n\nThe following are some detailed comments.\n\nSection 2: RELATED WORKS\n\nComment: Some important recent work is missing such as “Do Influence Functions Work on Large Language Models?” (EMNLP 2025).\n\nPage 4: “Additionally, our proposed Rank method ignores incorrectly predicted validation samples …”\n\nComment: Kindly explain why.\n\nPage 5: “[RQ4]. How reliably can … ”\n\nComment: Without first introducing what NDR is, stating this RQ here seems not useful for the readers.\n\nPage 7: Figure 2\n\nComment: How reliable are these results given the many details involved in this kind of experiment (such as the data distribution, the performance of the optimization algorithm)?\n\nPage 7: “For Llama, influence functions perform worst among all settings: none of the\nconfigurations surpass the uniform removal baseline …”\n\nComment: This perhaps questions the usefulness of influence functions for LLMs fundamentally. \n\nPage 8: “DataInf and Cosine frequently outperform TracIn, particularly on early–middle attention layers of deeper models, suggesting that these layers encode representations that are most informative for noise filtering.”\n\nComment: Given the limited scale of the experiments and the complexity of the problem, it is not clear how strong this empirical evidence is. \n\nPage 9: “We validate the hypothesis that NDR and AUC can serve as reliable proxy metrics\nfor influence estimation by …”\n\nComment: Kindly explain why intuitively these are good estimations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)