LayerIF: Estimating Layer Quality for Large Language Models using Influence Functions
Authors: Hadi Askari
Shivanshu Gupta
Fei Wang
Anshuman Chhabra
Muhao Chen
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Influence Functions
Layer Training Quality Estimation
Model Pruning
Mixture of Experts
Primary area: deep_learning
TL;DR: We propose LayerIF, a framework that employs Influence Functions for LLM layer quality estimation. Our method captures task-specific layer importance and improves both expert allocation in LoRA-MoE and layer-wise sparsity distribution in LLM pruning.

Abstract: Pretrained Large Language Models (LLMs) achieve strong performance across a wide range of tasks, yet exhibit substantial variability in the various layers' training quality with respect to specific downstream applications, limiting their downstream performance. It is therefore critical to estimate layer-wise training quality in a manner that accounts for both model architecture and training data. However, existing approaches predominantly rely on model-centric heuristics (such as spectral statistics, outlier detection, or uniform allocation) while overlooking the influence of data. To address these limitations, we propose **LayerIF**, a data-driven framework that leverages *Influence Functions* to quantify the training quality of individual layers in a principled and task-sensitive manner. By isolating each layer's gradients and measuring the sensitivity of the validation loss to training examples by computing layer-wise influences, we derive data-driven estimates of layer importance. Notably, our method produces *task-specific* layer importance estimates for the *same* LLM, revealing how layers specialize for different test-time evaluation tasks. We demonstrate the utility of our scores by leveraging them for two downstream applications: (a) expert allocation in LoRA-MoE architectures and (b) layer-wise sparsity distribution for LLM pruning. Experiments across multiple LLM architectures demonstrate that our model-agnostic, influence-guided allocation leads to consistent gains in task performance.

LAYERIF: Estimating Layer Quality for Large Language Models using Influence Functions
Hadi Askarit, Shivanshu Gupta Fei Wang Il, Anshuman Chhabra Muhao Chen' University of California, Davis University of California, Irvine University of Southern California UUniversity of South Florida
Abstract
Pretrained Large Language Models (LLMs) achieve strong performance across a wide range of tasks, yet exhibit substantial variability in the various layers' training quality with respect specific downstream applications, limiting their downstream performance. It is therefore critical to estimate layer-wise training quality in manner that accounts for both model architecture and training data. However, existing approaches predominantly rely on model-centric heuristics (such as spectral statistics, outlier detection, or uniform allocation) while overlooking the influence of data. To address these limitations, we propose LAYERIF, a data- driven framework that leverages Influence Functions to quantify the training quality of individual layers in principled and task-sensitive manner. By isolating each layer's gradients and measuring the sensitivity of the validation loss training examples by computing layer-wise influences, we derive data-driven estimates of layer importance. Notably, our method produces task-specific layer importance estimates for the same L.M,, revealing how layers specialize for different test-time evaluation tasks. We demonstrate the utility of our scores by leveraging them for two downstream applications: (a) expert allocation in LoRA-MoF architectures and (b) layer-wise sparsity distribution for LLM pruning. Experiments across multiple LLM architectures demonstrate that our model-agnostic, influence-guided allocation leads to consistent gains in task performance.
Introduction
Deep neural networks, particularly Large Language Models (LLMs), have become increasingly powerful, achieving impressive performance across numerous natural language processing tasks such question answering 21, natural language inference 31, commonsense reasoning 45,, and code generation [6] Recent models have demonstrated strong generalization across these benchmarks, such Mistral 7, Gemma R GPT-4 9 DeepSeek [T0]. LLaMA [1I, and Qwen [12 model families. However, despite their effectiveness, these models exhibit significant internal variability, with layers differing substantially in terms of training quality and contribution to overall model performance [3316]. Recent diagnostic tools, such as layer-wise adaptive learning rates [17] [8] and pruning strategies [19 informed by empirical spectral densities (ESDs), have highlighted that not all layers within deep model are equally well-trained or consequential to its final performance.
There is broad effort in the ML community interpret and explain the behavior of these complex models. Many approaches take a model-centric perspective, examining how the model's learned weights effect predictions (20 19]. Other methods use activations, attention patterns to deduce internal mechanisms 21-241 In contrast Influence Functions (IFs) [25 26 offer a more explicitly data-centric perspective by quantifying the effect of individual training points model's loss landscape. Rather than focusing on internal representations, Influence Functions (IFs) measure how model's output would change if given training example was perturbed removed, effectively
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Dataset
Layer
Expert Allocation Mapping Function
Layer
Layer
M2
Layer
M2
Layerwise Gradients
Experts Allocated
Layer
Layer
Layer
Layer 1
V,
Layerwise
V2
Layer
1556
Scores
V
Layer
0:46
0.48
Layer
Layer Sparsity Mapping Function
Model-only Information heuristics
Layer
0.47
Pre-trained Model
Layer Sparsity
Green Arrows: LayeriF workflow Red Pre-existing methods
Figure We present an overview of the LAYERIF pipeline via the green arrows. We quantify per-layer quality in pretrained LLM using Influence Functions (IFs), demonstrating that the same model produces distinct layer-wise scores across different datasets, revealing dataset-specific specialization. The pipeline begins by extracting gradients at each Transformer block of the LLM, computed separately for each dataset. These gradients and the dataset are then used to estimate layer-wise IF scores that serve data-driven proxies for layer quality. To demonstrate the utility of these scores, we consider two downstream applications:
(a) the allocation of optimal experts per layer in LoRA-MoE architecture, and (b) the computation of structured layer-wise sparsity ratios for model pruning. For each task, we apply a dedicated mapping function, to transform the raw IF scores into task-specifie layer importance measures. On the contrary, pre-existing methods only use model-only information or heuristics to compute these metrics as indicated by the red arrows. Figure iis is conceptual illustration contrasting our method with the baselines; the shown values illustrative, not experimental.
tracing the influence of training data on the model's predictions. They have proven effective in a
range of downstream machine learning tasks, including mislabeled data detection 27 29) optimal
subset selection 30 (0-32] 32), model interpretation 33 35], data attribution
[6], data valuation [3]
in-context learning 388 and analyzing model biases 39 40).
Since IF's offer an empirical approach to measure the impact of training samples on model predictions, they present a promising avenue for assessing and quantifying this variability in layer quality. Surprisingly, despite their clear potential, Influence Functions have not yet been formally applied systematically formulated for evaluating the quality or importance of individual layers within deep neural networks, particularly LLMss. In this work, we address this gap by introducing a novel framework, LAYERIF. that leverages Influence Functions rigorously quantify and interpret the training efficacy of individual model layers. By systematically analyzing each layer's gradient information through IFs, we obtain actionable insights into layer importance.
key advantage of using IF' for estimating layer quality their ability to combine perspectives from both the data- and model-level. For instance, the same model (if trained to different number of epochs) may exhibit wase specialization that varies depending on the dataset or task, and IFs capture this variability. Specifically, as part of our LAYERIF framework (illustrated in Fig. [1] we hypothesize that since IF's measure the sensitivity of model loss to individual training points and the loss of well-trained layers should show smaller changes, layer-wise IF scores serve proxy for layer quality. These task-specific layer quality estimates can then be mapped to downstream settings using corresponding transformation functions.
We evaluate our LAYERIF framework on two practical downstream settings: (a) expert allocation in Mixture-of-Exxe -xpeert (MoE) architectures, and (b) layer sparsity allocation for model pruning. In recent work, MoE. models have emerged as a compelling strategy for scaling L.LMs efficiently by activating only small subset of specialized subnetworks ("experts")) per input token [4] 42

---

While several srrametee-efffiient fine-tuning methods (e.g,, LoRA) have been extended to MoE settings by training multiple low-rank adapters (LoRA experts) with learned routing mechanisms 45 typically, the same number of experts are allocated to every Transformer block. Recent studies 46 47 have questioned this uniform allocation, finding that expert redundancy and routing overfitting can degrade performance. However, the alternatives proposed by them rely group-based heuristics or model-driven ablations. In contrast, we propose to use data- and model-driven IF-based layer quality estimates to inform expert allocation.
Similarly, for the setting of layer sparsity allocation for LLM pruning, the traditional approach of imposing a uniform sparsity ratio per layer limits the ability to push toward greater global sparsity without sacrificing performance [48] 49 Recent work on more principled approaches for allocating
sparsity across layers includes OWL 20 and AlphaPruning [9], which perform non non-uniform sparsity allocation using metrics derived from weight statistics (e.g., outlier activations or heavy-tailed spectral properties). These approaches rely solely on model weights and heuristics, without leveraging training data. Our method offers a complementary. data-driven alternative: we use LAYERIF to identify layers with lower aggregate influence and prune them more aggressively, while preserving better trained layers. In both pruning and MoE routing, our results show that LAYER[F-informed decisions yield improved accuracy trade-off's over uniform, heuristic, solely model-driven baselines. Moreover, our method applies to both pretrained and fine-tuned models and is agnostic to the model architecture.
Contributions. In sum, our work advances layer quality estimation in LLMs through the following key contributions and findings:
To the best of our knowledge, are the first work to show that Influence Functions (IFs) can be effectively used to analyze LLM layer quality using our LAYERIF framework.
We study the effectiveness of LAYERIF in two downstream tasks: (a) expert allocation in Mixture- of of-Experts (MoE) architectures, and (b) layer sparsity allocation for model pruning,
For expert allocation in Mixture-of -fterr--of-pppttttt (MoE) architectures we conduct experiments Mistral- 1b-v0.1 and Gemma-7b 7b by computing the post fine-tuning accuracy over several GLUE (CoLA, MRPC) and QA (CommonsenseQ. OpenbookQ, TextScience() datasets. This leads percentage increase of 1.61% % over the best performing baseline.
For layer sparsity allocation for model pruninng we conduct experiments on Mistral-7b-v0.1 and Gemma-7b on several by computing the post-pruning zero shot accuracy on several NLP tasks namely BoolQ, Hellaswag, Winogrande, RTE, OpenbookQA, ARC Easy and ARC Challenge. This leads to 0.90% increase over the next best baseline.
2 Related Works
Layer Quality Estimation. Prior work has examined the representational differences that emerge across layers within deep neural networks [50 Several probing techniques have been deployed to capture the semantics of internal layer representations 53 54]. Additionally, (55) employs random matrix theory to analyze the weight matrices of Deep Neural Networks. They show that these matrices exhibit heavy tail empirical spectral density (ESD) and decay coefficient of this ESD, Pl.Aapha Hill, can effectively gauge layer quality. More recent work has used Shapley Values (56) to identify which layers contribute more to the model's capabilities [57]. Other research has found that the middle layers in LLMS provide stronger features for downstream tasks [58]
Model Pruning and Layer-wise Sparsity Budgets. Pruning is long-established technique in neural network optimization, aimed at reducing the size of a trained model by eliminating redundant or non- essential parameters. [59 60). Modern works have used the similarity between the representations at different layers to identify the optimal layers to prune, finding that the deepest layers can be pruned with minimal degradation (61. [62] Compared the change cosine similarity of the input and output
representations of a layer to prune the model. Similarly, [63] prunes model structures based on
gradient information, followed by LoRA finetuning, [48] prunes based on the Hessian inverse, and
49 uses weights and activations of layers to introduce sparsity. Computing layer-wise sparsity ratios
has been commonly used in older pruning methods (44 and quantization approaches (65). [19 uses
the aforementioned PI. Alpha Hill metric (55) to calculate layer-wise sparsity ratios for pruning. Finally, (00 proposes a non-uniform, layer-specific sparsity scheme informed by the distribution of outlier activations across layers.
3

---

Allocating Mixture of Experts. Research in combining LoRA based finetuning 666 and Mixture of Experts Models 4 boost performance is an actively growing field. Several works have proposed uniform strategies for allocating experts in LoRA-MoF settings [67-7] Other works have looked into heuristics-based group methods [46] and Heavy-Tailed Reglaargulaiittioon Theory [47] based adaptive layer-wise methods to allocate experts. Other research directions aim to improve the composability of LoRA for cross-task generalization (72).
Influence Functions. Classical influence function approaches seek estimate the effect of training samples on the model's loss, either by leave-one-out retraining [26] 73/76) or through gradient-basee
approximation 2
78
35
79
However, most of these methods are inapplicable for large
language models due to computational inefficiency or convexity assumptions [80 29] More recently,
several influence function methods have been proposed for LLMs, such as DataInf 81 Arnoldi itera-
tion [82), alternative Hessian forms [3] [83, self inffuinff [84] 85), ensemble approaches [36 86). as
well as less performant but compute-efficient Hessian-frre fesse approaches (29 28 32
87. is important
to note that our LAYERIF framework is agnostic to the choice of influence function, so new advance-
ments made in this area can be directly utilized as part of our layer quality estimation approach.
Proposed Approach
In this section will first define the preliminaries and notation that we will use and then formally define our method.
3.1 Preliminaries and Notation
We clarify that in this work, the term layer refers specifically to a Transformer block, which comprises multiple weight matrices, including those associated with the attention mechanism and the mip projection layers. We will now describe the preliminaries for Influence Functions and then for the two tasks we consider this work: (a) allocating mixture of experts for LoRA finetuning and (b) layer sparsity allocation for model pruning.
Influence Functions. Let the training dataset be Dirain = {zi)== and the validation dataset be D = ( with fo representing a pretrained large language model with parameters E θ trained using loss L. The Influence Funetion (IF) quantifies the sensitivity of model parameters to individual training examples [25] 26 88) Formally, it characterizes how the optimal parameter estimate 0 changes when specific training point is infinitesimally up-weighted in the empirical risk objective. Specifically, for training point indexed by k E [ni] and perturbed by infinitesimal weight E R. we can obtain perturbed model parameters as (e).. Then, assuming the loss is twice differentiable, (27) showed that the influence of the k-th training point on the learned parameters 0* can be derived by taking the limit of the perturbed solution with respect to e, as + 0:
d0ck) To- (2x) de
c=0
Where (0) is the Hessian matrix. Subsequently, the influence of a training sample 2; E Ditain on the validation loss becomes:
I(x)) -
The influence function T((;)) measures the effect of an individual training example on the validation loss. Specifically,, it captures whether the sample 2; has beneficial or detrimental impact on the model's predictive performance. A larger negative/positive value of / (z,) indicates that the data point contributes to reduction/increase in the loss, thereby serving beneficial/detrimenttl example for model optimization.
LoRA-MoE. In the -ixture-of-Experts (MoB) paradigm, each Transformer layer is augmented with N parallel expert modules 42]. Given an input x E Rd, the MoE layer computes the output as: Wox Σn' (x(x) (x) where Wo is the pre-trained weight matrix, Ex(x) denotes the

---

output of the i-th expert, and (x) is the routing probability for expert i, typically produced by softmax over a trainable gating network: G(x) Softmax(xW, with W, g being the gating matrix. In practical implementations, only the top- K experts are selected to compute (x), and their outputs are aggregated with had-balancing losses to avoid expert collapse.
Layer Sparsity Allocation for Model Pruning. Given a pretrained Transformer-based language model with I, layers, we denote the set of prunable weights by W, and the model architecture by a tuple D (d1, d2..... ,....,,,,,, where de represents the number of prunable parameters in the i-th layer. The goal of layer-wise pruning is to remove a fraction of parameters in W while maintaining model performance. Rather than pruning uniformly across layers, a strategy is adopted allocate different pruning budgets based on layer quality.
We now define the process of selecting important layers based IF scores via LAYERIF. We then discuss how we can use LAYERIF for (a) allocating a mixture of experts for LoRA finetuning and (b) layer sparsity allocation for model pruning.
3.2 LAYERIF: Estimating Layer Quality via Influence Functions
To assess the relative quality of different layers in the model, we localize IF computation to each layer I. We denote the per-layer parameter vectors as g( for each layer I (I...., .). Let Vaco & and (0) be the gradient and Hessian restricted layer L. We define the layer-specific influence score of training point 24 as:
7 (2.)
m2 Σ
Vow (H))
V ges ((z;, 0)
For each layer I, we aggregate the influence scores across the training set, considering only positively influential samples after a sign inversion! [1 5(l) m (z,) > 0] (z:) where [[]] the indicator function. This yields a vector S E RN, where each element 5r()) captures the cumulative positive influence of training data on validation performance through layer 1.
Justification. We now provide some intuitive justification for why the LAYERIF framework can estimate layer quality with a high degree of accuracy. It is important to note that IF theory states that positive/negative influence scores for samples increase/decrease model loss and are hence, detrimental/bennefice to learning the task. In the same vein, to assess overall detriment/benefit to the end-task performance as a scalar, we can sum over all positive/negative influence sample scores. Moreover, by restricting influence to particular combination of training samples and layers we can obtain these scalar influence contributions for these layers. Since we do this layer-wise influence computation in LAYERIF for different layers but keep the training samples fixed, we essentially isolate the impact of different layers on end-task performance. Thus, LAYERIF can serve as an effective approach for estimating layer quality in data- and model-centric manner, unlike prior approaches. Layers with lower cumulative positive influence S(() exhibit less sensitivity to training data, indicating greater stability and training maturity. Conversely, higher scores suggest under- trained less generalizable layers. Finally, our layer quality scores obtained through LAYERIF can subsequently be used to guide pruning or resource allocation strategies that are dependent on accurate layer quality estimates, we show next.
3.3 Layer-wise Expert Allocation using LAYERIF
We now discuss how can our LAYERIF method to calculate the optimal distribution of fixed set of N parallel expert modules. To inform expert allocation, we evaluate the quality of each layer using LAYERIF, which measures the sensitivity of model predictions to individual training samples. We compute the IF matrix for each of the Transformer layers in an LLM on a train and validation set. To obtain a scalar score v; for each layer, we aggregate its corresponding IF values while excluding any training samples with negative influence, i.e., those that decrease the model's 's loss, from the final sum.
To map LAYERIF scores to expert allocations, we first invert the raw scores obtained for each layer. Since our IF values are inverted to reflect loss sensitivity, apply sign inversion to ensure that layers
'This sign inversion is done since it is more intuitive that positive influential samples are beneficial to the model.
5

---

with higher (i.e., less negative) IF scores remain lower in magnitude after transformation. Formally, for each layer i, we compute ö = - where = - Next, we apply a power transformation to modulate the sharpness of the allocation. Specifically, we raise the inverted values a thyerparaaeeter controlled exponent B > 0, yielding transformed scores é; a. This exponent controls the dispersion of the allocation: higher values of [3 amplify disparities between layers. We then scale the transformed values such that the total allocation, excluding a baseline of one expert per layer, matches the desired target sum T. Specifically, we compute f; ΣØ= 83 (T", where m is the number of
8
layers, and the floor of each fractional allocation fa is taken with a minimum of added: 8; = [fa]+ +1 This guarantees that each layer receives at least one expert. Due to flooring, the sum E; s, may be less than T. We compute the remaining allocation budget r = T Ee s; and distribute these remaining units to the r layers with the largest fractional remainders f; fil. ensuring the total sum constraint
satisfied: s; f s; + +, for top r layers with highest (f+ (f.l) This procedure produces final expert allocation vector S = [$1, $2,.... ..... Sm) such that E, = TT and s; 2 for all i.
For each module in layer i, let w E R denote the frozen pre-trained weight matrix. To construct the experts, we instantiate s; low-rank adaptation modules comprising trainable matrix pairs {A)", 1 15 where Ajr E: mR is initialized with random Gaussian weights and B E R=*x is initialized to zero. Here, r & min(m, n) denotes the rank of the adaptation [66].
A router st parameterized by a trainable weight matrix Wit E RaxN, dynamically assigns experts for a given input x. Following the MoLA framework [46], we adopt top- K routing strategy, where only the K most relevant experts are selected for computation. To encourage balanced expert utilization, we additionally incorporate a load balancing loss at each layer [89].
(SofinnSti 2 K),,
The routing process is formally defined as S
Σド "ipk(Sohmmx(WW""5 *)) where ()
denotes the normalized assignment weight for expert j. The final layer output h is computed as: h Wo' x + Σj= (z) BJ's xt where Wi' is pre-trained base weight matrix, and Aj. B3d are low-rank matrices parameterizing the j-th expert. Thus, the output 1 combines the base transformation of x with the aggregated contributions of the top- experts, each scaled by its corresponding assignment probability (z)) A step-by-step walkthrough of our algorithm is provided in Appendix C for greater clarity.
(z).
3.4 Layer Sparsity Allocation for Model Pruning via LAYERIF
We describe how our LAYERIF method can be applied to the problem of allocating layer sparsity bud- The motivating hypothesis is that the less well-trained layer is, the more it can be pruned as opposed to pruning each layer uniformly. We compute the IF scores for each Transformer layer in an LLM the positive influence contributions across all samples, resulting in a vector of layer quality scores (51, , 81.), where each 8; quantifies the total positive influence attributable to the i-th layer.
gets for model pruning 119 90
train and validation set. These scores are aggregated by summing
To ensure stable and meaningful sparsity allocation, we post-process, the raw layer-wise IF scores before applying the mapping function. First, we take the absolute value of each score, and normalize these values to the range [0, 1] using min-max normalization. This normalization step standardizes the range of IF magnitudes across layers. Next, we apply the Savitzky-Golay filter [91] to the normalized to smooth local fluctuations and reduce noise while preserving the overall trend in layer quality The smoothed scores are then used as input to the sparsity allocation function ó described below. This two-step normalization and smoothing process improves robustness and consistency in sparsity assignment, particularly in cases where the raw influence scores vary erratically between adjacent layers.
vector = ($1, ....... L
.......,,
To convert these quality scores into sparsity budgets, we build upon ideas in prior work (19] First, we
apply a normalized linear mapping øb RR_ + RR,,, namely Φ(3); Smaxe where Smai and Smax are the minimum and maximum elements in the vector s, and 61, C2 are tunable parameters that control the minimum and maximum sparsity levels per layer. The scaling factor is
1--Emin-- (e2 - e1) + +e)
Raw Influence Function (IF) scores at adjacent layers exhibit high- seehffreuueey noise due to stochastic Hessian approximations and mini batch gradient estimates. This variance is well documented by (34), whose layer-token heatmaps show similar oscillations.

---

Table 1: Accuracy across datasets using ALPHALORA, MoLA and LAYERIF variants. We report scores under three IF strategies: Top 25% positively influential samples, all positively influential samples (+ve), and all samples (top performers in bold).
MoLA
LAYERIF
Dataset
ALPHALORA
2468
5555
8642
+VE
ToP 25%
ALL
MRPC
75.30
82.02
80.64
80.70
82.49
82.63
81.63
Cola
82.07
82.26
85.14
84.56
83.60
83.80
84.47
Text Science Q
78.52
64.65
72.75
76.39
75.04
79.59
77.61
Common Q
82.06
80.91
79.93
81.00
80.26
80.75
80.34
Openbook Q
84.60
64.00
83.80
82.20
87.40
84.60
85,20
Average
80.51
74.77
80.45
80.97
81.76
82.27
81.85
computed to satisfy the global sparsity constraint S, such that E ((3) = E d, with d, denoting the number of prunable parameters in the i-th layer. This ensures that the total number of remaining parameters matches the target sparsity level. We then incorporate these layer- wise sparsity ratios in LLM pruning methods Magnitude [92]. Wanda [49 and SparseGPT [48] and evaluate zero-shot capabilities.
Experimental Results
We delineate our experimental setup for experiments and analysis, and discuss our dataset details, models used conduct the experiments, methods and basclines compare against.
Datasets and Base LLMs. For the layer-wise expert allocation in the LoRA-Mot application, we assess post-training zero-shot performance on two GLUE datasets and three commonsense reasoning tasks, following a protocol similar to that of Qing et al. 47). The datasets used include: MRPC [93] ScienceQA (95), Q 166. and OpenBookQA 97). For layer sparsity allocation for model pruning we again assess the post-pruning zero-shot performance of our models on several NLP tasks. Namely, BoolQ [98]. Hellaswag [99], Winogrande [00], ARC Easy and ARC Challenge [101, RTE [94], and OpenBookQA 197]. For both of the applications we conduct experiments on Mistral-7b-v0. 7 and Gemma-7b 7b g. All of our experiments run on 8xNVIDIA RTX 6000 Ada GPUs. All code and reproducibility configurations are provided in Appendix andd we provide further information on statistical significance of our results in
CoLA
94
Method Protocol and Baseline Configurationss For the layer-wise expert allocation application train our models for 3 epochs and then compare zero-shot accuracy on the datasets using a protocol similar to [47]. We experimented with 3 LAYERIF configurations: using all samples LAYERIF (ALL), using only positively influential training samples LAYERIF (+VE) and using the top 25 % of the most influential training samples for our layer-wise summation: LAYERIF (ToP 25 %)..
Additionally, we compare against ALPHALORA 47]. and 3 baselines from MoLA 46 MoLA- 2468, MoLA-5555 and MoLA- -662..
For layer sparsity allocation for model pruning, we apply the layer- sparsities derived from our LAYERIF method to traditional I.LM pruning methods such as Magnitude (92). Wanda [49], and SparseGPT [48). and compare zero-shot performance. All of these methods originally have uniform layer-wise sparsity, which we
baseline for comparison. We also compare with state-of-the-nn baselines ALPHAPRUNIN [19] and OWL (20). We prune our LLMs to 50% sparsity, since we can achieve strong accuracy while removing half of the model weights, hence maintaining the real-world utility of our method.
4.1 Results on Layer-wise Expert Allocation using LAYERIF
All LAYERIF configurations outperform baselines. Table [ presents accuracy across datasets for expert allocation methods, including ALPHALORA, three variants MoLA, and our proposed LFYERIF configurations. LAYERIF consistently outperforms prior approaches, achieving the highest average accuracy across datasets. Specifically, the best performing LAYERIF configuration had relative percentage increase of 1.61% % over the best performing baseline and the average of the
7

---

Table Comparison of average accuracies across different pruning strategies (Magnitude, Wanda, SparseGPT) and layer-wise sparsity allocation methods at 50% pruning on Mistral-7b-v0.1 (top performers bold).
Method
Magnitude
Wanda
SparseGPT
Average
UNIFORM
56.41
58.71
60,45
58.52
ALPHAARUNING
56.74
58.48
60.07
58.43
OWL
56.16
58.75
59.99
58.30
LAYERIF
56.89
58.94
58.81
LAYERIF configurations had a 3.52% percentage increase over the average other strong baselines. These results demonstrate that IF-based layer quality estimation provides more effective signal for expert allocation in LLMs.
Consistently strong performance across varying number of experts. In Figure we provide aver- age results for varying the number of experts. specif- ically 80, 160 and 224 experts on Mistral-7b-v0.1 1. MoLA(1234) and MoLA 10) are introduced as additional baselines, following the most perfor-
Aversse Score 196)
MOLAI123H
Layont180)
LayonFilso
mant pattern as claimed in the MoLA paper. We
NoLX200B100
can clearly see that LAYERIF clearly outperforms the baselines in all three settings by having an aver- age accuracy that is 3.64% percent increase over the second best average accuracy ALPHALORA. Ex- pert allocation results for Gemma-7b are provided Appendix
LayenFi224)
Exoets
182 Experts
224
Tolal Number of Experts
Figure Comparison between ALPIIALORA, MOLA and LAYERIF with varying number of total experts (80, 160, 224).
4.2 Results on Layer Sparsity Allocation for Model Pruning via LAYERIF
Superior performance on all pruning techniques. Table 2 clearly indicates that using LAYERIF for layer-wise sparsity allocation leads to performance gains over the baselines with 1.03% percentage increase over OWL and 0.90% increase over AL- PHAPRUNING in SparseGPT respectively. The re-
63
Accuracy
62
Uniform
61
AlphaPruning
sults reiterate the value of IF-bd layer quality estimation.
60 LayeriF
20
30
40
50
Robust performance across multiple pruning ra- tios. Figure demonstrates the consistently reliable performance of LAYERIF across multiple pruning ratios. LAYERIF consistently achieved the highest accuracy across all pruning ratios, while baseline methods exhibited greater performance variability, frequently changing ranks across sparsity levels. On average, LAYERIF outperformed the next best method, OWL, by 0.37% percentage increase. Moreover, we provide additional layer-wiss sparsity allocation results for Gemma-77 7b in Appendix E and results with and without the smoothing of IF values Appendix
Sparsity 1%)
Figure 3: Mean accuracy across sparsity lev- els (20%-50%) for Mistral-7b-v0.1 pruned us- ing SparseGPT.
Discussion
In this section, we provide discussion of the components the LAYERIF pipeline and elaborate on key design choices and ablation studies conducted part of experiments.
Layer-witt Allocation Differences. We show comparison between the number of experts allocated in LAYERIF versus other baselines in Figure Unlike ALPHALLOA and MoL., which apply the same allocations to every test task, LAYERIF dynamically adapts the allocations based on data-driven

---

LayeriF
LayerIF Cola
Text Sete_cc
Method
LayeriF Common Q
LayeriF Openbaok_0
AlphaLora
M,LA)224,6.8)
16
24
Layer Number
Figure 4: Heatmap showing expert allocations across Transformer layers for Mistral-7B-vO. at total of 160 experts, comparing our dataset-specifif approach LAYERIF to ALPHALORA and MoLA. Unlike ALPHALORA and MoLA, which apply the same allocations to every test task, LAYERIF adapts allocations based on the dataset. This is reflected in the diverse allocation patterss across the various LAYER rows as darker shades indicate higher expert allocation to that particular layer.
Table 3: Comparison of accuracies across different pruning strategies (Magnitude, Wanda, SparseGPT) between LAYERIF and reversed allocation at 70% pruning on Mistral-7b-v0.1.
Method
Magnitude Wanda SparseGPT Average
LAYERIF
33.44
32.50
41.14 38.45
35.69
Reversed Allocation
33.10
32.02
34.52
task-affinity estimates. This clearly indicates the unique benefit of our approach, as tune the allocation based on how much task affinity a model has, as opposed to prior work, which offers allocation for all datasets based model-only information best.
Interpretation of Layer Sensitivity. Influence Functions quantify how small perturbations in training data affect the model's loss. this view, layers that are well-trained and have effectively converged tend to exhibit lower sensitivity to such perturbations, as reflected by smaller IF magnitudes. To further examine this relationship, we conduct an ablation in which we invert our allocation strategy, assigning higher pruning ratios to layers with lower IF sensitivity. Under 70% sparsity budget on MISTRAL-7B-VO. 1, this reversed allocation leads to consistently worse downstream performance, reinforcing that layers with lower IF sensitivity are better trained and should be preserved more aggressively, as can be seen in Table
Guidelines for Influence Sample Selection. We provide a empirical perspective on how select the subset of influence samples used to estimate layer quality. Throughout the paper, we aggregate only positive-influenc samples (+VE) by default, as they reflect training points that reduce validation loss and thus carry constructive learning signal. In Section 4, we additionally examine two variants: (ALL) which includes all samples regardless of sign, and (TOP-25% which retains the top quartile of positive scores by magnitude. Empirically, the TOP-25% 25% variant achieves the highest average performance across datasets, suggesting is a strong default in practice.
From theoretical standpoint, this selection serves as noise-filtering mechanism. Because IF estimates computed from stochastic Hessian and gradient approximations, small-magnitude scores often correspond to spurious or weak training-validation interactions. Retaining only the upper tail of positive influences increases the signal-to-noise ratio. Prior studies have also shown that IF values in deep models are heavy-tailed [34], implying that small fraction of training samples account for most of the total positive influence. Hence, focusing this high-influence subset isolates the most informative training signals.
In practice, positive IF values in descending order and identifying the elbow point where marginal contribution drops sharply provides data-driven threshold. This adaptive thresholding remains exciting direction for future work.
determine the cutoff ratio using simple cumulative influence curve: sorting
Per Matrix versus Per Block. We compute the IF scores on a per-block basis within each Transformer layer. That is to say, the IF scores are aggregated across the attention and MLP heads

---

of a given layer, and the average score is uniformly applied to all associated weights, rather than computing separate scores for each component. This design choice follows the precedent set by prior work [99 20 To empirically assess the impact of this decision, we conducted an ablation study using MA2-7B [1] at 70% sparsity across all three pruning methods. We observed that computing scores on a per-block basis yielded 2.14% average improvement in accuracy, thereby corroborating the effectiveness of this approach.
Computational Complexity. One limitation of IF-based methods in general (at test-time) their high computational complexity, primarily due to the Hessian inversion. However, optimizing this compute time an active area of research 22 82 and our LAYERIF framework can employ any efficient IF method proposed in the future. Additionally, it is important to note that our framework not for use during inference. That is, since our framework is applied during model training, the computational overhead of IF is a minuscule fraction of the time it takes to pretrain LLMs. Detailed wall-clock times and memory consumption for varying model and dataset sizes are presented in Appendix
Comparison with Alternative IF Methods. One class of alternative IF methods are Hessian- free methods like TracIn (28) While Hessian- free IF methods are computationally more efficient, they generally lead drop in performance [38 29] To assess if Hessian-free methods such as TracIn (28) can be used for layer quality influence estimation, we compute the layer-wise expert allocations for each dataset from our main experiments using TracIn as the IF method in LAYERIF and compare these allocations with those derived from the Hessian-bbased method, DataInf [8]] (employed in our main ex- periments). We compute the Spearman's correlation coefficient and present our findings in Table We find that Hessian-free free methods fail to capture meaningful information about layer-wise training quality as there is little to no agreement in the resulting layer allocations across datasets when employing TracIn compared to Datalnf LAYERIF. Another class of alternative
Table 4: Measuring Spear- man's correlation coefficient between Hessian-bart Hessian-free methods when employed in LAYERIF.
Dataset
Correlation
MRPC
-0,22
Cola
0.44
Openbook
0.09
Text Science Q
-0.69
Common Q 0.12
IF methods are Gauss-Newton Hessians like the EK-FAC method [4
The main bottleneck with
the Gauss-Newton Hessian is the time required for IF computation. our experiments, the Gauss Newton Hessian roughly 160x slower than using the DataInf approximation, making Datalnf the only practical choice for our lcage experiments.
6 Conclusion
In this work, we introduced LAYERIF, novel framework that leverages Influence Functions (IFs) to quantify layer quality in large language models (LLMs). Departing from prior model-centric information and heuristics, our approach provides a data-informed perspective by capturing the sensitivity of model loss to individual training examples at the layer level. To validate the utility of our layer-wise IF estimates, applied them to two practical downstream tasks: expert allocation in Mixture-of-Expers (MoE) architectures and layer sparsity allocation for model pruning. Our method consistently outperformed the previously best approaches, achieving 1.61% % improvement in LoRA-MoE and a 0.90% gain in zero-shot accuracy after pruning. These results confirm that IF-based layer quality estimation can guide more effective structural adaptations in LLMs. Beyond its empirical gains, LAYERIF offers a general, training-date daaeeddttaaaaaaa framework for interpreting and optimizing deep models, applicable across architectures and deployment scenarios.
Acknowledgment
We appreciate the reviewers for their insightful comments and suggestions, and for helping strengthen work. Hadi Askari and Muhao Chen were supported by the NSF of the United States Grants ITE 2333736, OAC 2531126, and the Amazon Nova Trusted Al Prize. Anshuman Chhabra was supported by the USF Faculty Startup Fund.
10

---

References
[1] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP, pages 2383-2392.
Association for Computational Linguistics, 2016.
[2] Zhilin Yang, Peng Qi, Saizheng Zhang. Yoshua Bengio, William W. Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. 2018. URL http:::/ htmp::://arxiie /wwpxaaxivvv..///.//.a.//aaaa.a/ org/ http///b///190//. 09600
[3] Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad- covadd-overageeeo challenge for sentence understanding through inference. In Proceedings of NAACL-HLT, 2018.
challnnee corpus
[4] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA:: A question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings the 2019 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pages 4 4149-4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. (0.11853//11N19-111 1421.
URL https
//aclanthology org/N19-1421/
[5] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence?, 2019. URL/https:// pestps://arxivv hwtp:/orgggabb///./bea
[6] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code, 2021. URL attps://arzits http:///aaxvv...gaaaaa/////
03374
[7] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https://arxiv. wwwprivorgg/abs/2//0
06825
[8] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifrre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette- Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Blena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christoe Muraru, Grigory Rozhdestvenskly, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane labanowski, Jean- Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katic Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Macicj Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojeiech Stokowiee, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alck Andreev, and Kathleen Kenealy.
11

---

Gemma: Open models based on gemini research and technology, 2024. URL https:// arxiv. wtt.:r/ggab//...cooooo
[9] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
[10] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
[1I] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Ro- driguez. Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. 2023. URL BtLLhttp::// httthtttp//riiii.
http//aaxivv... www.aiiww.waaaa....a.a wttp/os////aa/////// 13971
[12] Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL htLpsts:: //te:////xxiii www.iii..gg///.. ab1/ 2412. 15115
[13] Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403. 240.:::270..18877,, 2024.
[14] Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, et al. Exploring concept depth: How large language models acquire knowledge and concept different layers? arXiv preprint 1XXv:2444.07066 2024.
[15] Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Peng Han, Shuo Shang, Aixin Sun, Yequan Wang, and Zhongyuan Wang. Not all layers of ilms are necessary during inference. arXiv preprint arXiv:2403.0218/ 2024.
[16] Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models, arXiv preprint arXiv:2502.02222 02013, 2025.
[17] Yefan Zhou, Tianyu Pang. Keqin Liu, Michael W Mahoney, Yaoqing Yang, et al. Temperature balancing, layer-wise weight analysis, and neural network training. Advances in Neural Information Processing Systems, 66:63542-63572.. 2023.
[18] Zihang Liu, Yuanzhe Hu, Tianyu Pang, Yefan Zhou, Pu Ren, and Yaoqing Yang. Model balancing helps low-data training and fine-tuning. arXiv preprint arXiv:2410.12277 2024.
[19] Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W Mahoney, and Yaoqing Yang. Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models. Advances in Neural Information Processing Systems, 37: 9117-9152. 2024.
[20] Lu Yin, You Wu, Zhenyu Zhang, Cheng- Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang. Michael Bendersky, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning Ilms to high sparsity, 2024. URL httts:://arx:: www.:ooggabbbo. 2atpb///131..........
[21] Sofia Serrano and Noah A Smith. attention interpretable? arXiv preprint arXiv:1906.03731, 2019.
[22] Samira Abnar and Willem Zuidema. Quantifying attention flow transformers, 2020. URI. /taps://arxiv www/abe.///eeoooooooott...t.aaet abs/ 2000 2005.00928
12

---

[23] Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation, 2019. URL https
// //arxiv. aaxiv..rg/aaie wtgggb////9/../// 04626)
[24] Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Self-attention attribution: Interpreting information interactions inside transformer, 2021. URL
11207
[25] Frank R Hampel. The influence curve and its role in robust estimation. Journal of the american statistical association, 69(346):383-333, 1974.
[26] R Dennis Cook and Sanford Weisberg, Characterizations of an empirical influence function for detecting influential cases in regression. Technometrics, 22(4):495-508, 1980.
[27] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference Machine Learning (ICML), pages 1885-1894. PMLR, 2017.
[28] Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances Neural Information Processing Systems, 33:19920-19930, 2020.
[29] Anshuman Chhabra, Bo Li, Jian Chen, Prasant Mohapatra, and Hongfu Liu. Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Models. In International Conference on Machine Learning (ICML). PMLR, 2025.
[30] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33: 2881-2891, 2020.
[31] Han Guo, Nazneen Fatema Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. Fastif: Scalable influence functions for efficient model interpretation and debugging. arXiv preprint Stdi::2012.1788.0 2020.
[32] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arX14:240..04333 2024.
[33] Xiaochuang Han, Byron C Wallace, and Yulia Tsvetkov. Explaining black box predictions and unveiling data artifacts through influence functions. arXiv preprint arXiv:2005.066 06676, 2020.
[34] Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308. 03296, 2023.
[35] Anshuman Chhabra, Peizhao Li, Prasant Mohapatra, and Hongfu Liu. "What Data Benefits My Classifier?" Enhancing Model Performance and Interpretability through Influence-Base Data Selection. In International Conference on Learning Representations (ICLR), 2024.
[36] Juhan Bae, Wu Lin, Jonathan Lorraine, and Roger Grosse. Training data attribution via approximate unrolled differentation. arXiv preprint arXiv:2405. 1816, 2024.
[37] Sang Keun Choe, Hwijeen Ahn, Juhan Bae, Kewen Zhao, Minsoo Kang, Youngseog Chung, Adithya Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, al. What is your data worth to gpt? Ilm-scale data valuation with influence functions. arXiv preprint 2rXiv:2405.13954 2024.
[38] Hadi Askari, Shivanshu Gupta, Terry Tong, Fei Wang, Anshuman Chhabra, and Muhao Chen. Unraveling indirect in-context learning using influence functions, 2025. URL https
// 1/arxiv.oz. org abs abs/ 2501 01473
[39] Hao Wang. Berk Ustun, and Flavio Calmon. Repairing without retraining: Avoiding disparate impact with counterfactual distributions. In International Conference on Machine Learning (ICML), pages 6618 86267666721 PMLR, 2019.
[40] Shuming Kong, Yanyan Shen, and Linpeng Huang. Resolving training biases via influence- based data relabeling. In International Conference on Learning Representations (ICIR),
1021.
13

---

[41] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gare mixture-of-experts layer. arXiv preprint arXiv: 110110013.330 2017.
[42] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
[43] Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. When moe meets Ilms: Parameter efficient fine-tuning for multi-task medical appli- cations. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1104
11 14, 2024.
[44] Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, et al. Loramoe: Alleviate world knowledge forgetting in large language models via moe-style plugin. arXiv preprint arXiv:2312.00977 2023.
[45] Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, and Sara Hooker. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. arXiv preprint arXiv:2309.0m 05444, 2023.
[46] Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and VS Subrahmanian. Higher layers need more lora experts, 2024. URL httpttt////aaiiiti org/ hbtg://b////// 2ttp:/2422.088588
[47] Peijun Qing, Chongyang Gao, Yefan Zhou, Xingjian Diao, Yaoqing Yang, and Soroush Vosoughi. Alphalora: Assigning lora experts based on layer training quality. arXiv preprint arXiv:2410. 10054, 2024.
[48] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot, 2023.
hhttps://arxii>: http:/abs/3301...000 00774
[49] Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1695, 2023.
[50] Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transferability of contextual representations. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 1073-1092 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18853/1/N11-111 1112. URL https //aclattho.oo...rg//Seeee
[51] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of BERT's attention. In Tal Linzen, Grzegorz Chrupata, Yonatan Belinkov, and Dieuwke Hupkes, editors, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286 Florence, Italy, August 2019. Association for Computational Linguistics. doi: 11118665/1111114482 URL https //aclantholog Mrg//W994488
[52] Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information, 2017. URL https:t/tts::iii www.aawwwvv..rvaraaa.aaaaee abrrgabss/17nn wwtps////3330000 00810
[53] Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? probing for sentence structure in contextualized word representations, 2019. URL http::htt///// /ttps//////xv///// wttp://bbbbb//..03..033333
[54] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2018. htttp::///rxiiv www.o//g...a...oooe.o// 01644
[55] Charles H. Martin and Michael W. Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning, 2018 URL.
attps://arxins www./rgg/////eeeeeee...eeeeeeeee wws//a10....0000 01075
14

---

[56] Lloyd S Shapley. A value for n-person games. In Harold W. Kuhn and Albert W. Tucker, editors, Contributions to the Theory of Games pages 307-317, Princeton University Press, Princeton, 1953.
[57] Yang Zhang, Yanfei Dong, and Kenji Kawaguchi. Investigating layer importance in large language models, 2024. URL
14381
[58] Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models, 2025. URL http::////aaxvvoooaaabbee// 02013
[59] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touretzky, ed- itor, Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann, 1989. URL attps::/pproceedinngg mewpeeeedddiinnnennn.seeee neurips. Saapeer_fileesppape//99//// 66G82bbaacc703332233111112776663
pdf
[60] Babak Hassibi and David Stork. Second order derivatives for network prun- ing: Optimal brain surgeon. In S. Hanson, J. Cowan, and C. Giles, editors, Advances in Neural Information Processing Systems, volume 5. Morgan-Kaufmant, 1992. URL
htwp:/eeeeediinnnnnerreeeeeeeeee neurips. wwtpp/ppe.fllleeppeeeeeeeeeeeeee /itees/paaper/99///1eee pdf
[61] Andrey Gromov, Kushal Tinumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. The unreasonable ineffectiveness of the deeper layers, 2025. URL httph:tt////// //axxiv org/
abs/ 0bs/2400 2403. 17887
[62] Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect, 2024. URL httttsttt////rraa org/ abs / 2403. 03853
[63] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner. On the structural pruning of large language models, 2023. URL httts::tttaacttee org/ barg/bb///0 ab2/2305. 11627
[64] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks, 2019. URL http:///aaaaaaa.g/////////0000/
[65] Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization, 2024. URL hhtpss
// //arxiv.org wrg. hwrgg/abb3336...mm 076299
[66] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, (2):3, 2022.
[67] Dengchun Li, Yingzi Ma. Naizheng Wang, Zhengmao Ye, Zhiyuan Cheng, Yinghao Tang, Yan Zhang, Lei Duan, Jie Zuo, Cal Yang, et al. Mixlora: Enhancing large language models fine-tuning with
mixture of experts. arXiv preprint arXiv:2404.15 15159, 2024.
[68] Shu Yang, Muhammad Asif Ali, Cheng-Long Wang, Lijie Hu, and Di Wang. Moral: Moe augmented lora for Iims' lifelong learning. arXiv preprint arXiv:2402.1 /200, 2024.
[69] Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, al. Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 1932-1999119 1945, 2024.
[70] Tongxu Luo, Jiahe Lei, Fangyu Lei, Weihao Liu, Shizhu He, Jun Zhao, and Kang Liu. Moelora: Contrastive learning guided mixture of experts on parameter-efficient fine-tuning for large language models. arXiv preprint arXiv:2402. 02..0...285.. 12851, 2024.
[71] Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mixture-of-loras: An efficient multitask tuning method for large language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLIN) 2024). pages 11371 1330, 2024.
15

---

[72] Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task task generalization via dynamic lora composition, 2024. URL httpps:/://axi
org/ abs/2307 13269
[73] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning (ICML). PMLR, 2019.
[74] Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas Spanos. Towards efficient data valuation based on the shapley value. In International Conference Artificial Intelligence and Statistics, 2019.
[75] Yongchan Kwon and James Zou. Beta Shapley: unified and noise-reduced data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics, 2022.
[76] Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas Spanos, and Dawn Song. Efficient task specific data valuation for nearest neighbor algorithms. Proceedings of the VLDB Endowment, 2018.
[77] Chih-Kuan Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks. Advances Neural Information Processing Systems, 2018
[78] Yuanyuan Chen, Boyang Li, Han Yu, Pengcheng Wu, and Chunyan Miao. Hydra: Hypergra- dient data relevance analysis for interpreting deep neural networks. In AAAI Conference on Artificial Intelligence, 2021.
[79] Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Datamodels: Predicting predictions from training data. arXiv preprint arXiv:220..0062.. 2022.
[80] Andrea Schioppa, Katja Filippova, Ivan Titov, and Polina Zablotskaia. Theoretical and practical perspectives on what influence functions do. Advances in Neural Information Processing Systems, 2024.
[81] Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models. In International Conference on Learning Representations (ICLR), 2024.
[82] Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up influence functions. In AAAI Conference on Artificial Intelligence, 2022.
[83] Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclere, and Aleksander Madry. Trak: Attributing model behavior at scale. arXiv preprint arXiv: 14186 2023.
[84] Irina Bejan, Artem Sokolov, and Katja Filippova. Make every example count: On the stability and utility of self-influence for learning from noisy nip datasets. In Conference on Empirical Methods in Natural Language Processing, 2023.
[85] Megh Thakkar, Tolga Bolukbasi. Sriram Ganapathy, Shikhar Vashishth, Sarath Chandar, and Partha Talukdar. Self-influence guided data reweighting for language model pre-training, In Conference on Empirical Methods in Natural Language Processing, 2023.
[86] Yung Kim, Kyungsu Kim, and Eunho Yang. Gex: A flexible method for approximating influence via geometric ensemble. Advances in Neural Information Processing Systems, 36, 2024.
[87] Guillaume Charpiat, Nicolas Girard, Loris Felardos, and Yuliya Tarabalka. Input similarity from the neural network perspective. Advances in Neural Information Processing Systems, 2019.
[88] R Douglas Martin and Victor Yohai. Influence functionals for time series. The annals of Statistics, pages 781-818, 1986.
16

---

[89] Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models. arXiv preprint urXiv:2202.0890M 2022.
[90] Lujun Li, Peijie Dong, Zhenheng Tang, Xiang Liu, Qiang Wang, Wenhan Luo, Wei Xue, Qifeng Liu, Xiaowen Chu, and Yike Guo. Discovering sparsity allocation for layer-wise pruning of large language models. Advances in Neural Information Processing Systems, 37: 141292-141317, 2024.
[91] Abraham Savitzky and M. J. E. Golay, Smoothing and Differentiation of Data by Sim- plified Least Squares Procedures. Anal. Chem., 36(8):1627-1639, July 1964. doi: 10. (021/ac602114047 HRLLhttp::/// //xx:
[92] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances neural information processing systems, 28, 2015.
[93] Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third international workshop paraphrasing (/WP2005), 2005.
[94] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint / 804.07461, 2018.
[95] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan, Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances Neural Information Processing Systems, 35:2507-2521, 2022.
[96] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensequ:: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:/8/1.0095: 2018.
[97] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint wrXiv:1809002780 2018.
[98] Christopher Clark, Kenton Lee, Ming-Wer Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 10044 2019.
[99] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: 1905.0780 2019.
[100] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
[101] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: /803005577 2018.
17

---

Appendix
A Limitations
A key limitation of Influence Functions is their reliance on an implicit convexity assumption, which may not strictly hold in LLMs. While this is known limitation of IF theory and an ongoing area of research, we find that our proposed LAYERIF framework demonstrates strong empirical performance on LLMs despite this. We also note that LAYERIF is training- time method and does not impact inference latency. Although computing IFs can be computationally intensive, particularly due to the Hessian inverse approximation, recent approaches such as Datalnf offer closed-form solutions that significantly reduce this overhead. Future work will explore integrating such scalable IF estimators to further improve the efficiency of our method. Another limitation of our method stems from its data-driven nature: the subset of data used to compute IF scores must be representative of the overall training and validation distribution. Failure to ensure representativeness can lead to biased or unreliable influence estimates, potentially degrading the effectiveness of the method.
B Broader Impact
Our data-driven approach to assessing layer quality using Influence Functions offers a promising direction for making pretrained LLMs more efficient for downstream tasks. By enabling more targeted pruning, models can be made both more lightweight and environmentally sustainable. Additionally, optimizing expert allocation further improving efficiency and reducing computational and environmental costs. Our method also contributes model interpretability by identifying which layers most responsible for performance on specific downstream tasks. This insight can guide further research, ultimately leading to more effective and efficient language models.
enhance the performance of smaller models specialized tasks,
C Clarifying Walk-Through Example
Here, we provide concrete step-by-ster walk-through of our method for greater clarity of our method.
Concrete walk-through for expert allocation (CommonQ dataset, 32-layer LLM, T=160 experts, 3=3) 3)
Below we provide the exact numbers produced by our implementation. For brevity only the first layers are shown; the remaining layers follow identically.
Layer Raw IF score
Inverted Power-scaled 6 Fractional
Final experts s;
-5.01 x 10
5.01 x 10a1
1.26 x 1035
1.12
2
8.36 x 10
8.36 x 10
5.84 x 1035
5.21
-7.99 10
7.99 x 10
5.10
1035
4.55
-8.46 10t1
8.46 x 10
6.05
1035
5,40
6
8.16 1
8.16 x 10
5.43 1035
4.84
6
Step1 (sign inversion). Invert the negative values to positive.
Step2 (power transform). Amplify disparities by raising to the power 33 to get é.
Step3 (normalise to budget). Scale so that the fractional allocations Σ fi equal m 128 extra experts (m 32 layers).
Step4 (floor + redistribute). Set lf + 1, guaranteeing 2 1 expert/layer, then distribute the remaining units to the layers with the largest fractional parts.
Finally, the complete allocation for CommonQ becomes:
2, 6, 6,6,6,6,,,83 7,
18

---

Interpretation. The middle layers receive the most experts, indicating they are comparatively under- trained for CommonQ: the first and very deep layers receive lower number experts, reflecting higher estimated quality.
D Gemma-7b Expert Allocation Results
This section presents results layer-wise expert allocation for Gemma-7B, using total of 160 experts in 5
Table 5: Accuracy across datasets using ALPHALORA, M0LA(3.5,7,8) and LAYERIF (+VE) for 160 experts Gemma-7b.
Dataset
ALPHALORA
M0LA(3,5,7.8)
LAYERIF (+VE)
MRPC
82.61
82.89
84.75
Cola
86.09
86.86
86.96
Text Science Q
94.15
93,88
93.66
Common Q
84.11
83,94
83.46
Openbook Q
87.80
88.80
88.20
Average
86.55
87.47
87.81
As shown in the results, LAYERIF achieves superior average performance compared to the baseline methods.
E Gemma-7b Sparsity Allocation Results
Here, we provide results layer-wise sparsity allocation for Gemma-7B for 50% sparsity in Table 6
Table 6: Comparison of average accuracies across different pruning strategies (Magnitude, Wanda, SparseGPT) and layer-wise sparsity allocation methods at 50% pruning on Gemma-7b.
Method
Magnitude
Wanda
SparseGPT
Average
UNIFORM
32.21
51.19
49.92
44.44
ALPHAPRUNING
34.21
50.76
51.39
45.45
OWL
33.27
47.15
47.48
42.63
LAYERIF
32.94
53.01
50.90
45.62
We can again see that LAYERIF has both the highest average and the best individual performance
after pruning.
F Sparsity Allocation Results with and without smoothing for Mistral-7b-vO.1
As described in Section 3, we apply Kovitzky-Gollay smoothing 91 to the min-man max normalized IF values within our layer-wise sparsity allocation algorithm. This smoothing step helps prevent any individual layer from being entirely pruned, while also enhancing the robustness and consistency of sparsity assignments. This particularly helpful in scenarios where raw influence scores fluctuate significantly between adjacent layers.
We used window-lenge length of 7 and polyorder of 3 for our filter. We present the percentage change in performance at 70% sparsity, comparing results with and without smoothing, for Mistral-7B-vO. in Table
We can clearly see that smoothing leads to an overall increase in performance across the different epsilon values and pruning methods.
Percentage change is computed as
with ancothing without smoothing without smoothing
x 100%
19

---

Table Percentage change between zero-shot accuracies across different values and LLM pruning methods.
Magnitude
Wanda
SparseGPT
0.1
(+) 1.49
(+)0.94
(+) 0.97
6
(+)
(+) 1.15
(+) 0.46
e 0.3
(+)0
(4)0.05
(+) 0,41
c 0.4
(+) 1.81
0,66
(+) 0.48
0.5
0.63
(+)0.78
0-)
G Statistical Significance of Results
We present the I-test statistical significance of our empirical results across both subsettings. Specifi- cally, we analyze the average accuracy of LAYER IF compared to baseline methods across all expert counts and datasets evaluated for layer-wise expert allocation, as shown Table
Table Computing the t-test metric to compare LAYERIF with MOLA and ALPHALORA on zero- shot accuracy across the different number of experts (** indicates statistical significance i.e., p-value < 0.05)
Algorithm
Average Accuracy
p-valuc
ALPHALORA
79.30
0,03152"
MOLA
78.97
0.03123"
LAYERIF
82.28
Table Cllearly shows statistically significant improvement over both of the other baselines.
Table 9: Computing the t-test metric to compare LAYERIF with ALPHAPRUNING, UNIFORM and OWL on zero-shot accuracy across the different number of pruning ratios (** indicates statistical significance i.e., p-value < 0.05)
Algorithm
Average Accuracy
p-value
ALPHAPRUNING
62.42
0.01969"
UNIFORM
62.58
0.03077
OWL
62.61
0.09156
LAYERIF
62.84
We now analyze the average accuracy of LAYERIF compared to baseline methods across all pruning ratios for layer-wise sparsity allocation, as shown in Table
Table 9 clearly shows that LAYERIF produces statistically significant improvements in all of the baselines, except OWL, where shows marginally significant improvement.
H
Wall Clock Times and Memory Consumption for Varying Dataset and Model Sizes
We report the wall-clock times of our Influence Function (IF) computation using the DATAINF approximation, compared against the TRACIN baseline, under varying dataset sizes and model scales. All experiments were performed on 4x NVIDIA RTX 6000 Ada Generation GPUs.
H.1 Scalability to Dataset Size
Table 10 presents runtimes for computing IFs over a single layer with increasing numbers training and validation samples. As expected, the runtime scales approximately linearly with dataset size. Although DATAINF is substantially slower than TRACIN, the total cost remains modest relative to the computational expense of pretraining large language models. For comparison, training LLAMA2- 70B reportedly required 1..22M GPU hours, whereas DATAINF adds only a minor diagnostic overhead that can be amortized across layers identify underperforming components during training.
The peak GPU memory consumption during influence computation for dataset of 300 training and 50 validation samples approximately 35000MiB for the 7B model. This footprint remains
20

---

Table 10: Wall-clock runtimes (in seconds) for computing IFs for single layer under different dataset
sizes on MISTRAL--B--O...
Dataset Size (TrainxVal)
Method
Runtime
DATAINF 163,721
3000x500
TRACIN 1.35 x 10-2
DATAINF 402.77
300x50
TRACIN 3.31 x 10-5
DATAINF 15.3
30x5
TRACIN 1.19 10
well within the typical per-GPU memory budget, indicating that influence-based diagnostics can be feasibly integrated into existing training pipelines.
H.2
Scalability to Model Size
We also evaluate scaling across model sizes in Table The runtime grows roughly proportionally to parameter count.
Table 11: Wall-clock runtimes (in seconds) for DATAINF and TRACIN under different model sizes (300x50 dataset).
Model
Method Runtime (s)
DATAINE 58.063
QWEN2,5-3B
TRACIN 2.79 x 10-5
DATAINF 2,824.16
QWEN2,5-32B
TRACIN 6.82 x 10-5
Overall, while DATAINF incurs higher per-layer computation cost than first-order approximations such TRACIN, it remains computationally tractable even for multi-billion-parameed models. Its cost negligible relative to the full pretraining budget, while providing rich layer-wise diagnostic signals that can inform targeted retraining pruning strategies.
I Code and Reproducibility
We follow the datasets and evaluation protocol of AlphaLora 47 for layer-wise expert allocation and the datasets and evaluation protocol of Alphapruning [19 for layer-wise sparsity allocation.
Our code has been anonymized and uploaded here: /tttps://github. htwp:thuu..com/aaaasaaaaaae com/HadiAskari/EXpprr Allocation and hntps:: //tit/gitttt gitt.b....... com Hadi Alkari/yyakkkrr Bactill___unnnnneeee
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes all claims accompanied with experimental results in Section 4, detailed discussion Section 5 and methodological grounding in Section 3.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, we provide limitations section in Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
We do not have theoretical results discussed in our paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The methods are clearly defined in Section 3 and Section Additionally, we include a Code and Reproducibility section in Appendix F.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification:
Yes, we provide our code in Appendix F.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
Phbllic//uidess/ddeettmiii
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ L/tipp.cc/public//giice http://pibbpccuuuuee////eeeitiia /CodeSubbeiiiccec iitoommsssiin ssi oeeicccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, details are provided in Section and Appendix F. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, we provide section on statistical significance in Appendix G.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
24

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, provide details in Section and Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpp/rrpi....cp.......aaacaaaaa
Answer: [Yes]
Justification: Yes, complies with the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide Broader Impacts section in Appendix B.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
25

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any models or datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we provide the respective citations for all assets that we use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
26

---

Answer:
[NA]
Justification:
We do not release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/codd/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not use any crowdsourcing or perform research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not do research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
27

---

Justification:
We did not use any LLMs for the core methods of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to LLM policy htteasas//eerrpppoccc///o/rerees for what should or should not be described.
28


Reviews:
[{"review_id": "Lac6oKOjo7", "reviewer": "Reviewer_w8JJ", "summary": "This paper studies how to adopt the classical influence functions (IF) to estimate the importance of each layer in LLM. The paper proposes LayerIF and shows how LayerIF can guide:\n- expert-count allocation in mixture-of-experts (MoE) training\n- layer-wise sparsity pruning.  \n\nExperiments on a suite of GLUE and commonsense benchmarks demonstrate modest but consistent gains over existing heuristics.  Because LayerIF scores are computed offline, the method does not add inference-time latency.", "questions": "See Weakness and Questions.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength\n1. The work is the first to leverage IF at layer granularity in LLMs, providing new insight into module-level contributions and may potentially inspire more principled schemes for MoE routing and structured pruning.\n\n2. IF computation is traditionally expensive. However, in LayerIF, computation of IF only occurs once during analysis or training.  Both the proposed MoE allocation and pruning methods rely on pre-computed IF scores, and the computation in the inference stage remains unchanged.\n\n## Weakness and Questions\n\n1. The technical presentation in Sections 3.3 and 3.4 is very difficult to follow. For example, Lines 204–232 interleave motivations, notations, math expressions, and constraints in long paragraphs, without the aid of any diagram or algorithm block.\n\n2. The method description is inconsistent. Section 3.3 states that negative-influence samples are excluded when aggregating IF values. However, Section 4.1 evaluates three variants, ALL, +VE, and TOP-25 %.\n\n3. Table 1 shows that the best-performing variant differs across tasks (e.g., +VE for OpenbookQA, ALL for CoLA, TOP-25 % for MRPC).  A principled guideline for choosing the variants or an automatic tuning strategy is missing.\n\n4. The paper claims that “IFs measure the sensitivity of model loss to individual training points and the loss of well-trained layers should show smaller changes”. This is a bit counterintuitive, removing an important layer should increase loss significantly.", "was_revised": true, "final_justification": "My primary concern is about the principles behind the three variants: TOP‑25%, +VE, and ALL. The authors's rebuttal has addressed my concern.", "rebuttal": "We are grateful to the reviewer for their feedback of our work, and are appreciative of their time, effort, and consideration. Below we provide more discussion on the points raised.\n\n* **Technical Presentation/Clarity**: \n\n    We thank the reviewer for their suggestion and will improve the clarity of Section 3.3 and Section 3.4 as requested. We will aim to provide a concrete step-by-step walk-through of our method, in the revised version of the paper, as described below:\n\n\n    **Concrete walk‑through for expert allocation (CommonQ dataset, 32‑layer LLM, T=160 experts, beta=3)**\n\n    Below we provide the exact numbers produced by our implementation.\n    For brevity only the first 5 layers are shown; the remaining layers follow identically.\n\n    | Layer $i$ | Raw IF score $v_i$ | Inverted $ṽ_i$ | Power‑scaled $v̂_i$ | Fractional $f_i$ | Final experts $s_i$ |\n    | --------- | --------------------------------------- | ------------------------------------------------------------- | -------------------------------------------------------------- | ------------------------ | --------------------------- |\n    | 0         | -5.01×10¹¹                            | 5.01×10¹¹                                                        | 1.26×10³⁵                                                              | \t1.12                    | **2**                       |\n    | 1         | -8.36×10¹¹                            | 8.36×10¹¹                                                    | 5.84×10³⁵                                                    | 5.21                    | **6**                       |\n    | 2         | -7.99×10¹¹                            | 7.99×10¹¹                                                   | 5.10×10³⁵                                                    | 4.55                     | **6**                       |\n    | 3         | -8.46×10¹¹                            | 8.46×10¹¹                                                    | 6.05×10³⁵                                                    | 5.40                     | **6**                       |\n    | 4         | -8.16×10¹¹                            | 8.16×10¹¹                                                   | 5.43×10³⁵                                                    | 4.84                     | **6**                       |\n    | …         | …                                       | …                                                             | …                                                              | …                        | …                           |\n\n\n\n  (1) **Step1 (sign inversion).** Invert the negative values to positive.\n\n  (2) **Step2 (power transform).** Amplify disparities by raising to the power beta to get v̂.\n\n  (3) **Step3 (normalise to budget).** Scale v̂ so that the fractional allocations ∑fi equal **T−m=128** extra experts (m=32 layers).\n\n  (4) **Step4 (floor+redistribute).** Set si=⌊fi⌋+1, guaranteeing ≥1 expert/layer, then distribute the remaining 24 units to the layers with the largest fractional parts.\n\n  Finally, the complete allocation for **CommonQ** becomes:\n\n    ```[2, 6, 6, 6, 6, 5, 7, 6, 8, 7, 6, 7, 6, 6, 6, 5, 6, 6, 5, 4, 5, 5, 4, 2, 4, 5, 3, 4, 2, 3, 4, 3]```\n\n    **Interpretation.** The middle layers receive the most experts, indicating they are comparatively under‑trained for CommonQ; the first and very deep layers receive lower number of experts, reflecting higher estimated quality.\n\n    Similarly we will also provide an example for the layer-wise sparsity allocation setting. If the reviewer would like us to take any other steps to improve clarity, we are more than happy to do so.\n\n___\n___\n\n* **The method description is inconsistent in Section 3.3**:  \n \n\n    We appreciate the reviewer’s attention to this apparent inconsistency and apologize for the confusion. We intend to clarify that Section 3.3 describes the default pipeline used throughout the paper. Specifically, we aggregate only positive‑influence samples (`+VE`) and discard negative ones.\n\n    In Section 4.1, we include two additional variants, `ALL`, which retains every sample, and `TOP 25%`, which selects the top quartile of positive IF scores by magnitude, as part of a controlled ablation study. These ablations were included at this stage because including them earlier would have introduced unnecessary complexity and we wanted to empirically test our design choices.\n\n    These variants were included solely to assess robustness and justify our aggregation choice; they were not used in the main experimental pipeline, as reflected in Figures 2 and 3 and Table 2.\n\n    Additionally, we perform another ablation in the rebuttal period where we compare keeping all of the samples against keeping only the positive samples on the Gemma-7b model.\n\n    | Dataset            | IF (+ve)  | IF (ALL) |\n    |--------------------|------------------|-----|\n    | MRPC               | 84.75            | 82.49 |\n    | CoLA               | 86.96            | 85.14 |\n    | Common_Q           | 83.46            | 82.06 |\n    | Openbook_Q         | 88.20            | 86.20 |\n    | Text_Science_Q     | 93.66            | 94.56 |\n    | **Average**        | **87.81**        | **86.09** |\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Here, we see that IF (+ve) was significantly better performing on average and individually more performant on 4 out of the 5 datasets. \n\n___\n___\n\n* **A principled guideline for choosing the variants or an automatic tuning strategy is missing:**\n\n\n    We thank the reviewer for highlighting this point. We would like to point that while Table 1 shows some variation across tasks as the reviewer mentioned e.g., `+VE` performs best on OpenbookQA, and `TOP‑25%` on MRPC and Text Science Q, the overall trend strongly favors the IF based methods, particularly `TOP‑25%`. Specifically, `TOP‑25%` achieves: \n\n    - the highest **macro-average** performance (82.27%) across all tasks,\n    - has the lowest standard deviation 1.84.\n\n    This robustness makes `TOP‑25%` a strong default in practice. The use of a fixed, global aggregation threshold is also consistent with precedent in prior influence function research. For instance, in [1], as evident in Table 5 and Table 6, aggregate performance trends across tasks are generally more informative than per-dataset maxima. This is also evident in our results. \n\n    However, we agree with the reviewer that dataset-specific optimal thresholding might yield further gains, and will definitely aim to study that exploration more in follow up work. \n\n___\n___\n\n* **Removing an important layer should increase loss significantly:**\n\n    We believe there may be a misunderstanding and would like to clarify our intended meaning. When we state in the Introduction that \"IFs measure the sensitivity of model loss to individual training points and the loss of well-trained layers should show smaller changes,\" we are emphasizing that well-trained layers tend to be more stable i.e., they exhibit lower sensitivity to perturbations in the training data. This interpretation is further elaborated in the “Justification” paragraph of Section 3.2. \n\n    We fully agree with the reviewer that removing important layers would significantly degrade model performance and increase loss. Our empirical results support this: the layers identified by our method as less sensitive (and thus likely better trained) receive fewer LoRA experts and are pruned more conservatively, leading to stronger downstream performance.\n\n    To further emphasize this point, we conduct an ablation where we reversed our methodology by pruning the less sensitive layers more at a 70% sparsity budget on Mistral-7b-v0.1. We present the results below.\n\n\n\n    |             Method  | Magnitude | Wanda  | SparseGPT | Average  |\n    |---------------------|-----------|--------|-----------|----------|\n    | Our Method          | 33.439    | 32.497 | 41.143    | 35.693   |\n    | Reversed Allocation | 33.101    | 32.015 | 38.450    | 34.522   |\n  \n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;We will add this ablation to the revised version of our paper. \n\n---\n___\n\nThank you again for taking the time to review our paper. We hope that our responses alleviated the concerns raised, and would be happy to engage further during the discussion phase in case there are any others remaining. \n\n---\n___\n\nReferences:\n\n[1] Chhabra, Anshuman, et al. \"Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Models.\" ICML 2025."}, {"review_id": "UnqTM1J2Q5", "reviewer": "Reviewer_8TjZ", "summary": "This paper introduces LayerIF, a novel framework for evaluating the quality of individual layers within Large Language Models (LLMs). The main idea is to adapt Influence Functions (IFs), a technique used to measure a single training data point's effect on model predictions, to determine the importance of each model layer.\n\nUnlike other methods which use model-centric metrics (like weight magnitudes), LayerIF gives a data-driven measure of layer quality.  The main computation involves determining the sensitivity to a given layers parameters to the training data, which involves calculating the gradient and Hessian per layer (or transformer block).\n\nThe authors test their method in two applications: (1) expert allocation in MoE models, where the method allocates experts based on the influence/importance score instead of standard uniform allocation and (2) layer-wise pruning, which prunes layers with higher influence scores.", "questions": "I alluded to my questions in the strengths/weaknesses but will state them explicitly here:\n1. Could you provide a more quantitative analysis of why Hessian-free methods like TracIn fail to produce useful layer-quality estimates? For example, in the Hessians the authors compute, how badly does the Gauss-Newton approximation fail?\n2.  Can they add in some discussion about the computational overhead?  For example, for a 7B parameter model on a typical GLUE task, what is the approximate wall-clock time and memory required to compute the scores?\n3.  There is an interesting direction for multimodal applications.  Currently, in many vision MoEs, there is really just an explicit routing to vision experts.  It would be very interesting if this method could be applied to get VLAs to perform better or if in those cases, it is always the case to explicitly route to vision experts.  Have the authors thought about how this might be used to set up training in those sorts of applications?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n* Applying influence functions to determine the importance of architectural components / layers instead of data points is a clever extension of an existing method.  The idea of generating different layer-importance rankings depending on downstream tasks makes intuitive sense over single allocation strategies like LoRA variants.\n* The authors provide a thorough empirical evaluation of their method. The consistent, though modest, performance gains across two different applications (MoE and pruning), multiple LLMs (Mistral-7B, Gemma-7B), and various experimental settings demonstrate the potential of their method.\n\nWeaknesses:\n\n* There are no major weaknesses.  The primary areas of improvement relate to the clarity of certain methodological choices and the practical implications of the method's computational cost.\n  * for example, the explanation of how raw LayerIF scores are mapped to expert scores could benefit from a bit more explanation and is quite dense.  It is not immediately clear how the total target sum $T$ is determined and why the specific power transformation and normalization steps were selected.  Seeing an explicit example here would be helpful.  Also, why are the raw scores noisy between adjacent layers?\n* The use of a Hessian-based method (DataInf), while efficient for LoRA, is a computational hurdle for the broader applicability of this method to large models perhaps in MoE routing during pretraining.  It is interesting that approximations like TracIn perform poorly for the task. Is the Gauss-Newton form a bad approximation of the true Hessian in this case?\n   * Including some numbers here would be useful, both for wall-clock time to compute the influence function for different model sizes, as well as how it scales as a function of the data points involved.\n   * It would be interesting to incorporate this method during pretraining, perhaps periodically updating the probabilities with which experts are routed, but there may be significant computational bottlenecks in that setting.  Otherwise, the broader applications of the method seem more limited to smaller-scale, finetuning setups.", "was_revised": true, "final_justification": "framework for evaluating the quality of individual layers within LLMs. The main idea is to adapt Influence Functions (IFs), a technique used to measure a single training data point's effect on model predictions, to determine the importance of each model layer.\n\nWhile the applications are somewhat limited due to the high computational complexity of evaluating the influence functions, which in my opinion remains the primary weakness of this work, the experiments are thorough and one could envision some downstream applications to multimodal MoEs.  The authors also addressed my questions about if a Gauss-Newton approximation could perform as well (for lower computational cost) through additional experiments.\n\nTherefore I am going with a decision of 'accept.'", "rebuttal": "We are grateful to the reviewer for their detailed feedback of our work, and are appreciative of their time, efforts, and consideration. Below we provide more discussion on some of the points raised.\n\n* **Explicit Walk-through of the algorithm**: \n\n    We thank the reviewer for their suggestion and will improve the clarity of the methods (as well as design choices) as requested. We will aim to provide a concrete step-by-step walk-through of our method, in the revised version of the paper, as described below:\n\n    **Concrete walk‑through for expert allocation (CommonQ dataset, 32‑layer LLM, T=160 experts, beta=3)**\n\n    Below we provide the exact numbers produced by our implementation.\n    For brevity only the first 5 layers are shown; the remaining layers follow identically.\n\n    | Layer $i$ | Raw IF score $v_i$ | Inverted $ṽ_i$ | Power‑scaled $v̂_i$ | Fractional $f_i$ | Final experts $s_i$ |\n    | --------- | --------------------------------------- | ------------------------------------------------------------- | -------------------------------------------------------------- | ------------------------ | --------------------------- |\n    | 0         | -5.01×10¹¹                            | 5.01×10¹¹                                                        | 1.26×10³⁵                                                              | \t1.12                    | **2**                       |\n    | 1         | -8.36×10¹¹                            | 8.36×10¹¹                                                    | 5.84×10³⁵                                                    | 5.21                    | **6**                       |\n    | 2         | -7.99×10¹¹                            | 7.99×10¹¹                                                   | 5.10×10³⁵                                                    | 4.55                     | **6**                       |\n    | 3         | -8.46×10¹¹                            | 8.46×10¹¹                                                    | 6.05×10³⁵                                                    | 5.40                     | **6**                       |\n    | 4         | -8.16×10¹¹                            | 8.16×10¹¹                                                   | 5.43×10³⁵                                                    | 4.84                     | **6**                       |\n    | …         | …                                       | …                                                             | …                                                              | …                        | …                           |\n\n\n\n  (1) **Step1 (sign inversion).** Invert the negative values to positive.\n\n  (2) **Step2 (power transform).** Amplify disparities by raising to the power beta to get v̂.\n\n  (3) **Step3 (normalise to budget).** Scale v̂ so that the fractional allocations ∑fi equal **T−m=128** extra experts (m=32 layers).\n\n  (4) **Step4 (floor+redistribute).** Set si=⌊fi⌋+1, guaranteeing ≥1 expert/layer, then distribute the remaining 24 units to the layers with the largest fractional parts.\n\n  Finally, the complete allocation for **CommonQ** becomes:\n\n    ```[2, 6, 6, 6, 6, 5, 7, 6, 8, 7, 6, 7, 6, 6, 6, 5, 6, 6, 5, 4, 5, 5, 4, 2, 4, 5, 3, 4, 2, 3, 4, 3]```\n\n    **Interpretation.** The middle layers receive the most experts, indicating they are comparatively under‑trained for CommonQ; the first and very deep layers receive lower number of experts, reflecting higher estimated quality.\n\n    Similarly we will also provide an example for the layer-wise sparsity allocation setting. If the reviewer would like us to take any other steps to improve clarity, we are more than happy to do so.\n\n___\n___\n\n* **Why are the raw scores noisy between adjacent layers?**: \n\n    We thank the Reviewer for this important question. Influence‑function estimates at adjacent layers are inevitably noisy: they are computed from stochastic Hessian approximations and mini‑batch gradients, and thus contain high‑frequency fluctuations  that can misrepresent the true differences in layer quality. Grosse et al. [1] visualize this variability directly: the layer × token heatmap in Figure 4 (p.20) reveals sharp red/teal oscillations across adjacent layers, reflecting high-frequency fluctuations in estimated influence. The authors also note in Section 2.1 that influence values are *extremely heavy-tailed*, which complicates traditional statistical analysis and motivates the need for robust aggregation strategies.\n\n    Hence, we apply a Savitzky–Golay smoother (window = 5, poly‑order = 2) to: a) Suppress spurious high‑frequency noise, while b) Preserving low‑frequency structure, the monotonic rise–fall pattern that truly indicates which layers of the LLM are under or over‑trained.\n\n    We will add a sentence clarifying this rationale in Section 3.4 accordingly.\n\n___\n___\n\n* **Wall Clock times and Memory Consumption for DataInf and TracIn under varying datasets and model sizes:**\n\n    We thank the reviewer for raising the important point regarding the computational times of Influence Function methods. Below, we provide wall-clock timings for computing the influence function for a single layer across varying training and validation set sizes. We used 4× NVIDIA RTX 6000 Ada Generation GPUs to perform these experiments. \n\n\n    | Scalability to different Dataset Sizes | Method  | Runtime (seconds) |\n    | -------------------- | ------- | --------------------------- |\n    | Mistral-7B-v0.1, 3000×500         | DataInf | 163,721                     |\n    |                      | TracIn  |  0.0135    |\n    | Mistral-7B-v0.1, 300×50           | DataInf | 402.77                      |\n    |                      | TracIn  | 3.314×10^-5                |\n    | Mistral-7B-v0.1, 30×5             | DataInf | 15.3                        |\n\n\n    As we can see the time increases non-linearly with larger datasets, but we intend to clarify that in our experiments, having approximately 300-500 training samples and 50-100 validation samples is typically sufficient to gauge how well an LLM layer generalizes to a task. Moreover, the current implementation of DataInf runs primarily on CPU; preliminary tests show that offloading key computations to the GPU yields up to a 2× speedup, indicating substantial room for further efficiency improvements. \n\n    Similarly, we report the runtime for computing the influence function for 1 layer for different model sizes. \n\n\n    | Scalability to different Model Sizes | Method  | Runtime (seconds) |\n    | -------------------- | ------- | --------------------------- |\n    |                      | TracIn  | 1.192×10^-6                |\n    | Qwen2.5‑3B, 300×50   | DataInf | 58.063                      |\n    |                      | TracIn  | 2.789×10^-5                 |\n    | Qwen2.5‑32B, 300×50  | DataInf | 2,533.46                    |\n    |                      | TracIn  | 5.245×10^-6                 |\n\n\n    As expected, larger models incur higher influence computation times; however, these costs are     negligible relative to the overall expense of pretraining large language models. For context, training     LLaMA2‑70B required approximately 1.72 million GPU hours. By comparison, our method     represents a minor overhead and can be integrated into the broader pretraining pipeline to diagnose     underperforming layers and nudge the model toward more uniform training dynamics. This can     support developers in identifying and addressing layer‑specific inefficiencies during model     development.\n\n\n    Finally, the peak GPU memory usage during influence computation for a dataset with 300 training and 50 validation samples was approximately 35,000 MiB for a 7B parameter model. This footprint is relatively modest given the model size and suggests that influence-based diagnostics can be feasibly incorporated into GPU-based training pipelines without exceeding typical memory constraints.\n\n___\n___\n\n* **Gauss-Newton Approximation's applicability:**\n\n    The main bottleneck with the Gauss-Newton Hessian is the time required for IF computation. While doing initial experiments, we considered utilizing the Guass-Newton Hessian instead of the DataInf approximation. However, even the time taken to compute the **Gauss-Newton Hessian IF** for 30 Train and 5 Validation samples was **2494.72 seconds** for a 7B model whereas the equivalent experiment with the **DataInf IF** approximation took only **15.3 seconds**. Hence, it made it unfeasable for our experiments, which required 300 Train and 50 Validation samples.\n\n___\n___\n\n* **Applicability VLA settings:** \n\n    Thank you. We appreciate the reviewer’s suggestion and agree that multimodal mixtures‑of‑experts (MoEs) and particularly VLAs represent an exciting next step. There is a conceptual fit as our influence‑based layer‑quality metric is modality‑agnostic: it only requires gradients and Hessian–vector products for the parameters under consideration. In principle, we can therefore:\n\n    1. **Compute per‑expert influence** for vision, language, and cross‑modal experts separately;  \n    2. **Compare their sensitivity profiles** on joint vision–language objectives (e.g., image‑text contrastive loss, VQA loss);  \n    3. **Adapt the routing policy** according to these influence scores.\n\n    We have actually begun conducting some preliminary research in this very exciting area as part of follow-up work, and will be releasing it in the near future for the community.\n\n\n---\n___\n    \nThank you once again for all your efforts in helping to strengthen our work. We hope that our comments helped address your concerns and if so, would be extremely obliged if you could consider increasing your score. \n\n___\n---\n\nReferences:\n\n[1] Grosse, Roger, et al. \"Studying large language model generalization with influence functions.\" arXiv preprint arXiv:2308.03296 (2023)."}, {"review_id": "XtBN04XrpP", "reviewer": "Reviewer_df7d", "summary": "This paper introduces LayerIF, a novel framework for estimating the layer-wise quality in Large Language Models (LLMs) using Influence Functions (IFs). LayerIF quantifies layer quality by measuring each layer’s sensitivity to training data, producing task-specific importance scores. The authors validate their approach through two downstream applications: (1) data-driven expert allocation in Mixture-of-Experts (MoE) architectures, which optimises the distribution of LoRA adapters per layer , and (2) layer-wise sparsity distribution for LLM pruning, where layers identified as being of lower quality are pruned more aggressively.", "questions": "1. Please explain the correct form of the influence function, $I(z_i)$, and the layer quality score, $S^{(l)}$, in the paper. \n2. Could a theoretical analysis or an empirical ablation study be provided to support the formulation of $S^{(l)}$, which involves summing only the positive influences?\n3. In the MoE expert allocation section, please explain the relationship between $v_i$ and $ S^{(l)}$. If $v_i = S^{(l)}$, please clarify the definition for the function $\\tilde{v_i}^{\\beta}$.\n4. Please explain the computational efficiency, approximation error, and memory overhead associated with the calculation of the Hessian inverse.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The work addresses a critical gap in LLM efficiency: data-driven layer optimisation. The framework's ability to inform task-specific structural adaptations, such as dynamic expert allocation, is a significant step towards the more efficient deployment of LLMs.\n2. To my knowledge, this paper is the first to leverage Influence Functions for the purpose of estimating layer quality in LLMs. This presents a novel application of a classic statistical technique and provides a new, data-centric lens for analysing model internals.\n\nWeaknesses\n1. Inconsistency in Formulation: The paper's formulation of the Influence Function contains a significant theoretical inconsistency that creates considerable confusion. The definition of \n$I(z_i)$ on page 4 is missing a leading negative sign when compared to the original formulation in the cited reference [26] (Understanding Black-box Predictions via Influence Functions). While the mathematical formula is different, the paper's textual interpretation—that a positive influence score corresponds to a detrimental sample that increases the loss —aligns with the original work. This discrepancy between the paper's mathematical definition and its textual interpretation is a notable flaw.\n2. Lack of Justification for Score Aggregation: The paper defines the layer quality score, $S^{(l)}$, by exclusively summing the positive influences. It is unclear why negative influences are excluded. The rationale for this choice over alternatives, such as  the net sum of positive and negative influences, is not provided. Theoretical support or ablation study comparing these different aggregation strategies would be necessary to validate this design.\n3. Lack of Clarity in MoE Allocation Method: Specifically, it introduces a new variable, $v_i$, to represent the layer score without explicitly stating its relationship to the previously defined score $S^{(l)}$ from Section 3.2. If $v_i$ is equal to $S^{(l)}$ and $S^{(l)}$ is a positive number, then $\\tilde{v_i}$ is negative. Then For any non-integer value of $\\beta>0$, $\\tilde{v_i}^{\\beta}$ is not well-defined within the real numbers and would yield complex values. \n4. Insufficient Analysis of Computational Feasibility: The paper claims computational efficiency can be achieved by leveraging recent methods like DataInf, but it does not provide a concrete analysis of this. A quantitative comparison of the runtime, approximation error and memory overhead is needed to substantiate claims of practical efficiency.\n5. Minor Issues  \n(1) Line 89, \"pruninng\" should be \"pruning\".  \n(2) There is an inconsistency in the model version used. The text states experiments were conducted on Mistral-7b-v0.3 , but the captions for Table 2 and Figure 2 specify Mistral-7b-v0.1.  \n(3) In Appendix G, \"laywer-wise\" should be \"layer-wise\".  \n(4) In the title for Table 4 in the Appendix, \"accross\" should be \"across\".", "was_revised": true, "final_justification": "The reviewer’s concerns have been addressed. The reviewer appreciates the authors’ efforts to improve the work and has raised the score to 4.", "rebuttal": "We are grateful to the reviewer for their detailed feedback of our work, and are appreciative of their time, efforts, and consideration. Below we provide more discussion on the points raised.\n\n* **Inconsistency in formulation:** \n\n    Thank you for pointing out this typo, we appreciate it. We sincerely apologize for the missing (leading) '-' sign in the formulation, and will easily make this correction in the paper revision. However, after adding the - sign, the rest of the paper can remain unchanged and no other major changes are required.\n\n    We would also like to mention that the typo in the formulation does not change the validity of our experimental design or the results of our experiments, and we apologize for the inconsistency once again. \n\n___\n___\n\n* **Lack of Justification for Score Aggregation:** \n\n    We thank the reviewer for raising this point. Our choice to aggregate only positive influence values was guided primarily by empirical evidence rather than theoretical assumptions. To validate this design choice we conducted this ablation in Section 4.1 where the `Top 25%` variant was the highest performing. Additionally, we perform another ablation in the rebuttal period where we compare keeping all of the samples against keeping only the positive samples on the Gemma-7b model.\n\n\n    | Dataset            | IF (+ve)  | IF (ALL) |\n    |--------------------|------------------|-----|\n    | MRPC               | 84.75            | 82.49 |\n    | CoLA               | 86.96            | 85.14 |\n    | Common_Q           | 83.46            | 82.06 |\n    | Openbook_Q         | 88.20            | 86.20 |\n    | Text_Science_Q     | 93.66            | 94.56 |\n    | **Average**        | **87.81**        | **86.09** |\n\n\n    Here, we see that IF (+ve) was significantly better performing on average and individually more performant on 4 out of the 5 datasets. \n\n    We acknowledge that this does not preclude the possibility of more nuanced aggregation schemes. Future work could explore adaptive thresholds or weighting strategies that combine both helpful and harmful samples in a principled way. We will expand our discussion to acknowledge this broader design space in the revised version of this paper.\n\n___\n___\n\n* **Lack of Clarity in MoE Allocation Method:** \n\n    We apologize for the lack of clarity. The $v_i$ is indeed corresponding the the individual layer scores in $S^{(l)}$. Since our implementation followed the DataInf convention as described above, the layerwise values in $S^{(l)}$ were all negative, hence the inversion and then power transformation did not lead to complex values. We will revise this and make things consistent in the paper. \n\n___\n___\n\n* **Computational feasibility for DataInf and TracIn under varying datasets and model sizes:**\n\n    We thank the reviewer for raising the important point regarding the computational times of Influence Function methods. Below, we provide wall-clock timings for computing the influence function for a single layer across varying training and validation set sizes. We used 4× NVIDIA RTX 6000 Ada Generation GPUs to perform these experiments. \n\n\n    | Scalability to different Dataset Sizes | Method  | Runtime (seconds) |\n    | -------------------- | ------- | --------------------------- |\n    | Mistral-7B-v0.1, 3000×500         | DataInf | 163,721                     |\n    |                      | TracIn  |  0.0135    |\n    | Mistral-7B-v0.1, 300×50           | DataInf | 402.77                      |\n    |                      | TracIn  | 3.314×10^-5                |\n    | Mistral-7B-v0.1, 30×5             | DataInf | 15.3                        |\n\n\n    As we can see the time increases non-linearly with larger datasets, but we intend to clarify that in our experiments, having approximately 300-500 training samples and 50-100 validation samples is typically sufficient to gauge how well an LLM layer generalizes to a task. Moreover, the current implementation of DataInf runs primarily on CPU; preliminary tests show that offloading key computations to the GPU yields up to a 2× speedup, indicating substantial room for further efficiency improvements. \n\n\n    Similarly, we report the runtime for computing the influence function for 1 layer for different model sizes. \n\n\n    | Scalability to different Model Sizes | Method  | Runtime (seconds) |\n    | -------------------- | ------- | --------------------------- |\n    |                      | TracIn  | 1.192×10^-6                |\n    | Qwen2.5‑3B, 300×50   | DataInf | 58.063                      |\n    |                      | TracIn  | 2.789×10^-5                 |\n    | Qwen2.5‑32B, 300×50  | DataInf | 2,533.46                    |\n    |                      | TracIn  | 5.245×10^-6                 |\n\n\n    As expected, larger models incur higher influence computation times; however, these costs are negligible relative to the overall expense of pretraining large language models. For context, training LLaMA2‑70B required approximately 1.72 million GPU hours. By comparison, our method represents a minor overhead and can be integrated into the broader pretraining pipeline to diagnose underperforming layers and nudge the model toward more uniform training dynamics. This can support developers in identifying and addressing layer‑specific inefficiencies during model development.\n\n\n    Furthermore, the peak GPU memory usage during influence computation for a dataset with 300 training and 50 validation samples was approximately 35,000 MiB for a 7B parameter model. This footprint is relatively modest given the model size and suggests that influence-based diagnostics can be feasibly incorporated into GPU-based training pipelines without exceeding typical memory constraints.\n\n\n    Finally, for error analysis, we defer to the approximation error bounds provided in [1], as we directly employ their DataInf method to compute influence scores. To further validate the reliability of these scores, we compare them against alternative estimators, notably TracIn, and demonstrate its limitations as a proxy for layer quality in our setting (possibly owing to the Hessian being discarded). \n\n___\n___\n\n* **Minor Issues:** Thank you for pointing these out, and apologies for missing these prior to submission. We shall fix these small errors and typos. \n\n---\n___\n\nThank you once again for all your efforts in helping to strengthen our work. We hope that our comments helped address your concerns and look forward to engaging with you more during the author-reviewer discussion phase regarding any others that remain. \n\n___\n---\n\nReferences:\n\n[1]. Kwon, Yongchan, et al. \"Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models.\" ICLR 2024."}, {"review_id": "EVPUNWh5NP", "reviewer": "Reviewer_zyf6", "summary": "This paper proposes LayerIF, an influence function-based method for estimating the importance of individual layers across various downstream tasks. The method is evaluated through experiments on expert allocation in LoRA-MoE architectures and layer-wise sparsity distribution for LLM pruning, demonstrating its effectiveness.", "questions": "Please see Strengths And Weaknesses", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1.    The paper is well written, with clear motivations and a thorough explanation of the proposed methodology.\n\n2.    The connections to prior work are appropriately discussed, situating LayerIF within the context of existing research.\n\nWeaknesses and Questions:\n\n1.    In lines 32–33, the classification of activation-based methods as \"model-only heuristics or information\" may not be entirely accurate. These methods typically require training data to obtain activations for estimating layer importance. Therefore, shouldn't they also be considered at least partially data-centric, similar to LayerIF?\n\n2.    In Figure 1, the results lack context. What specific experiment or task do these results correspond to? What baseline are they compared against? Since the figure includes concrete values, instead of using some likely imaginary numbers, the authors should provide details about the experimental setup that generated these results.\n\n3.    The authors mention in lines 191–192 that “layers with lower cumulative positive influence exhibit less sensitivity to training data, indicating greater stability and training maturity.” This is an intuitive claim. Can the authors conduct a reverse experiment (e.g., prioritizing layers with higher influence scores) to empirically validate this hypothesis?\n\n4.    Another concern is the algorithmic inconsistency between expert allocation and pruning. Specifically, both applications use LayerIF only as an initial step, followed by distinct procedures for score calculation, normalization, or finalization. This suggests that LayerIF may not yet be a well-established or truly plug-and-play method for downstream tasks. Instead, different applications may require varying levels of task-specific post-processing to make effective use of LayerIF.\n\n5.    In lines 296–297, how are the 1.61% and 3.54% performance metrics calculated? Please clarify the source and computation of these figures.\n\n6.    Similarly, in lines 311–312, how are the reported improvements of 1.03% and 0.9% obtained? I couldn't obtain the same numbers.\n\n7.    The improvement from LayerIF in the model pruning experiments appears marginal. Compared to the uniform sparsity ratio, all reported gains are less than 0.5%, which raises concerns about practical significance.", "was_revised": true, "final_justification": "The authors have addressed some of my concerns during the rebuttal period. Although certain limitations remain — such as limited generality and seemingly marginal improvements — I recommend an overall score of 4.", "rebuttal": "We are grateful to the reviewer for their detailed feedback of our work, and are appreciative of their time, efforts, and consideration. Below we provide more discussion on the points raised.\n\n* **Partially data-driven analysis methods:**\n\n    We thank the reviewer for raising this important clarification. We agree that activation-based methods such as those proposed in [20–23] do typically require access to training (or validation) data to compute layer-wise activations, and therefore are not entirely “model-only” in the strictest sense.\n\n    Our original phrasing in Lines 32–33 aimed to distinguish **methods that rely primarily on internal model representations** (e.g., weight norms, activations, attention maps) from those that **explicitly quantify model–data interactions**, such as Influence Functions. While activation-based techniques indeed use input data to probe activations, they do not trace the *causal effect* of individual training examples on model predictions by analyzing the loss. In contrast, Influence Functions estimate the  impact on the loss of perturbing or removing training samples.\n\n    We will revise our language in the introduction to clarify this distinction and cite other papers that are more absolutely model-centric. A more accurate phrasing would be:\n\n    > “Many approaches take a model-centric perspective, examining how the model’s learned weights effect predictions [1,2]. Other methods use activations, or attention patterns to deduce internal mechanisms [3,4,5,6]. In contrast, Influence Functions offer a more explicitly data-centric perspective by quantifying the effect of individual training points on model's loss landscape.”\n\n    We appreciate the reviewer’s suggestion and will make this clarification in the final version.\n\n___\n___\n\n* **Clarifying Figure 1:** \n\n    Figure 1 is just a hypothetical illustration (not with real numbers) of our method against model only or heuristic based methods. We will add this clarification in the caption. \n\n    To improve the clarity of our method we will aim to provide a concrete step-by-step walk-through, in the revised version of the paper, as described below:\n\n    **Concrete walk‑through for expert allocation (CommonQ dataset, 32‑layer LLM, T=160 experts, beta=3)**\n\n    Below we provide the exact numbers produced by our implementation.\n    For brevity only the first 5 layers are shown; the remaining layers follow identically.\n\n    | Layer $i$ | Raw IF score $v_i$ | Inverted $ṽ_i$ | Power‑scaled $v̂_i$ | Fractional $f_i$ | Final experts $s_i$ |\n    | --------- | --------------------------------------- | ------------------------------------------------------------- | -------------------------------------------------------------- | ------------------------ | --------------------------- |\n    | 0         | -5.01×10¹¹                            | 5.01×10¹¹                                                        | 1.26×10³⁵                                                              | \t1.12                    | **2**                       |\n    | 1         | -8.36×10¹¹                            | 8.36×10¹¹                                                    | 5.84×10³⁵                                                    | 5.21                    | **6**                       |\n    | 2         | -7.99×10¹¹                            | 7.99×10¹¹                                                   | 5.10×10³⁵                                                    | 4.55                     | **6**                       |\n    | 3         | -8.46×10¹¹                            | 8.46×10¹¹                                                    | 6.05×10³⁵                                                    | 5.40                     | **6**                       |\n    | 4         | -8.16×10¹¹                            | 8.16×10¹¹                                                   | 5.43×10³⁵                                                    | 4.84                     | **6**                       |\n    | …         | …                                       | …                                                             | …                                                              | …                        | …                           |\n\n\n\n  (1) **Step1 (sign inversion).** Invert the negative values to positive.\n\n  (2) **Step2 (power transform).** Amplify disparities by raising to the power beta to get v̂.\n\n  (3) **Step3 (normalise to budget).** Scale v̂ so that the fractional allocations ∑fi equal **T−m=128** extra experts (m=32 layers).\n\n  (4) **Step4 (floor+redistribute).** Set si=⌊fi⌋+1, guaranteeing ≥1 expert/layer, then distribute the remaining 24 units to the layers with the largest fractional parts.\n\n  Finally, the complete allocation for **CommonQ** becomes:\n\n    ```[2, 6, 6, 6, 6, 5, 7, 6, 8, 7, 6, 7, 6, 6, 6, 5, 6, 6, 5, 4, 5, 5, 4, 2, 4, 5, 3, 4, 2, 3, 4, 3]```\n\n    **Interpretation.** The middle layers receive the most experts, indicating they are comparatively under‑trained for CommonQ; the first and very deep layers receive lower number of experts, reflecting higher estimated quality.\n\n    Similarly we will also provide an example for the layer-wise sparsity allocation setting. If the reviewer would like us to take any other steps to improve clarity, we are more than happy to do so.\n\n___\n___\n\n* **Reverse experiment**: \n\n    We thank the reviewer for this suggestion. We provide an ablation where we reversed our methodology by pruning the less sensitive layers more at a 70\\% sparsity budget on Mistral-7b-v0.1 and find that our design choice leads to improved performance over reversed allocation. We present the results below.\n\n\n\n    |             Method  | Magnitude | Wanda  | SparseGPT | Average  |\n    |---------------------|-----------|--------|-----------|----------|\n    | Our Method          | **33.439**    | **32.497** | **41.143**    | **35.693**   |\n    | Reversed Allocation | 33.101    | 32.015 | 38.450    | 34.522   |\n\n\n    We will add this ablation to the revised version of our paper. \n\n___\n___\n\n* **LayerIF having algorithmic inconsistency:**\n\n    We would like to clarify the generality of our approach. Influence Functions (IFs), by design, are diagnostic tools intended to inform model developers about the sensitivity of model components to training data. LayerIF extends this diagnostic capability to estimate per-layer quality in a principled, data-driven manner.\n\n    While it is true that expert allocation and pruning use distinct post-processing steps after the initial LayerIF score computation, this is by design: each downstream application naturally requires a different mapping from layer quality estimates to actionable decisions (e.g., allocating experts vs. assigning sparsity levels). However, the core LayerIF signal remains the same and is easily integrated across use case.\n\n     Importantly, LayerIF is not specific to the two tasks studied in this paper. It is a general-purpose tool that can be applied in the future to any task where layer quality matters (e.g. layer-wise unlearning). Its ability to improve performance across diverse tasks and compete against methods that are tailor to those specific tasks, in our view, is a strength rather than a limitation.\n\n___\n___\n\n* **Clarification on numbers:** \n\n    All these numbers follow the relative change formula of (New-Old)/Old * 100. We provide the calculations below:\n\n    * **1.61:** IF (`Top 25%`) and MoLA-8642. (82.27-80.97)/80.97 * 100. \n    * **3.54:** Average of IF variants and Average of baselines. (81.96-79.175)/79.175 * 100. This is a typo the correct value is 3.52. We will change this.\n    * **1.03:** IF SparseGPT and OWL SparseGPT. (60.61-59.99)/59.99 * 100.\n    * **0.90:** IF SparseGPT and AlphaPruning SparseGPT. (60.61-60.07)/60.07 * 100. \n\n    We hope this clears the confusion on how we arrived on these numbers.\n\n___\n___\n\n* **Improvement appears marginal:** \n\n    We appreciate the reviewer’s concern and would like to offer some clarification. While the absolute gains over the uniform sparsity baseline in the pruning experiments may appear numerically modest (under 0.5\\%), this level of improvement is consistent with recent state-of-the-art work that we compare against OWL (ICML 2024) and AlphaPruning (NeurIPS 2024).\n\n    Moreover, our method is not designed exclusively for pruning. Model pruning was included as one illustrative downstream task to demonstrate the applicability of LayerIF, not as the core objective of the paper. LayerIF is a general-purpose diagnostic tool, and its effectiveness across multiple settings speaks to its broader utility beyond the pruning context.\n\n    Thus, we believe that providing a unified, data-driven method that performs competitively across such diverse applications is a notable contribution.\n\n---\n___\n\nWe thank the reviewer for their comments and we hope that we provided sufficient clarifications. We hope to discuss further in the discussion period in case any others remain. Thank you. \n\n___\n---\n\nReferences:\n\n[1]. Yin, Lu, et al. \"Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity.\" arXiv preprint arXiv:2310.05175 (2023).\n\n[2]. Lu, Haiquan, et al. \"Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models.\" Advances in neural information processing systems 37 (2024): 9117-9152.\n\n[3]. Serrano, Sofia, and Noah A. Smith. \"Is attention interpretable?.\" arXiv preprint arXiv:1906.03731 (2019).\n\n[4]. Abnar, Samira, and Willem Zuidema. \"Quantifying attention flow in transformers.\" arXiv preprint arXiv:2005.00928 (2020).\n\n[5]. Wiegreffe, Sarah, and Yuval Pinter. \"Attention is not not explanation.\" arXiv preprint arXiv:1908.04626 (2019).\n\n[6]. Hao, Yaru, et al. \"Self-attention attribution: Interpreting information interactions inside transformer.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 14. 2021."}]

Author rebuttal:

Meta-review: The paper presents a novel approach of using influence functions to estimate the importance of different layers in a large language model (LLM). Although the idea of influence functions is not new, its application to LLMs as done in this paper is novel. Some applications of the proposed LayerIF approach, including expert allocation and pruning, have been discussed and experimented in this work too, which illustrates the usefulness of LayerIF in practice. I believe that LayerIF provides a new perspective for a more systematic understanding of LLMs, which is a very important problem. 

The reviewers had some comments especially about the clarity of some parts in the paper and experimental results. After discussions, all the reviewers recommended accepting this paper.
Decision: Accept (poster)