KDP: Simplifying Representation Dynamics in Kernel Space
Authors: Zeyu Ma
Wanying Wang
Guchu Zou
Mingang Chen
Jianhong Wu
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models; Model Compression; Structured Pruning; Kernel Space
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: This paper proposes a layer pruning method called Kernelized Dynamics Pruning (KDP), which simplifies the non-linear transformations between an LLM's consecutive layers by projecting them into a kernel space where they become approximately linear.

Abstract: This paper proposes Kernelized Dynamics Pruning (KDP), a novel layer pruning method from the perspective of simplifying representation dynamics within large language models (LLMs). Motivated by the high similarity between consecutive layer representations, we view the LLM's forward pass as a discrete-time dynamical system. We speculate that this phenomenon indicates the model's internal dynamics have entered a ``slow manifold'', which exhibits computational redundancy. Based on this insight, we project the representations into a kernel space where the complex, non-linear transformation between them is simplified to an approximately linear one. Then, a simple network learns the inverse kernel transformation, thereby enabling the pruning of the entire layer block. Both theoretical analysis and extensive experiments validate the effectiveness of KDP, demonstrating its superiority over existing pruning baselines.

Published as a conference paper at ICLR 2026
KDP: SIMPLIFYING REPRESENTATION DYNAMICS IN KERNEL SPACE
Zeyu Ma 1.2 Wanying Wang", Guchu Zou?, Mingann Chen", Jianhong Wu'* Shanghai Normal University
2 Shanghai Key Laboratory of Computer Software Testing and Evaluating Shanghai Institute of Ceramics, Chinese Academy of Sciences wujianhong@shnu. edu. cn, {mzy, wangwy, cmg) @sseenter....... sh. cn, zouguchuêmail aic.ac.co an
ABSTRACT
This paper proposes Kernelized Dynamies Pruning (KDP), a novel layer pruning method from the perspective of simplifying representation dynamics within large language models (LLMs). Motivated by the high similarity between consecutive layer representations, ical system. We speculate that this phenomenon indicates the model's internal dynamics have entered "slow manifold", which exhibits computational redun- dancy. Based on this insight, we project the representations into a kernel space where the complex, non-linear transformation between them is simplified approximately linear one. Then, a simple network learns the inverse kernel trans- formation, thereby enabling the pruning of the entire layer block. Both theoretical analysis and extensive experiments validate the effectiveness of KDP, demonstrat- ing its superiority over existing pruning baselines.
view the LLM's forward pass as discrete-time dynam-
INTRODUCTION
Large language models (LLMs) have demonstrated exceptional performance across various tasks and domains Touvron et al.) 2023 Grattafiori et al. 2024 Yang et al. 2025 Liu et al. 2024). How- ever, models increase in scale, the substantial computational and hardware deployment costs pose significant challenges, limiting their broader application in real-world scenarios. Pruning is a pri- mary technique for model compression aimed at alleviating the aforementioned problems (Louizos) et al., 2017: Xia et al. 2023 Sun et al., 2024 Ma et al. (023). Among various pruning methods, layer pruning is garnering increasing attention because it naturally leads inference acceleration and model size reduction without requiring special handling.
The prevailing layer pruning paradigm involves removing redundant layers (Song al. 2024 Men et al. 2024) or substituting them with compact modules (Yang et al. 2024 Chen et al. 2025 with a primary emphasis on the choice of pruning locations and the techniques for performance recovery (e.g., fine-tuning or distillation). However, this line of work largely overlooks the fundamental properties the model's internal dynamic flow, thereby leaving the potential simplification patterns unexplored, see Appendix C for details related work. We suggest that pruning be viewed not only as a method to "construct" a smaller sub-network, but also as a process to "search" for a geometric viewpoint that reveals the inherent simplicity of complex dynamics.
LLMs exhibit a notable characteristic: their consecutive layers tend to learn highly similar repre- sentations (Ding et al.; 2025). As demonstrated in Figure we employ both the centered kernel alignment (CKA, Kornblith et al. 2019) and the cosine similarity metric to measure the similarity of layer representations from the perspectives of the original space and the kernel space, respectively, and find that multiple consecutive layers exhibit consistently high similarity. This observation raises a key research question: Does representational similarity equate to computational redundancy which can be substituted by a simple function?
"Corresoonding author; Jianhong Wu

---

Published as a conference paper at ICLR 2026
Viewing the forward pass of LLM discrete-time dynamical system, high representational similarity signifies the system's 's entry into a "slow manifold", characterized by a small velocity. We assume that in such regions, the system's 's short-term evolution can be described by much simpler function (e.g., a linearized first-order approximation), rather than requiring the nonlinear dynamics of a complete Transformer block. However, substituting the non-linear forward pass with linear dynamics may result in the loss of fine-grained details, which could contain low-variance, task- relevant information (Cloos al. 2025). Notably, Figure[[show that the similarity measured in the kernel space is substantially higher than that in the original space. This observation suggests that the kernel space possesses stronger capability in modeling high-dimensional relationships, consistent with the intuition behind SVMs (Cortes & Vapnik, 1995). Consequently, it exhibits higher similarity and suggests a potential for linear simplification. Motivated by this, we aim to identify Hilbert space in which the complex dynamics become amenable to linear approximation, thereby enabling effective layer pruning via linear simplification in kernel space.
Based on these insights, we propose Kernelized Dynamics Pruning (KDP), simplifying repre- sentational dynamics within a kernel space to reduce computational redundancy. Specifically, our method proceeds in the following two steps. First, jointly optimize a learnable kernel trans- formation and the linear coefficients mapping between layers. Second, an inverse transformation network maps these representations from the kernel space back to the original space, reconstructing the original layer to enable model pruning.
We validate the effectiveness of our method through both theoretical analysis and extensive exper- iments. First, we present theorem providing an error bound for approximating multi-layer rep- resentations with linear transformations in the kernel space, thereby demonstrating that cross-layer representations can be linearly approximated within this space. We further show that the linear ap- proximation capability in the kemee space surpasses that in the original space, providing theoretical foundation for KDP. Extensive experimental results empirically answer our guiding research ques- tions, demonstrating that exploiting representation similarity significantly simplifies computational processes by eliminating redundancy. KDP learns simplified internal representational dynamics in kernel space using only localized representation supervision, without requiring fine-tuning of the entire model downstream tasks. This method enables effective modeling with limited data and obviates the need for post-training to restore performance. Our main contributions are as follows:
We reformulate layer pruning as the search for an optimal geometric embedding within a Repro- ducing Kernel Hilbert Space and propose Kernelized Dynamics Pruning (KDP) method, which linearizes representational dynamics using the kernel trick to simplify consecutive layers.
We provide a theoretical error bound for lincarization in the kernel space and demonstrate the modeling and linear simplification advantages of the kernel space, with experimental results cor- roborating the theoretical analysis.
We conduct extensive experiments on 15 benchmarks, demonstrating that our method maintains superior performance while requiring only a small number of trainable parameters and limited calibration data, eliminating the need for additional post-training.
2
PRELIMINARIES & THEORETICAL ANALYSIS
2.1 PROBLEM SETUP& & KERNELIZATION
LLM Forward Formulation. Consider an LLM composed of a sequence of N Pre-Norm Trans- former layers, Fi,...,FF FN. Given an input x. for the (-th layer, the forward pass is written as
h (x) hx(x) + ((Norm(h
Here, Norm(.) denotes the Layer Normalization function, hr(x) E denotes the I-th layer repre- sentation and (-). denotes the backbone function of the Transformer layer composed of multi-head attention (MHA) and feed-forward neural network (FFN). For simplicity, write h for h(-)) when
the context clear.
Kernel Model. A kernel is a function k(x,y) y) that measures the similarity between two data points x and y. is formally defined as an inner product in a feature space H. Gaussian Radial Ba- sis Function (RBF) kernel (i.e., k(x,y) exp(->|lx y)(²)) is the canonical choice due to
2

---

Published as a conference paper at ICLR 2026
Sieilarity (COSN))
Sivllarity Nattic (OOA)
Algorithm 1: Kernelized Dynamics Pruning
1: Require:
Pretrained L.M E calib. set D, RFF dim m., r, block length Kaaxe: budget
stability "y. weight W. training iters T.
Lot @ be the set of B pon-overlapping model blocks maximize
Fi-t+ Fmn+KKax CKA(b; ; hy-
similarity score
3: procedure JOINTKERNELLINEARDE (I,-.. .I Kmax) E 9 do
Detine so. operators (A.) for minibarch C D
(())++(B)) + (TT A.) Po (b, (B))
8:
C(-) Linn Leas
Update (A.)
10:
return (0(9, (A(1) (1) hieg
11:
procedure TRAININVERSENETWORRS for (l, d- Kmax) do
12:
13:
Define inverse network R2
14:
for minibatch C 12 do
WAM
15:
h1+2 (p(h)++))
16:
C(-) fuse fuorm
17:
Update $
18:
return f0' hec
19:
procedure FOLDBLOCK
20:
Define 5(h) <- Tix (II'=, A.) po(())
Figure 1: Inter-layer Cosine and CKA Similar-
21:
Replace F....., Fi+s with
ity of 3 LLaMA LLMs.
its universality and consistency (SasisttecySteinwwarr Steinwart 2002 In this paper, we employ learnable Random
Fourier Features (RFF) kernel to learn a data-driven, anisotropic Gaussian RBF kernel (Rahimi &
Recht (007). which enhances representational capacity and reduces computational cost. The ker- nel is approximated by the inner product of low-dimensional feature map sp.) Rd + 2m
y(x,y) x p(x) T "py),, where cxx) is defined as:
1 <(x) = Vm
(cos (WTx x b) sin (W x + +b)")²
(2)
Here the columns of weight matrix W dd are i..d samples drawn from (00 E). and the ele- ments of bias vector b E R are i.i.d. samples drawn from U(0, 2m). In contrast to the original RFF kernel, which relies on a pre-defined spectral distribution, here the covariance matrix is learnable and parameterized Σ D LL D diag(exp(X) is a diagonal matrix parameterized by learnable vector À E Rd and L E gdxr is a learnable low-rank factor matrix, where the rank is hyperparameter that satisfies < d. Equation 2 effectively learns anisotropic Guassian RBF kernel k(x,y) = cxp (x - y) - y)) whose distance metric is adapted to the data.
Symbol Denotation. In this paper, the 4p-norm of a vector v = -Ui, (m,) is denoted by || A Em 1/,, with the subscript omitted for the standard fn-norm (i.e., ||v|| A ||v[]2), the operator norm for matrix is denoted by || llop- (x)+ max 0} denotes the positive part of vairable v, and O-)) denotes the standard Big-O notation for an asymptotic upper bound. For detailed definitions of technical machine learning terms, such as ERM and population risk, please refer to the Appendix
2.2 FORWARD DYNAMICS ON A SLOW MANIFOLD
The residual connection described in Equation motivates us to view the Transformer's forward pass discrete-time dynamical system. The term fi(-) can be seen "velocity" "update" vector that perturbs the current state h,, thereby producing the state of the next layer, h
As shown in Figure consecutive layer representations in LLMs exhibit strikingly high similar- ity. Within this dynamical framework, high similarity between h441 and hy is directly equivalent to the update vector having small relative norm, i.e., |[ (Norm (h:))ll < ||bı]], indicating that the
3

---

Published as a conference paper at ICLR 2026
system's trajectory has entered "slow manifold". If the transformation between layers is minor, the local dynamics of a Transformer block fi could potentially be approximated by a far simpler function. This is a common practice in fields such as the numerical analysis of partial differential equations (PDEs) and model order reduction (e.g. (Murray 2007). more detailed analysis can be found in the Appendix [H). While first-order approximation (i.e., local linearization of the dynam- ies) is a natural approach, the Transformer's forward pass is fundamentally nonlinear. Applying such an approximation directly in the original representation space would therefore lead to signifi- cant information loss, as it would neglect crucial nonlinear feature interactions. Note that in Figure IJ the CKA similarity generally exceeds cosine similarity, which is consistent with the property that the high-dimensional representation induced by the kernel space captures complex representa- tional relationships more effectively. Therefore, we assume that there exists a kernel function yp-) that induces Hilbert space where the layer-wise transformation becomes approximately linear, i.e., sp (h(+1) (pp)))).
2.3 THEORETICAL RESULTS
As argued in Section 2.2 when adjacent layers exhibit high representational similarity, the forward propagation enters a slow manifold where a simpler rule can capture short-horizon dynamics, an approximation that becomes more accurate in suitable kernel space. In this section, we present the theoretical formalization of these arguments. First, we derive the error bound for linear fitting of multi-layer representations in the kernel space, demonstrating that consecutive layers with highly similar representations can be reduced to linear transformations in this space. Subsequently, we prove that the kernel space exhibits superior fitting capacity for layer simplification compared the original representation space. The detailed proofs are provided in Appendix [
Theorem (Kernel Linearization Error Bound). Let Po Rd - R22m be a feature map such that li (b)ll2 I Rup for all representations h. Let A; be linear operators for the i-th step transition with ||Aillop I BA. Let (0, (A.)) be the Empirical Risk Minimizationi ERM) solution on an n- sample training set, and let LERM denote the minimum empirical one-step squared error. Then for any 8 E (0, 1), with probability at least 1 - 8 over the draw of the training set, for every starting layer and horizon k 2 1, the k-step error E0, (A.)) = -Po (11 A.) so (hi)
satisfies
2m log (2m/8)
(, (A:))l <
V LERM + CB3 R2
BR 1-- B
(b)
Theorem [ yields two-factor bound on k-step error: a single-step term (a) that captures the linear model's s in-kernel fitting ability plus its generalization gap, and a multi-step accumulation factor (b) that governs how errors accumulate across replaced layers. Theoretically, for the bounded parame- ters Rs and BA, the error bound converges at a rate of ((1/Vn) with the sample size n. In sub- sequent experiments illustrated in Figure 5 we establish empirical values for BA and Rg demon- strating that these bounds are well-behaved, i.e., they are significantly smaller than vni. Therefore, Theorem I demonstrates that consecutive layers can be well simplified linear transformations in kernel space.
Further, we can prove that compared to learning linear approximation in the original representation space, the kernel space yields a smaller linear approximation error. Before that, the fitting capability of kernel space is given in the following lemma.
Lemma 1 (Kernel Existence). For any given >> O, there exists Random Fourier Feature map pp R2dd + (defined by suitable parameters W, b, and a sufficiently large dimension 2m) and a corresponding linear operator A, such that: En,-D [llip (h1+1) - Aip (h/)lll <
The proof of Lemma [ is a direct application of the Universal Kernels (Micchelli et all 2006). Next, our analysis begins with two function classes. Consider the following two function classes at the (-th layer of an LLM: the identity class of the original space FiD == {z + Ar: l|A I BA), with a population risk of (D(AA = E [[ Ahi the RFF class FRFF == {a + (x)( || S BA, [|x(())) R2p 0 E {}, with population risk of
4

---

Published as a conference paper at ICLR 2026
(R, A) E (bbta) App (hi))l Note the minimum population approximation error of the two classes as:
apxin = inf SBA
(A)(A).
aPERFF =
@E BA.ll))))sA RREE (0, A).
inf
Let Am and (0, ARFF) be the ERM solutions on the respective classes. We can get the following conclusion on the superior fitting capacity of Kernel space.
Theorem (Kernel Advantage). Lel A = apx that for RFF dimension 2m > mm' and
For any there exist m', cc > such
2
16
CB2
2cm Re 2m log
R2
dog ad
8
(4)
A
the following holds with probability at least - 8: R RRF ARFF) S Rm(Am)
Theorem establishes that in the kernel space, a well-trained RFF with dimensionality greater than 2m achieves lower population risk compared to simplification in the original representation space. This result further substantiates the necessity of performing linearized model compression in the kernel space. Experiments in Table empirically validate the superiority of the kernel space over the original representation space.
3 KERNELIZED DYNAMICS PRUNING
As argued Section 2 on both intuitive and theoretical grounds, representational similarity implies computational redundancy, so that the complex module can be simplified better in kernel space. Building on this foundation, this section details our pruning method, Kernelized Dynamics Pruning (KDP). The algorithm is presented in Algorithm
3.1
KERNEL LINEARIZATION JOINT TRAINING
The first step is to identify which parts of the network to prune. Inspired by prior work (Chen 2025 Ding al. 2025 Gromov et al., 2025), we focus on replacing blocks of multiple consecutive Transformer layers. To control the cumulative error as shown in Equation I we set the consecutive layers be replaced with maximum length Kmax- For each block. we first compute the CKA similarity between the output representations of the first and last layers of consecutive (e.g..Ding blocks. Following standard practice (e.g., Ding al.] 2025), we exclude the initial and final 10% of the model's layers from consideration, as they empirically known to be sensitive to pruning. Combining these criteria, we rank all eligible blocks according to their CKA scores and advance the highest-scoring candidate the kernel linearization training stage.
For the candidate block for pruning, we learn a replacement module by jointly optimizing the learn- able RFF kernel parameters and the multi-step linear operators {A.) sample E D, the estimate (Ø, (A:)) is given by:
Kmax, For each training
Kyna
arg min (.(A,)
Σ [[A (h)=--(xx)) - (Ir+s(x)) 1|
(5)
1=1
RED
+ - W @ cos (Arpe (b(+)-1(x)) so (hi+s(z))))).
Here, @ denotes element-wise multiplication. The loss function is composed of two terms: a re- construction loss and weighted cosine similarity loss. The latter designed to encourage a finer- grained alignment of the representations' geometric structure. Specifically, this cosine similarity term is modulated by a weight matrix W, which assigns greater importance to tokens based on their position, prioritizing those towards the end of the sequence. To improve convergence stability, A; parameterized as A, = BBB that preserves the additive nature L.M''s s forward pass (see Equation[]] where the scalar and the matrix x B, trainable parameters. In addition, initialize A; at the start of each iteration with its Ordinary Least Squares (OLS) estimate in the current RFF kernel space accelerate training.
5

---

Published as a conference paper at ICLR 2026
Kernelized Dynamics Pruning
Step 1: Linearization in Kernel Space
F
Original Space
Kernel Space
by by-1 buy Kase
pp(-)
busi
Kernei Mapping
Transformation
Lineor Openators A
Network
bi
Kernel Linear
Magsing
Stop 2: Back to Original Representation
((b,.)) (IEL, (1(b))
MLP I(-)a
h Kuse
Inverse Transformation Network
Overall illustration of KDP. KDP replaces a multi-layer Transformer block via a two-step process: first, it projects and linearizes non-linear representations in kernel space; second, it lears an inverse mapping from the kernel space back to the original space using a simple network.
Figure 2:
3.2 INVERSE TRANSFORMATION NETWORK
Once the joint training converges, we can get k-step prediction in the kernel space, i.e., ((hr+*) (IT A (hi). However, this output must be mapped back to the original space for layer replacement. Thus, an inverse transformation network, I(-) 2 + Rd, takes the kernel-space prediction (h)++*) input, to reconstruct representation in original space hetk, denote h The estimate @ given by:
arg min
Σ
I ( (bu+4 (o))) (x)2
We parameterize the inverse transformation network as a two-layer MLP with a sealar scaling factor a, i.e. T(x) MxP(x). We observe that the iterative application of the linear operators {A,} the kernel space can lead to a significant decrease in the representation' norm. The scaling factor explicitly compensates for this norm decay, enabling more stable and efficient recovery of the original representation' s scale.
EXPERIMENTS
4.1
SETUP
Models. Our experiments are conducted on 6 popular open-source models: LLaMA2-7B, LLaMA2- 13B (Touvron et al. 2023), LLaMA3. 1-8B, LLaMA3-SB (Grattafiori al. 2024). OPT-2.7B and OPT-6.7B. (The results for the OPT model are presented in the Appendix E Following prior work, prune the layers that account for approximately 25% of the original parameters.
Training Datasets. For training, we construct composite dataset of 4000 samples by randomly selecting 500 samples from the training set of each of the following datasets: PIQA Bisk al. 2020). CMMLU (Yüksel et al.) 2024), BoolQ (Clark et al. 2019). C3 (Sun et al. (020). MNL Williams et al. 2018), Race- High/Hiig/id (Lai et al.) 2017). and SlimPajama (Shen al. 20244.
Benchmarks. For a comprehensive evaluation, we assess the models on 15 standard datasets. We group the evaluation tasks into two categories: generation and classification. For the classification, the evaluation is conducted on the test sets of the aforementioned datasets, excluding SlimPajama, and supplemented with additional datasets: CHID (Zheng et al.] 2019). HellaSwag (Zellers) al. 2019). CoQA (Reddy et al. 2219), WSC (Levesque al. (012). SST2 (Socher et al. 2013), and C3 (Sun et al. 2020). We also evaluate 3 benchmarks for generative capabilities. For WikiText
MMeriy al. 2017 7 and C4 (Raffel al. 2020), we evaluate their Perplexity (PPL), and for XSum
Narayan et al. 2018), we evaluate ROUGEI. We use the LM Evaluation Harness (Gao et al. 2024) with detault parameters in our evaluations.
Baselines. Beyond the dense model, we consider six structured pruning methods: ShontGPT MMen al. 2024) and SLEB (Song et al. 2024), which prune the network directly via metrics without

---

Published as a conference paper at ICLR 2026
retraining, and LaCo (Yang al., 2024). Streamline (Chen et al. 2025), SliceGPT (Ashkboos al. 2024]. and LLM-Pruner (Ma et al. 2023), which first prune or replace network components and then employ a retraining phase to recover performance.
4.2 MAIN RESULTS
Table 1 and Table show the effects of different pruning methods on classification and generative tasks. The two tables present detailed scores on each benchmark, the overall average, and the re- tained performance rate. In the tables, "Dense" refers the original model and "w/o kernel" is an ablation of our method that only removes the identified layers. "Ours" denotes our proposed method, while "Ours!" refers to our method after being fine-tuned using Parameter-Efficieen Fine- Tuning (PEFT). (Details of PEFT are provided in the Appendix B Table[] that our method's retained performance surpasses that of the best- pest-performing baseline by 9.1%, 8.3%, and 9.3% on LLaMA2-7B, LLaMA2-13BB and LLaMA3-8B, respectively. Our method also achieves comparable performance on generative tasks. Notably, requires no retraining phase to recover performance.
It is worth noting that on the SST-2 dataset, simple binary classification task, existing methods show a notable performance degradation despite performing well on other complex benchmarks. We suppose that this phenomenon arises because simpler tasks rely more heavily on coarse-grained information, which prior methods tend to prune inadvertently, thereby leading to instability. con- trast, our method consistently maintains its effectiveness, indicating its superior capacity preserve the model's s essential capabilities.
The comparison with the "w/o Kernel" demonstrates the effectiveness of KDP. While directly prun- ing consecutive layers leads to a significant degradation in performance, our method of simplifying these p kerpel space Lppcce yields substantial gains. Specifically, it boosts the average retained performance by 31.9%, 23.1% ,, and 18.9% percentage points for the three models, respectively.
be learned within the kernel space. Figure 4 presents compari- son between the predicted representations and the ground-truth representations viewed in the kernel space. Specifically, we feed sample from the CMMLU dataset into LLaMA2-7B, where layers 10 through 14 have been replaced by KDP. RFF kernel projects the 14-th layers' output representations into a (sequence length 1024) matrix. To visualize these representations, we take rows 0, 25, and 50, reshape each into a 323 x 32 matrix, and render them as heatmaps. As observed, minor deviations tend to increase with the sequence position, which motivated our weight-scaling design in Equation 5 To this end, the predicted representations closely match the actual ones. Notably, KDP not only captures the general trends of the representations but also accurately fits outlier points, which are eritical model performance (Sun et al. 2024). This highlights KDP's ability to effectively learn and preserve essential patterns of the representations in the kernel space.
Representational patterns
As training progresses, BA and R& behave well. Figure 5 illustrates the evolution of the operator norms for } and (p()) during the training of two LLMs. R&p remains consistently below 1.5, while BA decreases rapidly before converging. These observations validate the effectiveness of the error bound presented in Theorem [
Step 1 dominates the pruning performance, while Step 2 serves primarily as the inverse kernel mapping. Figure g intuitively illustrates the effectiveness of kernel-space fitting in Step 1. To gain deeper insights into the KDP process and identify the component responsible for high performance, we analyze its training dynamics in Figure 3 The left panel shows that in Step 1, the loss drops sharply and converges within approximately 100 epochs. Concurrently, the middle panel depicts the cosine similarity between the predicted and ground- truth representations increasing as the loss decreases, confirming effective learning in the kernel space. By contrast, the right panel shows that
Stap1 Loss
Stap1 Cosine Similarity
Stap2 Loss
1.0
103
30a
0.5
10
Tigal apor
100
2000
300
4000
500
100
200
300
400
3500
20
40
20
800
150
Epoch
Epoch
Epach
Figure 3: Training loss on LLaMA2-13B. From left to right, the panels show: (a) the loss curve during Step (b) the cosine similarity during Step 1, and (c) the loss curve for Step 2.
7

---

Published as a conference paper at ICLR 2026
Table I: Performance comparison of different baselines on classification benchmarks. u. indicates the results in the original paper. "f" indicates the results after fine-tuning. Average (Avg.) represents the arithmetic mean of Accuracy across all datasets. Retained performance (RP.) represents the percentage of the original model's performance retained by the pruning method.
CMNLI
HcSw
PIQA
CHID
WSC
CoQA
BoolQ
MMLU
CMMLU
Race
SST2
LLM
Method
Ratão
Avg.
RP:
Dense
0.0%
34.9
73.3
79.7
41.6
81.2
66.2
72.3
48.9
30.8
42.8
93.2
44.8
59.14
100.0
SLEB
20.1%
29.7
63.3
66.1
19.3
75.7
48.5
68.2
24.2
26.4
28.8
78.8
35.3
47.02
79.5
ShonGPT
27.1%
31.3
51.6
66.3
25.5
70.9
49.0
59.6
34.4
28.5
37.7
68.4
35.9
46.59
78.8
LLaMA 2-7B
LaCo"
27.1%
34.4
55.7
69.8
36,1
40.4
45.7
64.1
26.5
25.2
23.6
39.7
41,93
70.9
Streamline
27.0%
32.9
54.3
63.6
26.9
70.7
42.5
64.3
37.7
29.5
34,9
90.5
32.6
48.37
LLMPreeee
71.4
25.7
SliceGPT
25.4%
31.9
49.9
68.7
16.7
48.6
28.2
25.2
78.3
30.7
43.54
SlizeGPT
25.4%
32.1
53.3
69.2
20.4
61.2
44.4
61.9
29.7
22.5
26.7
87.1
28.7
44.77
75.7
w/o Kemel
24.8%
30.8
34.5
57.1
12.8
69.1
11.2
50.3
25.3
24.8
20.5
51.5
23.5
34.28
58.0
Ours
22.8%
31.6
65.1
70.1
27.2
73.9
45.4
71.6
44.7
28.
39.9
94.0
43.7
53.11
89.9
Ours
22.8%
34.6
63.0
68.8
29.5
74.8
49.0
72.3
41.5
26.7
40.4
92.3
37.3
52.52
88.8
Dense
0.0%
48.1
74.4
77.5
47.9
88..5
65.6
73.9
57.7
38.9
39.7
93.7
47.5
64.45
100.0
SLEB
19.5%
34.7
63.1
67.2
39.0
77.7
51.7
52.
25.6
24.6
53.5
77.9
31.7
49.9
77.4
ShonGPT
24.6%
31.7
61.5
72.5
38.4
70.0
$1.0
62.8
54.4
33.7
57.3
76.1
45.0
$4.53
84.6
LLaMA-2 2-13B
LaCo
24.6%
32.9
64.4
74.3
40.1
52.9
52.7
45.9
32.6
55.6
44.9
50.94
Streamline
LLMPruner
24.4%
27.8
55.3
61.9
30.7
63.4
49.7
53.0
23.1
20.0
65.8
30.3
42.03
SliceGPT
23.6%
33.2
53.1
60.8
18.4
74.8
44.3
36.6
25.1
25.7
22.4
68.3
26.1
40.73
63.5
SticePPT
23.6%
31.5
45.9
60.3
18.9
67.1
39.7
37.8
29.1
28.4
23.4
78.0
26.9
40.58
63.0
w/o Kemel
24.3%
32.8
63.1
67.5
33.6
72.5
19.3
60.6
35.1
27.1
33.3
61.7
33.5
45.01
69.8
Ours
23.4%
40.0
64.1
74.5
39.1
82,5
62.5
69.4
52.7
37.1
58.2
89.1
49.5
59.89
92.9
Ours
22.8%
41.2
63.5
73.9
40.2
63.8
63.3
$1.9
38.0
55.6
87.8
48.5
58.98
Dense
0.0%
33.9
75.7
77.9
72.8
86.4
72.5
75.3
67.2
50.1
73.5
93.0
62.1
70.03
100.0
LLMM 1-33888 SB3
SLEB
19.0%
33.0
65.1
63.0
28.0
74.6
31.5
67.7
49.3
27.3
24.1
77.9
37.2
48.23
68.9
ShonGGP
65.4
73.0
44.8
34.2
82.3
44.1
73.8
24.0
Ours
25.8%
33.9
61.5
73.5
23.9
83.6
37.7
71.0
55.5
34.0
38.4
81.4
38.9
$2.77
75.4
Ours
22.8%
30.1
57.5
74.1
25.1
84.4
33.0
62.4
50.4
38.5
34.0
68.3
35.5
49.44
70.6
Table Performance comparison of different baselines on generation benchmarks. For Perplexity, lower score means better performance (L), while for ROUGE, higher score means better (t).
LLaMA2-7B
LLaMA2-130
LLaMA-3.1 3.1--B
Models
RP(PPL.)
RP(Rouge)
WIKT
C44
Xsum?
WIKT
C41
Xsum?
WTKL,
C41
Xsum?
Dense
7.0
19.2
4.9
6.5
22.7
6.1
8.8
27.2
100%:
100%:
Streamlinet
17.1
19.7
43.7
81.7
19.5
377.1
201.1
21.5
5.3%
87.8%
SLEB
12.9
19.3
10.0
33.3
45.5
17.3
55.8
79.6
13.1
15.7%
58.5%
o/o Kernel
13.2
17.4
12.8
20.7
27.9
16.0
120.7
103.6
17.5
12.8%
67.0%
Ours
13.5
15.3
18.0
205.7
113.6
21.8
10.1%
79.7%
Ours
8.3
9.3
17.0
13.2
15.0
19.3
31.7
14.6
20.2
42.1%
81.8%
8

---

Published as a conference paper at ICLR 2026
h
Target Layers
op of LaMV
Original Space/
Space
Poipenunn SILLIAA 388
Figure 4: Kernel space representation visualization. The heatmap panels show (from left to right) the pre- dicted representation, the ground-truth representation. and their difference for sample from CMMLU. The color range is unified for the first two panels to high- light their similarity.
240
Figure 5: Operator norms of (p() and {A,} during training. The line corresponds to ||A/llop: the shaded region is the norm of (1 for the shared kernel space, cal- culated from mean-subtracted samples.
Model Performance(Ac)) Comparison Pruning
Modal Parformance(PPL) Comparison Across Pruning
Poses
JPe Stda
Modes
Lasttea
LLrM0-2.119
Acturacy (56)
60.03
304.1
27.30
20244
13.02
220.5
20.29
198.
26.75
0%
10%
20%
3.10
49%
30%
08 10%
20%
30%
40%
50%
Pruning Rate
Pruning Rate
Figure 6: Performance of CMMLU and WIKI when we prune with increasing pruning ratio.
the loss in Step 2 converges within only 20 steps, suggesting that the inverse transformation network I carries out a relatively simple task. We also conduct an ablation study to evaluate the performance impact of barely training for each step, with the results detailed in the Appendix FF] In conclusion, these results demonstrate that Step is the primary contributor to the pruning performance of KDP.
4.4 ABLATION STUDY
Different Pruning Ratios. As shown in Figure 6 evaluate method the CMMLU and WIKI Text across various models and pruning ratios. The method exhibits strong retained perfor- mance on CMMLU, maintaining over 90% of the original performance at pruning rate of around 28% and over 80% at rates exceeding 50%, which confirms its robustness. The results show that the models' overall performance degrades linearly with an increasing pruning rate. However, on generative tasks, the PPL behaves differently across models. Specifically. LLaMA3-8B exhibits rapid increase, whereas the PPL for LLaMA2-7B and LLaMA2-13B remains relatively stable.
Different Linearization Methods. To validate the importance of the kernel transformation, we compare KDP against a simplified baseline that performs linear fitting directly in the original rep- resentation space. As shown in Table both OLS- methods exhibit drastic degradation in performance on the LLaMA3-8B model. contrast, KDP significantly outperforms both baselines across all tasks. These results provide strong evidence that the transformations between Transformer layers are highly non-linear. Linear approximation in the original representation space, as attempted by OLS, is incapable of capturing these complex dynamics, resulting in severe information loss. This observation further corroborates Theorem demonstrating the necessity of learning linear transformations in the kernel space.

---

Published as a conference paper at ICLR 2026
Different Max Length. To analyze the impact of maxx incrementally prune layers from LLaMA2-7B and compare two strategies for block selection: expanding a fixed initial block or choosing
based on CKA similarity. Table demonstrates method's robustness in simplifying short-term dynamics. Model performance is stable when replacing few layers (aax S layers replaced. This observed non-linear degradation strongly supports our theoretical analysis (Theorem D which shows that the approximation error grows exponentially with the number of layers, leading instability.
3) but degrades catastrophically
Table Comparison with direct linearization the original space via OLS on LLaMA3-888
Table 4: Ablation study on max length K max-
Kmxr
Pruned Layers
HeSW
CoQA
MMLU
Method
HeSW
CoQA
MMLU
[6.7]
48.5
[6,7,8]
71.0
Dense
75.7
72.5
67.2
[10,11,12]
71.3
65.1
47.7
Direct OLS
31.3
0.0
26.7
[6,7,8.9]
65.1
57.8
43.8
[12,13,14,15]
66.1
61.6
44,9
Sequential OLS
30.7
0.0
28.2
[6,7,8.9.1]]
47.6
41.8
24.0
Ours
61.5
37.7
55.5
[11.12.13.14115
43.2
40.4
27.3
CONCLUSION
In this paper, we introduce a new perspective on layer pruning that simplifies the information flow by operating within kernel space. Based this perspective, we propose novel method named as KDP. We demonstrate the effectiveness of our method through both theoretical analysis and em- pirical experiments. Our future work includes exploring more kernel functions and adapting our pruning method for multimodal LMs.
REPRODUCIBILITY STATEMENT
We have taken several measures to ensure the reproducibility of our work. A detailed description of our proposed framework, algorithms, and experimental settings is provided in the main text and Appendix B Additional implementation details and ablation studies are included in the paper. For theoretical results, present all assumptions and complete proofs in the Appendix. All datasets used in our experiments are publicly available. Furthermore. we submit an anonymous link to the source code and scripts for reprodueing our experiments.
ACKNOWLEDGGENT
Wu gratefully acknowledges the financial support from the National Natural Science Foundation of China (Grant No. 72173086).
This work is also funded by Science and Technology Commission of Shanghai Municipality Pro- gram, China (No. 240.24022202100 and Shanghai Municipal Commission Economy and Informati- zation, China (No.R7-RGG2NN00-25--1666
REFERENCES
Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation-Baa Based Adaptive Struc- tured Pruning for Large Language Models, Proceedings of the AAAI conference on Artificial Intelligence, pp. 10865-10873, 2024.
Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari Do Nascimento, Torsten Hoefler, and James Hensman. SliceGPT: Compress Large Language Models by Deleting Rows and Columns. In The Twelfth International Conference on Learning Representations, 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, and others. PIQA: Reasoning about Phys- ical Commonsense in Natural Language. In Proceedings of the AAAI conference on Artificial Intelligence, volume 34, pp. 7432-7439, 2020.
10

---

Published as a conference paper at ICLR 2026
Xiaodong Chen, Yuxuan Hu, Jing Zhang, Yanling Wang, Cuiping Li, and Hong Chen. Stream- lining Redundant Layers to Compress Large Language Models. In The Thirteenth International Conference on Learning Representations, 2025.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lingnistics: Human Language Technologies, volume 1, pp. 1924-2936, 2019.
Nathan Cloos, Moufan Li, Markus Siegel, Scott L. Brincat, Earl K. Miller, Guangyu Robert Yang, and Christopher J. Cueva. Differentiable Optimization of Similarity Scores Between Models and Brains. In The Thirteenth International Conference on Learning Representations, 2025.
Corinna Cortes and Vladimir Vapnik. Support- Networks. Machine Learning. 20(3):273-297, 1995.
Xuan Ding, Rui Sun, Yunjian Zhang, Xiu Yan, Yueqi Zhou, Kaihao Huang. Suzhong Fu, Angelica I Awiles-Rivere Chuanlong Xie, and Yao Zhu. A Sliding Layer Merging Method for Efficient Depth-Wist Wise Pruning in LLMs, 2025. URL //tptpp::///xxv.../// org/ abs /502 19159
Elias Frantar and Dan Alistarh. SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot, In International Conference on Machine Learning, volume 202, pp. 10323-10337, 2023.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Beflding,Jfffree Hsu, Alain Le Noac'h, et al. The Language Model Evaluation Harness, 2024. URL/https://zenooae wwtp:/oooorrrr////e//////oo////e
Yuri Gardinazzi, Karthik Viswanathan, Giada Panerai, Alessio Ansuini, Alberto Cazzaniga, and Matteo Biagetti. Persistent topological features in large language models. In International Con- ference on Machine Learning, 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The Llama Herd of Models, 2024. URL https: /ttpss//aarxvvcoom www.wiwww..aaaaaa..a.aa. wttp:////aa////...2//1/
21788
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. The Unreasonable Ineffecttivnness of the Deeper Layers. The Thirteenth International Conference Learning Representations, 2025.
Lumer Günter. Bochner's theorem, states, and fourier transforms of measures. Studia Mathematica, 46(2):135-140, 1973. htRl:htt://eeet
orgg/occ//1776
Shwai He, Guoheng Sun, Zheyu Shen, and Ang Li. What Matters in Transformers? Not All Atten- tion is Needed, 2024. URL htRl:tttt::////
org/ abs 15786
Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Blyoung-Kyu Song. Shortened LLaMA: A Simple Depth Pruning for Large Language Models, 2024. URL httpt:::t//aattitee: orq/ abs/ 2202244222220 02834
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of Neural Network Representations Revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMIR, 2019.
Vedang Lad, Jin Hwa Lee, Wes Gurnee, and Max Tegmark. The Remarkable Robustness of L.MMs: Stages of Inference?, 2024. URL / 1438.19384
abs
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang. and Eduard Hovy. RACE: Large-scale ReAding Comprehension Dataset From Examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 785-794, 2017.
Qi Le, Enmao Diao, Ziyan Wang, Xinran Wang, Jie Ding, Li Yang, and Ali Anwar. Probe Pruning: Accelerating LLMs through Dynamic Pruning via Model-Probing. In The Thirteenth Interna- fional Conference on Learning Representations, 2025.

---

Published as a conference paper at ICLR 2026
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. Tomr. SNIP: Single-shot Network Pruning based on Connection Sensitivity. In The Seventh International Conference on Learning Repre- sentations, 2019.
Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The Winograd Schema Challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representa- tion and Reasoning, pp. 552-561, 2012.
Xun Liang, Hanyu Wang, Huayi Lai, Simin Niu, and Shichao Song. SEAP: Training-free Sparse Expert Activation Pruning Unlock the Brainpower of Large Language Models, 2025. URL
org/ abs 2503. 07605
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek- V3 Technical Report, 2024. URL
https://arxiv. org/ abs 19437
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time. In International Conference on Machine Learning, pp. 22137-22176, 2023.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through I. O regularization, 2017. URL httpssss::/ //t::///axtte
wttpivxvvvv..//.a// v.. wtt.aawwww....aa.a/ooa/o/// 11t./.11100111
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-Pruner: On the Structural Pruning of Large Language Models. In Advances in Neural Information Processing Systems, 2023.
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. ShorGGPT: Layers in Large Language Models are More Redundant Than You Expect. In Findings of the Association for Computational Linguisties, 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer Sentinel Mixture Models. In The Fifth International Conference on Learning Representations, 2017.
Charles A. Micchelli, Yuesheng Xu, and Haizhang Zhang. Universal Kernels. Journal of Machine Learning Research, 7(95):2651-2677, 2006.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Ma-
chine Learning. 2018 URL
htw.://sss..biss.....ooooooeeeee Eoundations-ofs machine htt.aee--aaannn---pppppeee/////-
J.D. Murray, Mathematical Biology: An Introduction. Interdisciplinnary Applied Mathematics. Springer New York, 2007. ISBN 9780387224374. URL/sttts://books.co google co. jp/
90Gk1YC
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't 't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140): 1-7, 2020.
Ali Rahimi and Benjamin Recht. Random Features for Large-Scale Kernll Machines. In Advances in Neural Information Processing Systems, volume 20, 2007.
Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: A Conversational Question An- swering Challenge. Transactions of the Association for Computational Linguistics, 7:249-266, 2019.
Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness. Natalia Vassilieva, Daria Soboleva, et al. SlimPajama- DDC Understand- ing Data Combinations for LLM Training, 2024. URL 10818
12

---

Published as a conference paper at ICLR 2026
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Proceedings the 2013 Conference Empirical Methods in Natural Language Processing, pp. 1631-1642, 2013.
Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jae-Joon Kim. SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks. In International Conference on Machine Learning, 2024.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribu- tion. Curran Associates Inc., Red Hook, NY, USA, 2019.
Ingo Steinwart. On the Influence of the Kernel the Consistency of Support Vector Machines. 2: 67-93, 2002.
Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension. Transactions of the Association for Compulational Linguistics, 8:141-155, 2020.
Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A Simple and Effective Pruning Approach for Large Language Models. In The Twelfth International Conference on Learning Representa- lions, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykow, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open Founda- tion and Fune-Tuned Chat Models, 2023. BtLLhttss///axxxt org/ abs / /2077 7. 09288
Adina Williams, Nikita Nangia, and Samuel Bowman. A Broud-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, volume 1, pp. 1112-1122 2018
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared LLaMA: Accelerating Lan- guage Model Pre-training via Structured Pruning, 2023. URL httpps://axxi...rrpg/...
06694
An Yang. Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 Technical Report, 2025. URL https://
arxiv. wwwp:/orggabbb////ee 225/555../255.... 2505. 09388
Yifei Yang, Zouying Cao, and Hai Zhao. LaCo: Large Language Model Pruning via Layer Collapse. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 6401-6417, 2024.
Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. Jump to Conclusions: Short- Cutting Transformers with Linear Transformations. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp.
2024.
Arda Yüksel, Abdullatif Köksal, Lütfi Kerem Senel, Anna Korhonen, and Hinrich Schuetze. Turk- ishMMLU: Measuring Massive Multitask Language Understanding in Turkish. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 7035-7055, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can Ma- chine Really Finish Your Sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open Pre-trained Transformer Language Models, 2022. URL https: / arxiv. .rg. abs 22002 01068
Chujie Zheng, Minlie Huang, and Aixin Sun. ChlD: A Large-scale Chinese IDiom Dataset for Cloze Test. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 778-787, 2019.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A PRELIMINARIES
Empirical Risk Minimization. Empirical Risk Minimization (ERM) is fundamental principle in
statistical learning theory. Given a class of functions H and dataset D = {(r;, yi) sampled
i.d. from unknown distribution P, ERM selects the function that minimizes the empirical loss:
R(h) 1
Σ v1).
hERM = arg min R(h). bEH
ERM serves ((x,)>< [((h(z),)).
tractable surrogate for minimizing the population risk R(h)
The Population Approximation Error (PAE) measures the dis- crepancy between empirical performance and the true expected performance:
Population Approximation Error:
apx(h) =
RRh)- R((h) |.
A small approximation error indicates that the empirical distribution a good proxy for the under- lying population distribution.
L' L'-Lipschitz Continuity. A function f: - Rd L' L'ippchtitz continuous if
llf(z) f(z')l L'| r'll.
V Vx, x' X.
Rademacher Complexity. Given class of real-valued functions F and samples {a1,..., En,) the empirical Rademacher complexity of defined as
1 oi (x.;
sup fer i=1 where o; are i.i.d. Rademacher random variables taking values #1 with equal probability. Rademacher complexity measures the expressive capacity of the function class F: larger value indicates richer class that may overfit, while a smaller value suggests better generalization ability. A standard uniform generalization bound is
Rr (F) E
R(f) < R(f) + 22%, +0(
(1))
holding for all f with probability at least 1 6.
B REPRODUCIBILITTY STATEMENT & EXPERIMENTAL DETAILS
Hyperparameters. our main experiments, the maximum sequence length for representation extraction is set to 128. For Step 1, configure the training process as follows. The dimension of the kernel transformation (2m) is set to 1024. We use the Adam optimizer with an initial learning rate of 0.01. A learning rate scheduler is employed, which reduces the learning rate by a factor of 0.5 if the performance does not improve for 3 consecutive epochs (patience = 3). The model trained with batch size of 100 until convergence. The weights, denoted by W, are set as geometric sequence with base of 2 and exponents ranging from 0 to num num_layers 1. For Step 2, the model trained for 100 epochs using the Adam optimizer. The learning rate is set to fixed value of le-3, and the batch size is 128. Pruned layers in the main results are shown in the following table 5 Training was conducted on an 8x NVIDIA A100 server, leveraging pipeline parallelism to distribute different layers of the model across the GPUs.
We fix the two base random matrices and V with their vectorizations Vec(Z) ~ Nax m (0, I) and Vee V) Nrxm (0, I) at initialization. At each forward pass construct frequencies Ω 1 LV, where D and L are learnable. As + oc, the empirical covariance (1/m)Ω2? concentrates around = + LLT We do not resample Z, V during training; learning occurs via D and L.
Post training. To counter the rise perplexity induced by kernelized layer replacement, we apply low-rank adaptation only at the output projection, i.e., the final Milden-to-vocauull mapping.
14

---

Published as a conference paper at ICLR 2026
Table 5: Pruning Results for LLaMA Models.
LLM
Pruning Ratio
Pruned Layers
LLaMA-2-7B
22.8%
[6.7], [10,11], [14,15], [18,19], [22,23]
LLaMA-2-13B
23.4%
[3,4], [7.8], [9,10], [17,19], [27.28)
LLaMA-3-8B
25.8%
[6.7]. [9,10], [12,13], [15,16], [18.19]
When the output head is tied to the input embeddings, we insert lightweight 1 x linear "logit adapter" before the head and place LoRA on this adapter; otherwise, LoRA is attached directly to the head. All other parameters remain frozen (including the kernel map sps, the linear maps {Ai}. and the inverse mapping), so the structural simplification of KDP is preserved while the "representation-llos""
distribution is re-calibrated. Training uses teacher forcing with knowledge distillation from the unpruned teacher: the objective is C A (L Pstude)) + X) CE with 1.0; we find [1. 2] behaves similarly, and report T 1.0. LoRA hyperparameters are rank = 8, 32, and dropout 0.05. Optimization uses Adam (Ir 1 x 10-1, no weight decay), bf16, gradient clip 1.0, warmup 3% with cosine decay, and early stopping on validation PPL. We train for 1-2 epochs on the calibaration dataset, which is reported in Section 3.1 and sequence length of 2048. After training, the LoRA weights are merged into the output head, adding no runtime overhead.
= 0.5 and temperature
C RELATED WORK
Redundency in LLMs. The existence of substantial redundancy within LLMs has been demon- strated through multiple perspectives. Liu et al. (2023) and Yom Din et al. (2024) reveal that suc- cessive transformer blocks in LLMs exhibit remarkable similarity in their representational outputs. Gardinazzi et al. 2025)) hierarchically analyze representations by examining topological features. They observe that the early layers exhibit numerous short-lived topological structures. In contrast, structures in the middle layers become long-lived (or persistent), before short-lived structures reap- pear in the late layers. This indicates that the middle layers possess significant redundancy, [Song al. (224) indicates the residual connection structure inherent in transformer architectures con- strains each block to make only incremental contributions to the overall representation, resulting in high similarity between block representations, MMnn al.) (2024) and Gromov al. (2025) find mul- tiple layers perform similar or overlapping computations to refine predictions. Lad et al. (2024) also shows that LLMs exhibit inherent redundancy, particularly in intermediate layers, where dropping middle layers yields minimal impact final model performance. To reduce model redundancy, pruning has emerged as a mainstream method for removing parameters and improving efficiency.
Weight Pruning. the perspective of the pruning paradigm. Unstructured pruning removes individual weights or neu- rons based on predefined importance scores (Lee et al. 2019 Frantar & Alistarh| [023] Sun al.) 2024]. Although effective in reducing model size, unstructured pruning typically leads to irregular sparsity patterns, which significantly hinder hardware efficiency and deployment flexibility. Struc- tured pruning has emerged as powerful technique for optimizing neural networks by removing entire groups of weights while maintaining computational efficiency.
Weight pruning is divided into structured pruning and unstructured pruning from
Structured pruning eliminates entire columns within weight matrices |An et al. 2024 Ashkboos
ct al. 2024 Le al.
2025 Liang et al., (2025.. attention heads (He al. 2024), or even graph-based
structures (Ma et al. 2023). resulting in more hardware- -eiinarrrffellly sparsity. Despite their structural
advantages, these methods often disrupt the model's original architectural flow and typically require customized software or hardware to realize actual inference acceleration.
Layer Pruning. To preserve architectural integrity, layer-wise structured pruning removes entire layers based on their representational redundancy. LaCo (Yang et al.] 2024) groups consecutive layers and replaces them with layer-wise parameter difference to achieve compression. ShortGPT (Men et al. 2024) computes BI scores-aae akin to cosine similarity-betweer
layers to identify and re- move less important ones. SLEB (Song et al. 2024) estimates layer importance using perplexity and prunes those deemed insignificant. Ding al. (2025) using Centered Kernel Alignment (CKA, Ko- imblith et al. 2019). map representations to a kernel space to measure similarity and remove blocks with high CKA scores. Streamlining (Chen et al. 2025 adopts cosine similarity to identify redun-
15

---

Published as a conference paper at ICLR 2026
dant layers and utilizes knowledge distillation to restore performance post-pruning. However, these studies are typically metric-driven, focusing on identifying layers to prune rather than approach- ing the problem from model dynamies perspective by fundamentally replacing parameter-heavy blocks with simpler modules. As a result, they either directly eliminate redundant layers (Song et al. [0244] Men et al.] 2024) -nevitably incurring performance degradation-
require extensive
data for fine-tuning (Kim et al. [024; Yang et al., [024]] Gromov et al. (025) or distillation (Chen) al. 2025) mitigate the loss. This overlooks the opportunity uncover simplified structural alternatives that enable rapid post-prune pest-pruning recovery with minimal data.
D PROOF
First, we establish high-probability generalization bound on the one-step lin- ear approximation error. For any given step I, we approximate (h(+11) using a linear transformation A₁ on (hh)). The error vector for this one-step prediction is defined as:
Proof of Theorem 7
ej = (h++1) - App (hr).
The corresponding expected loss under the true distribution is given by:
L(, Ar) E [lvg((h++)) - Arpg(hi))
(8)
We invoke the standard uniform convergence bound based on Rademacher complexity (Mohri et al, 2018 Theorem 1). Since ||p[h)]] s R, and ||A,|lop I Ba, the squared loss E(y, y²) = ly y | is L'-Lipschitz on the ball {1lvll Ilv'll S (Ba + +1)Rp) and has range in [0, (BA + 1)2 [2)) Therefore, with probability at least 6, for all (0, {A:]) the hypothesis class H' h Aipo (h) R
| op S BA. E 6, [b)]l S H& J. it holds that
(, (A+)) & TERM + 2L RR, + (BA + 1) log(1/0)
(9)
2n
Next we bound R2 (H'). For each row Ar(i] of A, we have || < BA, and [|<(bb] < Rip. hence the linear class satisfies Rr (?l') S B R. Applying the standard contraction
(shrinkage) lemma for Lipsehitz losses to fo H' yields
2m log(2m/8))
TERM
CBA R
(10)
for universal constant C C>> 0.
From one-step to k-step. Let éj (Pg(h-1) A A be the one-step residual. By a telescop- ing expansion and triangle inequality, have
1+k-1
b+k-2 Σ
II A=!
MAHR-1-..AAJ41
(11)
From the Equation 10 follows that for the learned model parameters, with probability 1 - 5 there a single-step prediction error vector with the number of paradigms bounded in the:
2m log log(2m/5)
| || = |1 - A (p((h))) S VV LERM + C CAR
(12)
Using the Equation 12 for any consecutive step state sequence, the cumulative emror of & is defined as E Pg (h)+)) I-k-1 A, Pa (hr). By submultiplicatiitty, we have:
14k
1+k-1
||& Σ
|eell S
Σ B¹
||edl
(13)
Let j = t- (0,... k - 1}. we have:
& - Σ
B
||448l|
(14)
1=
16

---

Published as a conference paper at ICLR 2026
here, let
b
A
As. a as b, a> b
A.
(15)
I,
Given that the Equation 12 has indicated, with the probability 1 - ô event, each step is | e|
we can adjust å so that the probability applies equally all 1 S j S k steps by means of the union bound. Therefore, under the same high probability event, substitute the above equation and extract the common factor outside the sum equation to obtain the upper bound of the cumulative error norm of the step:
V LERM + CB3 R2
dlog(d/8]
k-1 Σ B
||& =
( RB
/LEENN +
2m (2m(2m/6)
(16)
12
Here, Σ5=0 Bh-1- is the linear mapping error of the k step-by-step accumulation amplified co- efficient via the gradual transfer. When BA < 1, the coefficient converges to (1 -Bx) and BA degenerates into linear growth of k..
Proof of Lemma We prove in two steps. First, in the ideal induced by a universal, translation-invarian kernel (e.g.. Gaussian RBF), any continuous target mapping can be approximated arbitrarily well by a linear operator the canonical feature map. Second, by Random Fourier Features, the infinitedimeesionnll feature map can be approximated by finite feature map & to arbitrary accuracy on the support of D. Combining the two approximations and using the triangle inequality yields the desired bound.
feature space
Universality of the kernel and linear representability in the RKHS feature space. Let k(r, y) be a continuous, positive definite. translation invaarian kernel on Rd whose spectral measure has full support (e.g., the Gaussian RBF kernel). Result from (Micchelli Hrs in the tight set IC on which is consistently dense for continuous functions: for any continuous function f : IC -> TR. and any 17 > O, exist f H4 such that sup,ck If*(()
al.] (006) says that its RKHS
(x) - f(x:)] < n. Denote : R2d - Hd is the feature mapping, (xxx), P(y)) = k(r, y). Let the true mapping between lay- [(z).. For any u E Hr. consider g.. (hr) == (u, @bc4
ers be hr+1
(u, & ((hh)) H,
H4. When ʃ. & is continuous, 9u is continuous in K. By the consistency densities in the previous
paragraph, given arbitrary 5 >> O, exists fü E 7l make
sup 11gu (h,) - fü (h:)ll < <8.
(17)
lueK
The nature of the regenerating nucleus suggests that Sä (hı) = (w., + (hi)) Take a set of {ur}īz C He to fetch d6-) of the former j coordinates, we can get the corresponding wy, and let every upper bound on the agreement error for each coordinate is ōj. Let the linear operator A 1 (hi) = Ej= (w,, o (hi)) n. Uz. So there is consistent error bound
1/2
sup 1 (h4+1) Ax 2 (bi)l <
(18)
h EK
Choose {ō,} so that right term S 1 (for example. 51 = =/2)), we can get
sup
| (h++1) Aoc 1 (h:) e/2.
(19)
h EK
Since the expectation is less than the upper bound, there is
En,~D | (he41) A. + (hr)ll (4/2)²
(20)
Approximating the infinite-dimensiooal map by Random Fourier Features. By Bochner's 's theo-
rem (Günter 1973), for -aniaaiionnnnvaiaan kernels there exist RFF maps & : Rd - 2m such that inner products of şo approximate the kernel. |Rahimi & Rechi (007)) shows that, with high probabil- ity over the draw of the features, one can uniformly approximate popular shift-invariant kernels on compact sets to within any tolerance E > 0 using only D) = 2m = O(E log(1/ (1/s))) dimensions:
sup (())) k(z, v)1 S Ek: z
(21)
17

---

Published as a conference paper at ICLR 2026
for any compact K. C RR containing the support of the data, provided m is sufficiently large (high- probability statement).
Define the (possibly infiite-dimmenionaal feature map & associated with & that k(x, v) (D(x), (yy). The uniform approximation above implies that, on K,
[[(((z), +(v)) - (0(z), (v))lI & for all x, E E.,
(22)
i.e. x ((r) is an -g-ismmetric embedding of + (xx) the level of pairwise inner products.
Let A be the operator satisfying
sup
o (hu+1) Aox t (hr) S el with E1
(23)
Consider the finite-dimensions least-squares problem in RFF space and let A be minimizer of
min B(b 1ls (h++1) Ap (h/)ll². BA
(24)
Because the squared loss expands into inner products and norms, and because RFF inner products uniformly approximate kernel inner products on K., the objective in the RFF space uniformly ap- proximates the kernel-space objective up an additive O (ex) term (with constant depending only on R4 and BA. Consequently, there exists a matrix A with ||||| < BA such that
E || (h++1) App (hi)ll & E 1 (h/+1) Ax & (hr)l² + Cek.
(25)
Choosing m large enough so that En S e2(2) and using the bound in Equation 20 we obtain
E 11 (h1+1) Ap (hi)l <E
(26)
which completes this part.
Proof of Theorem? 2 We start from Lemma [ which guarantees the existence of an RFF map Pa- (with sufficiently large 2mh) and linear operator A such that RRFF A*) S apxp A. The rest is samile comparison: let AnD and (0, ARFF) be the ERM solutions in the two classes; standard uniform-come converrenceeeee bounds control their true risks by the respective optimal risks plus generalization terms FnD [n, 5) and EREr(n, 5). Choosing n so that ERI (n, 6) + (n) S A then yields RRFF( (0, ARFF) < RmD Am), establishing the kernel advantage with probability at least 1 1-6. 5.
Existence of a low-risk RFF predictor. By Lemma 1 pick e = apx A > 0. Then there exist Or and A with ||pp I BA and sufficiently large 2m such that RRFF (0 A+) S E; equivalently, apxarr S apxp-
Uniform convergence and ERM risks. We now turn finite samples. argmin Ae. RR A).
Let AID
arg min ACFi (A) and (0, ARFF
Let AiD
arg minFio (AD(A) and (0", A*) arg argmi RRFF (0, A), which implies that:
apxip == RuD (Aip), apXRFF := RRFF (0*, AA).
(27)
Fix 8 E 10. There exists an event & with probability at least 1 double-sided uniform deviations hold simultaneously for both classes:
oh2 on which the following
5
sup ACPm RID(A) RRD(AA) s EID n,
2
(28)
sup (R.F)
RaFF (0, -RRE()) A)
S ERFF
n, 2
with
dlog (d)
2m log
EnD 8) CBA R2
CRFF(N, CBB R²
(29)
n
n
Define for brevity
(n 8) 2aID (n, }),,
EREP (n, = 2ERFr (n, 2)-
(30)
18

---

Published as a conference paper at ICLR 2026
On 8, by Equation 28 have
RRFF (0, (RFF)) S aPxRRFF + 2FRFF n. 6
(31)
RRFF 0, (RFF S SPxRRF (RFF(R,
On the same high-probability event on which the double-sided uniform deviation bounds hold, we
also have
(AD(AID 2 apııD (n, 5),
EnD(n, 8) = 2 2 (n,
(32)
Proof of Equation 32 By the deviation bound, (AD(A) 2 RID(A - FiD(n,6/22 for all A E FID-
Hence
RID (An() 2 (Am) EID (n. 9) 2 (AD(A - EID (12, 2 2 RiD(AD) - 2 (n, 2
where the middle inequality uses ERM optimality of Å,p and the last one uses the deviation bound again. Since apxiD (A))) Equation 32 follows.
Comparison via the approximation margin. From Lemma I there exists > 0 such that
apXurr S apx -A
(33)
Combining Equation 33 with the already established bound in Equation 35 and the ID lower bound in Equation 31 we obtain on the same event:
RRFF ARFF) I apxiD - + ERFF(N,
RID(AID 2 apxp EnD 8). (34)
Therefore, sufficient condition for
RRFF (0, ARFF) S RId(AA)
FRFF (RFF(n,0) () FID A.
(35)
Choice of Sample Size n. We now choose sufficiently large to ensure that the RFF model's true risk falls below that of the identity model by a margin of at least A.
Here we require such that the sum of the two classes' generalization error bounds is bounded by A. Since EREF typically larger than EnD we can further simplify this to the (conservative) requirement ERFF(n, 8) 2 A/2 2 and En(n, 8) < A/2. Apparently it's similar the other way around. Substituting the definitions of ERFF and EyD we obtain the explicit condition on n.
2m log 4rm
A
d dlg 2nd
A
RB
and CB R2
(36)
72
4
4
This condition can be rewritten more compactly as:
16
CB3
R2g
2m log
2nd 4em Ra ddlog
5
2 N d, 2m,
(37)
2
Here (z) = max xx, 0} denotes the positive part, which appears because if the term in parentheses negative (i.e. if Rt²2 /2m log(4cm/5) < RC / log(2ed/5)) then the RFF class actually has no larger complexity term than the ID class, and the sample requirement can be significantly weak. In the typical case 2m > d, the term parentheses will be positive, and the N given above is the dominant sample complexity threshold.
E EXPERIMENTS ON OPT MODEL
The performance is further evaluated on two OPT models (OPT-2.7B and OPT-6.7B, Zhang al.) 2022) with the results presented in Table and Table 7 The results indicate that our method main- tains robust performance
relatively small models. While experiencing marginal decline on generation tasks, our method's performance on generation tasks stays comparable, showing no signs of collapse.
19

---

Published as a conference paper at ICLR 2026
Table 6: Performance comparison on OPT-2.7B and OPT-6.7B.
LLM
Method
CMNLI
HrSw
PIQA
CHID
WSC
CoQA
BoolQ
MMLU
CMMLU
Race
SST2
Avg.
RI.
Dense
34.2
45.2
75.4
37.9
78.0
51.1
60.6
25.9
26.3
35.6
54.3
36.8
46.78
100.0
SLEB
30.4
40.4
74.2
22.1
72.5
44,8
58.2
25.0
25.3
31.0
55.4
35.6
42.91
91.7
OPT-2.78
SliceGPT
31.0
27.8
50.5
19.9
57.1
37.3
50.7
24.8
23.4
25.8
60.6
25.1
36.17
77.3
SliceGPT
33.6
40.8
63.1
24.1
58.5
37.0
46.5
25.1
24.5
25.0
53.4
26.0
38.12
81.5
w/o Kemee
30.8
34,5
57.1
15.8
69.1
11.2
50.3
25.3
24.8
24.5
51.5
23.5
34.87
74.5
23.5
25.7
35.4
Dense
33.4
60.2
78.6
42.5
81.7
55.0
63.2
25.3
25.55
35.3
76.8
39.6
51.43
100.0
OPT-6.7B
SLEB
36.2
53.2
76.6
23.1
74.7
43.7
46.2
24.1
24.8
31.0
71.6
38.6
45.32
88.1
SliceGPT
21.8
26.5
SliceGPTT
30.0
29.3
61.7
18.8
69.5
27.6
34.7
29.7
25.0
27.1
62.0
37.7
37.76
73.4
w/o Kernel
32.8
63.1
67.5
23.6
72.5
39.3
40.6
25.0
24.2
23.3
61.7
33.5
42.26
82.2
Ours
33.7
53.0
24.5
27.1
76.5
44.5
49.4
29.3
26.1
31.3
69.0
39.5
46.16
89.8
Table 7: Performance of OPT model on generation benchmarks. For Perplexity, lower score means better performance (J), while for ROUGE, a higher score means better performance (t).
LLM
Method
WIKL
C44
Xsumt
RP(PLL)
RP(Rouge)
Dense
15.1
31.3
6.5
100.0
100.0
OPT-2.7B
Ours
28.0
37.2
2.8
69.0
43.1
Dense
10.9
12.3
13.7
100.0
100.0
OPT-6.7B
Ours
13.9
22.1
6.0
67.0
43.8
F MORE ABLATION STUDIES
Different Pruning Ratios(HellaSwag and BoolQ). Although CMMLU (Figure 7 is a compre- hensive benchmark, the dense model's performance on it remains relatively low. To mitigate this potential side effect, we present performance curves on HellaSwag and BoolQ across various prun- ing rates. The results confirm that the overall performance still exhibits consistent linear decline.
Different Kernel Sizes. A kernel function for objective must address fundamental trade- off; must be powerful enough to simplify the system's dynamics, yet sufficiently constrained to retain the core information required by the task. For example, trivial kernel that maps all
representations to a constant would perfectly linearize the dynamics, but at the cost of a complete "information collapse" that invalidates the model. Therefore, our goal is to design a non-trivial kernel that can effectively model the simplified dynamics on the slow manifold. Given such a kernel, can simplify the transformations between layers, thereby enabling layer pruning. To validate this point, beyond the end-to-end downstream experiments in the main text, we further investigate how the method's performance varies with different kernel sizes (i.e., 2m).
Figure 8 illustrates the impact of varying kernel sizes on the performance of LLaMA2-7B across three representative datasets (Hellaswag, CoQA and MMLU). With smaller kernel sizes, such 256
Model Performance on Hellaswag Across Pruning Rates
Model Performance en BoolQ Across Pruning Rates
77.50
85.15
72.56
77.55
67.52
Lartea-1000
Tansttaling
74.25
Accuraty
62.00
72.25
$7.75
63:75
$2.41
67.15
47.68
64.55
42.04
38.00
0%
156
20%
20%
50%
0%
10%
20%
20%
40%
55%
Pruning Rate
Pruning Rate
Figure 7: Performance of HellaSwag and BoolQ when prune with increasing pruning ratio.
20

---

Published as a conference paper at ICLR 2026
and 512, there is significant drop in performance, which suggests that critical information is lost due to excessive compression. Conversely, once the kernel size reaches 1024, the performance gains become marginal. This indicates that 1024-dimensional kernel space, induced by learnable RFF can encode the model's essential information.
Data efficiency. As presented in Table 8 method demonstrates high data efficiency, utilizing only marginally more data than SliceGPT. Critically, our method does not require any post-training restore performance.
Table 8: Data usage statistics for each methods.
Dataset Accuracy Com Across Kerne Sizes
Method
Need Training
Need Fine-tuning
Total Sample
SL.B
X
ShortiP
X
/
LaCo
X
IB
$13
1024
2048
Streamline
30k
Kernel Size
LLMPPuner
SliceGPT
X 1k
Figure 8: Ablation study on kernel size m.
Ours
4k
Sensitivity analysis for each step. To further investigate the importance of each step, conduct an ablation study under extreme conditions on LLaMA-2-7B. We individually train either Step or Step for minimal number of epochs, forcing the subsequent steps to handle all performance modeling. As shown in Table 9 we observe a significant performance collapse in both scenarios. A significant discrepancy in performance becomes evident after short training period. Training Step I for just 5 epochs, despite recovering some performance, still falls considerably short of the original baseline. Conversely, training Step 2 for five epochs achieves performance almost on par with the fully trained method. This provides strong support for our central thesis presented in the main text: that the trained kernel function models performance, while the inverse transformation network's s role is to learn the corresponding inverse transformation.
Table 9: Sensitivity analysis for each step.
Method
HeSW
CoQA
MMLU
Original
65.1
45.4
28.1
Barely Step I (1 Epoch)
43.9
17.0
24.2
Barely Step 1 (5 Epoch)
52.1
27.0
26.7
Barely Step 2 (1 Epoch)
43.7
13.0
24.4
Barely Step 2 (5 Epoch)
60.0
43.7
26.3
Speed up. To further assess the practicality of our method, compare its performance and effi- ciency across several mainstream LLaMA models, with detailed results in Table I0] We mainly fo- focuses on quantifying the impact of reducing inference computation (FI.OPs) the models' average performance (Avg RP). For LLaMA-22 FLOPs were reduced by 23.13%, which brought 1.30x theoretical inference speedup. For LLaMA-2-13B. FLOPs were reduced by 23.64% achiev- ing a 1.30x speedup, and the performance retention rate reached 92.9%; this is the result with the best anrformance-efficiinee trade-off among the three. In summary, the experimental results demon- strate that our method effectively accelerates the inference of LLMs. It offers a tunable mechanism for trading acceptable performance degradation for significant gains in computational efficiency. This feature holds substantial practical value for deploying large models in resource-constrainee environments.
Different training datasets. To evaluate the performance of our method across various datasets, we conduct experiments on the LLaMA2-7b model. The results are presented Table IT Specifically, the "Fewer" setting utilizes a training set comprising 200 samples each from BoolQ, CMMLU, and Slimpajama. In contrast, the "More" setting augmented this base dataset with data from CoQA and HellaSwag, totaling 11.4k samples. Here, "Avg." denotes the average performance across the datasets. "RP. on Dense" and "RP. on std." indicate the Retained Performance relative to the Dense model and the standard dataset (reported in the main text), respectively. The experimental results
21

---

Published as a conference paper at ICLR 2026
Table 10: Performance and efficiency comparison.
Model
Method
FLOPs (G)
Change (%)
SpeedUp
Dense
845.71
SLEB
664.38
27.29
1.27 x
ShonGPT
612.57
38.06
1.38
LLaMA-2-7B
Streamline
586.67
30.63
1.44x
SliceGPT
717.63
15.14
1.17x
Ours
650.06
23.13
1.30x
Dense
1645.01
SLEB
1320.20
19.74
1.24x
ShorGPT 1239.00
24.68
1.33x
LLaMA-2-13B
Streamline 1229.76
25.24
1.34x
SliceGPT
1392.89
15.32
1.18x
Ours
1255.99
23.64
1.30x
demonstrate the remarkable efficiency of our method. Our standard configuration ("Standard") using only 4k data samples, achieved 91.4% of the performance of the Dense model (average score of 58.69). Notably, the "Fewer" configuration, with merely 0.6k samples, attained 97.7% of the performance of the standard configuration, indicating that our method is not sensitive the data volume and possesses strong generalization capabilities. Furthermore, by increasing the dataset size, the "More" configuration surpassed the standard version's performance (103.1%) which validates the scalability of our method. In conclusion, this ablation study highlights the robustness of our method with respect to data quantity.
Table 11: Performance comparison of LLaMA-2-7B under different training data setups.
CMNLI
HeSt
PIQA
CoQA
BoolQ
MMLU
CMMLU
Race
SS12
LLM
Datasets
Avg.
RP. on Dense
RP. on std.
LLaMA 2-71
Dense
34.9
73.3
79.7
72.3
48.9
30.8
42.8
93.2
58.69
100.0
Standard (8d 4k)
33.6
65.1
70.1
45.4
71.6
44.7
28.1
39.9
94.0
43.7
53.62
91.4
100.0
Fewer (3d 0.6k)
33.1
72.7
76.5
46.0
72.7
29.3
26.4
36.3
90.5
40.2
52.37
89.2
97.7
More (10d 11.4k)
34.9
73.0
77.0
47.9
70.7
44.7
30.3
38.0
92.4
44.3
55.32
94,3
103.1
G MORE VISUALIZATION
This section presents additional visualizations of the fitting results. Following the mechanism pre- viously detailed in Figure 4 these figures provide clearer illustration of the fitting quality.
H MORE ANALYSIS OF SLOW MANIFOLD HYPOTHESIS
The manifold hypothesis posits that high- -ishddimensioall data often reside on manifold with a dimen- sionality much lower than that of the ambient space, which is beneficial in various fields (Song & Ermon 2019). This leads us to consider whether this property also holds for the LLM's internal diw-dimensional manifold. The internal repre-
representations, allowing them to be embedded sentations of the model are sequential. As shown in Equation in standard Pre-Norm Transformers, h141 obtained by adding an increment h; therefore define this increment as the velocity of change in the representation.
Although CKA and Cosine similarity are measures of similarity rather than distance, they are closely correlated with distance measures the manifold. This implies that high similarity between two representations corresponds to a small distance between them. This two similarity matrices in Figure I provide quantitative evidence for this. When the representation change step is small, we consider the representation to be stable state on the manifold, which is often feasible model this stable evolution using simple dynamical model.
single propagation
The concept of using linear approximations for nonlinear dynamics finds parallels in many fields. Beyond the fundamental example of caleulus, typical case in partial differential equations (PDEs)
22

---

Published as a conference paper at ICLR 2026
Figure 9: Additional Visualizations Representation Fitting.
23

---

Published as a conference paper at ICLR 2026
3D t-SNE Visualization of Layer Representations
Layer
Layer
Layer
Layer 24
SNE Component
Layer 28
Ino
2
3o 22 S-S Component
45 5n
16 12 Layer Index
Figure 10: The representation propagation trajectories for 100 samples within LLaMA2-7B. The X and Z axes denote the t-SNE coordinates, while the Y-axis represents the layer index. An in- spection along the Y-axis (layer depth) reveals that the trajectories for individual samples remain well-separated and exhibit strong linearity. Furthermore, the transformation between adjacent lay- ers is gradual.
is the Fisher equation. Its full form is: = Duza + ru E =ua However, in the vicinity of K, this equation is often simplified into the following linear PDE: i DUzz rv.
To provide further intuitive support for our hypothesis, we visualized the representational trajectories across different layers for 100 samples from LLaMA2-7B using t-SNE in Figure TO The resulting plot clearly exhibits strong linear trend, which in turn validates the feasibility of our method.
I ADDITIONAL OVERHEAD ANALYSIS
KDP introduces additional parameters during two steps: Kernel Lincarization Joint Training requires training the parameters 0, A, defined in Equation 5 Inverse Transformation Network involves train- ing the parameter from Equation 66) Specifically, when replace block of Knax consecutive layers, the parameter counts for the three components (0, A;, σ) are as follows:
0: The RFF kernel learns a data-driven anisotropic Gaussian RBF kernel. Its learnable parameters are derived from the parameterization of the covariance matrix Σ = D+LL This parameterization parameterized by the learnable vector E Rd, contributing d parameters. Second, L E Raxr is a learnable low- rank factor matrix, which contributes Params(6) = d + dr d(1 r).
consists of two components: First, the diagonal matrix D diag (A)))))
parameters. Therefore, the total parameter count is:
A;: Each operator A; maps the kernel space representation -(b E R2m to p(h)-:) R 2m Consequently, A; is (2m) x (2m) matrix. To ensure stable training, A; is parameterized as A; B,. The learnable parameters for each A; thus consist the scalar 7, and the matrix B, E 22m This results in total parameter count of Params(A.)
Kmax (1 + (2m)²).
Notably, once training is complete, only the product (II A.) needs to be stored. Consequently, during the inference phase, the final parameter count is reduced to only: Params( (aamms(()) = (2m)²
textbfØ: This represents the parameters for the inverse transformation network I(-). The network is parameterized as I(x)) := ML.((z), where a is a learnable scalar and MIP((-) stands for two-layer MIP network. The total parameter count given by: (arams(d)= = 1 + (2m dritten + dhidden) + (dhidden d + d).
Therefore, during the training phase, the total introduced parameter count is
Triam d(1 r) + Kmax (1 (2m)²) )) + (1 + dhiaden (2m d + d),
24

---

Published as a conference paper at ICLR 2026
and during the inference phase, the total introduced parameter count is Inference (KDP) d(1 +) (2m)² (1 chidten (2m + d 1) + d).
Params Params-Inereence
The FLOPS and corresponding parameter counts under different & are in the Table 12 which demon- strate that the additional overhead introduced by KDP is minimal.
Table 12: FLOPs and parameter retention under 2 and k
LLM
FLOPs (G)
Params (M)
h
2
=
k 2
k
Transformer Block
51.80
77.71
404.77
607.15
LLaMA-2-7B
Fold Block
2.89
12.40
Retention Rate
5.59%
3.73%
3.07%
2.05%
Transformer Block
81.20
121.80
634.41
951.61
LLaMA-2-13B
Fold Block
3.4
14.83
Retention Rate
4.19%
2.79%
2.34%
1.56%
J THE USE OF LLMS
We thank the Large Language Model (LLM) for its assistance in proofreading and polishing the manuscript's 's language. The LLM was not involved in the idea, theoretical development, or experi- mental aspects of this research.
25


Reviews:
[{"review_id": "P1tNdYJFLX", "reviewer": "Reviewer_1BU2", "summary": "The paper introduces a novel layer pruning method for LLMs called KDP. The key idea is to view a forward pass as a discrete-time dynamical system, where high similarity between consecutive layers implies entry into a slow manifold, indicating redundant computation. KDP projects intermediate representations into a Hilbert space, where complex nonlinear dynamics between layers become approximately linear. It then learns linear operators to approximate transitions and an inverse mapping network to reconstruct the representations back in the original space, enabling whole-layer pruning without retraining. Experiments across benchmarks indicate that KDP is a performant structured pruning method on LLMs.", "questions": "What is the additional overhead of KDP compared to the LLM? Providing time or FLOPs analysis might be helpful.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The method is novel, significantly different from previous pruning methods.\n\nThe evaluation is conducted on various models of different scales; benchmarks are comprehensive.\n\nThe paper is overall easy to follow.", "weaknesses": "My major concern lies in the additional overhead introduced by KDP. Kernel mapping and linear operators might introduce non-trivial costs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZdGM8RmxXf", "reviewer": "Reviewer_d1gK", "summary": "- The paper introduces Kernelized Dynamics Pruning (KDP), a layer-pruning method that interprets the forward pass of LLMs as a discrete-time dynamical system and leverages representational similarity to simplify internal dynamics.\n- By projecting layer representations into a suitable kernel space, the method simplifies the non-linear transformations between consecutive layers and learns an inverse kernel transformation to reconstruct the pruned representations.\n- The authors provide theoretical guarantees on the linearizability of representations in kernel space and empirical validation across 15 benchmarks, showing strong performance while maintaining efficiency and minimal retraining requirements.", "questions": "- Is there any possible explanation for the significant rise in PPL for the Llama3.1-8B results? (in the case where there is no fine-tuning)\n- Within this framework, is it possible or beneficial to fine-tune the modified model beyond the output projection layer? Specifically, could the pruned model, with its replacement modules, serve as a foundation for further task-specific fine-tuning, or are these modules primarily effective only when substituting already well-learned transformer representations?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Interesting and well-motivated idea of replacing transformer blocks with lightweight modules that approximate their simple dynamics\n- The paper establishes a theoretical error bound for kernel-space linearization, bridging empirical results with formal justification.\n- KDP offers layer-wise compression while preserving performance better than other pruning methods.\n- Extensive empirical validation as the method is evaluated on many benchmarks, suggesting robustness and generality of the proposed method.", "weaknesses": "- The paper primarily compares inference cost and FLOPs only against the original dense model (Table 10) rather than existing pruning baselines, limiting understanding of relative efficiency gains. It would provide a clearer and more fair comparison if the speedup was also compared across all the baselines\n- The parameter count of the replacement modules are not clearly reported; it would be helpful if they were provided for the final pruned models that are evaluated\n- While representation similarity is highlighted, further quantitative analysis of the “slow manifold” hypothesis or ablation on kernel choices would strengthen the conceptual link.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WmAtwF8Z6p", "reviewer": "Reviewer_E6ZX", "summary": "This paper presents a novel layer pruning method called Kernelized Dynamics Pruning (KDP). The method replaces several consecutive layers within a transformer model with a kernel mapping, linear transformation, and inverse mapping. This process is able to prune around 25% of the weights in the network while retaining higher accuracy than existing methods across several tasks. Theoretical results and comprehensive ablations explain the value of all components of the method and relate findings to previous observations in layer representational similarity.", "questions": "What is the cost of identifying and training the transformations? How does this compare to other layer pruning methods?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "There are several strengths that make this an interesting paper. The findings are quite well explained and strong. The findings hold consistently across a wide range of experimental settings. The theory relates findings to and builds upon previous results. Ablations demonstrate the value of each component. The method is explained clearly via figures, an algorithm, and descriptions.", "weaknesses": "N/A. This is a strong and well-organized paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BXQj3D2RkS", "reviewer": "Reviewer_D7Lt", "summary": "The paper introduces Kernelized Dynamics Pruning (KDP), a new method for pruning layers in large language models by viewing their forward computation as a discrete-time dynamical system. It observes that consecutive layers often produce highly similar representations, indicating redundancy. To exploit this, KDP projects representations into a kernel space where nonlinear transformations become approximately linear, allowing simpler modeling. A linear operator and an inverse mapping network then replace entire Transformer blocks. The authors present a theoretical error bound showing that multi-layer dynamics can be linearly approximated in this kernel space and prove that it provides superior fitting capacity compared to the original representation space. Extensive experiments on fifteen benchmarks demonstrate that KDP effectively prunes models while maintaining performance, without requiring fine-tuning. Overall, KDP provides a geometric and theoretically principled framework for simplifying internal model dynamics and reducing redundancy in large language models.", "questions": "Here are a few more detailed questions that I’d like to be addressed\n- From Figure 1, it seems to me that CKA saturates to 1 very frequently (particularly for llama3-8B). It is argued that this is due to similarity in kernel space being more effective. However, if I understood correctly theorem 1 argues that stacking too many layers accumulate errors, even if they are all similar. Indeed, from the results it seems that the algorithm prefers to cut non consecutive layer blocks, than one-two larger consecutive blocks, which would be more similar to previous work. Which part exactly of the algorithm deals with this decision? if it is entirely based on CKA value, and the value is very close to one across the whole depth, is this stable?\n- I haven’t found a definition of ERM, first mentioned in theorem 1.\n- Did the authors verify whether the algorithm would cut early and late layers and verify the drop in performance? they say they exclude them based on previous literature, but would they be excluded by the algorithm anyway or is this to be input as prior knowledge?\n- Can the author explain exactly what “we prune about 25% of the parameters mean in 4.1 setup-models?\n- in the ablation study, the authors use CMMLU, for which performance is low for all models even for the dense case. Wouldn’t it be safer to consider a benchmark for which the dense case performs relatively well? Testing around random performance might have some side effect.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Originality: while the fact that LLMs are characterised by highly redundant layers is well known, with several papers attempting to interpret this fact in different ways, this work’s idea on exploiting kernel space to identify similarity and learn the linear map to prune layers is, to the best of my knowledge, novel and effective.\n- Quality: the authors take a significant effort in stating clearly their theory and in building a solid set of experiments\n- Clarity: The problem is clearly set and the results and experiments are well setup and presented.\n- Significance: the effort of reducing LLM is certainly timely and important. Before this work, papers arguing that layers of pre-trained models were redundant mostly offered solutions which dealt with pruning layers and then fine-tune to cure the cut. The proposal of this paper of learning the map to bypass the pruned layers seems to be a relevant step forward.", "weaknesses": "I have a couple of general objections:\n- About the broad framing of the paper: the authors talk frequently (abstract, introduction) about the fact that they view pre-trained models as “discrete-time dynamical systems” and talk about “slow manifold” and “reduced velocity” during the redundant phase. These concepts are not developed throughout the paper, and it would seem that the paper’s results and claims would be unaffected by removing this interpretation. While I’m totally in favour of trying to give a physical interpretation of LLMs representations, I think the paper would benefit if the authors could expand on this interpretation more (maybe in appendix?), or remove it altogether.\n- I think the paper might benefit from a short interpretability discussion about which layers exactly are being cut and why, also wrt to previous literature. Previous work (ex. Men et al. as referenced in the paper) seem to show that layers after the first half of the model depth are good candidate for pruning, and that cutting earlier, or late layers, causes a drop in performance. A study of this on similar models is found e.g. on Gardinazzi et al. ICML 2025. In other words: there are two things determining viable pruning: size of the cut and position of the cut. The latter is not discussed from what I can see, and this work might have a good angle on this, starting from theorem 1. Moreover, in appendix it seems that several non-consecutive blocks are pruned, which is in contrast with previous work: is this discussed anywhere in the paper? \n- The theoretical part, especially around theorem 1, suffers a bit from lack of background on the techniques used: the authors talk about ERM (empirical risk minimization?), population risk, etc without connecting this to the problem at hand. To someone not familiar with these concepts, this is a bit hard to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)