Equilibrium Language Models
Authors: Yikun Jiang
Huanyu Wang
Tianhong Ding
Wenhu Zhang
Yiming Wu
Hanbin Zhao
John C.S. Lui
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Model Compression
On-Device Inference
Fixed-Point Network
Primary area: foundation or frontier models, including LLMs

Abstract: Large Language Models (LLMs) excel across diverse applications but remain impractical for edge deployment due to severe memory bottlenecks at the edge devices. We propose Equilibrium Language Models (ELMs), a novel compression framework that replaces groups of Transformer layers with a lightweight fixed-point network, reinterpreting deep computation as solving for an equilibrium state. To achieve ELMs, We introduce *Group Pruning Policy Optimization*, which automatically learns optimal pruning intervals. Moreover, we propose *One-Step KV-Cache*, which drastically reduces memory overhead by storing only the final iteration cache without compromising the accuracy, to enable effective deployment at the edge devices. Across different tasks such as common sense reasoning, mathematical problem solving, and code generation, ELMs prune 28\% of parameters while retaining 99\% of the accuracy of dense fine-tuned LLMs, establishing a new direction for memory-efficient edge deployment of large models.

Published as a conference paper at ICLR 2026
EQUILIBRIUM LANGUAGE MODELS
Yikun Jiang', Huanyu Wang', Tianhong Ding',
Wenhu Zhang", Yiming Wu', Hanbin Zhao', Johnn C.S. Lui
Huawei Technologies Co., Ltd., The Hong Kong University of Science and Technology, Thhe University of Hong Kong, *Zhejiang University, "Chinese University of Hong Kong j jiang yikuukp edu.cn cn
ABSTRACT
Large Language Models (LLMs) excel across diverse applications but remain im- practical for edge deployment due to severe memory bottlenecks at the edge de- vices. We propose Equilibrium Language Models (ELMs), a novel compression framework that replaces groups of Transformer layers with a lightweight fixed- point network, reinterpreting deep computation as solving for an equilibrium state. To achieve ELMs, We introduce Group Pruning Policy Optimization, which auto- matically learns optimal pruning intervals. Moreover, we propose One-Step KV- Cache, which drastically reduces memory overhead by storing only the final itera- tion cache without compromising the accuracy, to enable effective deployment at the edge devices. Across different tasks such common ematical problem solving, and code generation, ELMs prune 28% of parameters while retaining 99% of the accuracy of dense fine-tuned LLMs, establishing a new direction for memory-efficiee edge deployment of large models. We provide the implementation of our method in https///ithub.cmm//kk-2222
reasoning, math-
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable success diverse applications, includ- ing dialogue systems, code generation, and mathe- matical reasoning. As the demand for mobile and edge intelligence continues to grow, on-device de- ployment of LLMs has become an inevitable trend. However, such deployment is fundamentally con- strained by the limited capacity of edge devices, where storing billions of parameters in RAM at inference time often exceeds the available mem- ory budget. This resource bottleneck has spurred extensive research into model compression tech- niques, such pruningAsshkboos al. (2024); Xia et al. (2023); Frantar & Alistarh (2023); Sun et al. (2023) Ma al. (2023); Lu al. (2024), quanti-
Dence LLMs
Shertened-LLaMA
LLM-Streamline
ELM
ReploceMe
Sheored-LLoMA
SIQA
OBQA
PIQA
MMLU
MBPP
65M8K
HumonEval
zation (Frantar et al. 2022 Xiao et al.] 2023 Lin al.] 2024), and knowledge distillation (Hinton al. 2015 Xu et al., 2024 Sreenivas et al. 2024). Overall, common compression methods can be cat- egorized into two groups, i.e., (1) simultaneously
MATH
Benchmark evaluation of pruning baselines on (wwen2.5-7B-Instreen
Figure 1:
reducing both computational cost and model size, and (2) reducing computational cost only.
Methods in the first category typically leverage statistical metrics to prune low-contribution parame- ters (Men et al. 2024 Kim et al. 22024 Yang et al. 2024)), or replace consecutive intermediate layers
with lightweight modules (Chen al. 2024 Shopkhoev et al. 2025). While effective in redueing
model size, these approaches often degrade performance on challenging generative tasks, such as
"Corresponding Author.

---

Published as a conference paper at ICLR 2026
code synthesis and mathematical problem solving, due to the loss of model capacity. Methods in the second category adaptively skip layers conditioned on the input (Raposo et al. 2024 Diang et al. 2024
Tan et al., 2024), thereby reducing computation at inference, yet they offer minimal savings in parameter and fail to meet the stringent low-memory requirements of edge devices. Compared with these approaches, our method focuses on alleviating the memory bottleneck of edge deployment, enabling LLMs to run efficiently on constrained devices without compromising accuracy.
To this end, we propose Equilibrium Language Models (ELMs), novel model compression framework for Transformer-based LLMs. Rather than pruning individual network components, which often results significant accuracy loss, ELMs replace groups of intermediate transformer layers with lightweight fixed-point network. In this formulation. the computation of deep layer stack is reinterpreted as solving for an equilibrium state of a fixed-point system conditioned on the input. Formally, the output is defined as the solution =" to non-linear system, that is, =" = f(=",=), where x represents the conditional input. Drawing on theoretical insights from deep equilibrium models (Bai et al. 2019), this implicit formulation achieves a representational capacity comparable to explicit deep stacks, while requiring significantly fewer parameters.
Implementing ELMs requires addressing two critical chal- lenges: determining which layer interval to compress as the fixed-point network and how to efficiently handle the KV cache during inference. Prior methods typically rely on heuristic metrics such cosine similarity perplexity to determine iterations the importance of each layer. However, such metric-based cri- teria cannot be directly applied to ELMs, since they fail to capture the approximation error that arises when substituting multiple layers with a fixed-point formulation. To address this limitation, we introduce Group Pruming Policy Optimization (GPPO), reinforcement learning policy optimization(Sutton et al. 1999) framework that automatically identifies the opti- mal layer indices for equilibrium modeling. Concretely, GPPO samples the index of the first candidate layer from a learnable distribution, and utilizes a supervised fine-tuning loss as the reward signal. In addition, we propose One-Step KV, which retains only the cache from the final iteration during infer- ence. This approach dramatically reduces memory overhead comparing with storing caches for all intermediate iterations, without compromising accuracy. Moreover, it enables token- adaptive fxxed-point iterations to proceed without recomputing missing intermediate KV caches.
Transformer Layers
Fixed-point Layer 0(2", h)
Transformer Layers
Illustration of ELMs. The group of intermediate trans- former layers is replaced with lightweight fixed-point layer.
Figure 2:
We evaluate ELMs on three challenging downstream tasks: common sense reasoning, mathematical problem solving, and code generation. Our method consistently outperforms the state-of-the-an methods. Even with 28% reduction in parameters, ELMs retain on average 99% of the accuracy of fully fine-tuned dense LLMs, demonstrating the effectiveness of deploying on edge devices.
PRELIMINARY
We next review autoregressive language models with KV caching and fixed-point networks with cquilibrium-bar based training, which form the foundation of our method.
2.1 AUTOREGRESSIVE LANGUAGE MODELS
Large language models are typically built upon decoder-only Transformer architectures (Vaswani et al.] 2017). where tokens are generated in an autoregressive manner. Each Transformer layer employs a residual structure that integrates multi-head self-attention with feed-forward network. Formally, the operation of the 1-th layer can be expressed as
h 1+1 f (hh) FFN(MHA(()))
(1)
where h' denotes the hidden state input to the I-th layer, MHA represents the multi-head attention module and FFN is the forward network. In decoder-only Transformers, causal mask is
2

---

Published as a conference paper at ICLR 2026
applied to prevent information from subsequent tokens from leaking into preceding ones. This prop- erty enables caching of key and value tensors in self-attention (Pope et al. 2023). thereby avoiding redundant computations. For instance, when generating the (T 1) token, the output of the I-th layer can be written as
h h = f( Vi...r h h1+1),
where Kj.... and Vi... represent the cached key and value tensors of preceding tokens at the I-th layer. This KV cache mechanism has become standard acceleration technique for LLM inference, reducing the attention complexity from ((T2) ((T).
2.2 FIXED-POINT NETWORKS
Conventional neural networks transform features by explicitly computing forward through a stack of layers. In contrast, fixed-point networks, a class of implicit neural networks, define the model output as the equilibrium solution of the fixed-point equation 2t f(2", The equilibrium state obtained through an iterative optimization process,
lim 2k+1) lim f(2 x)) E-+00
(3)
The superscript & denotes the iteration step. Formally, fixed- fixed-point network is equivalent weight- tied network of infinite depth, thereby achieving stronger expressivity with fewer parameters. How- solving for the fixed-pole point generally requires variable number of iterations and incurs substan- tial memory overhead during training. Deep equilibrium models (DEQs) address this challenge by analytically backpropagating through the equilibrium point 2" using implicit differentiation, which eliminates the need to store intermediate activations and results in constant memory usage:
O2" a(-)
0f(=" x1)=,,)))))))"
0f(=",a)
(4)
0z
a(.)
To accelerate training, Jacobian-Free Backpropagation (JFB) (Fung al. 2022 drops the inverse Jacobian term and instead computes the gradient only at the final iteration, thereby circumventing the cost of matrix inversion. Stochastic JFB (SJB)(Bar & Melas -yriazi 2024) introduces randomized iteration steps during training, achieving superior results in large-scale experiments.
3
EQUILIBRIUM LANGUAGE MODELS
3.1 WORKFLOW OVERVIEW
From holistic perspective, an Equilibrium Language Model (ELM) replaces consecutive redundant layers fixed-point module in the transformer architecture, Given language model F, which contains N transformer layers, we prune consecutive M (M < <N) layers
F(x) P
MoM-loorioo ....o (x).
(5)
Pruned Layers
Following this pruning procedure, we embed a fixed-point module f at the position of the I-th transformer layer and the model becomes
F(x) f
(z)
(6)
Fixed-Point Module
Let h E Rd denote the conditional input of the fixed point module and d represents the dimension of hidden feature. We denote the iterative value as z (kk) E [ddxd, where A represents the iteration step. The module ip produces the equilibrium state x" = =p<=","') .t') through iterative rule
lim z (x++1) = lim (²z (k) n²). k00
k0000
The implicit fixed-point layer to be trained is composed of two fully connected layers W WA Rd and transformer layer ft. The network of the fixed- -oint module is defined
E
(z) (k) h²) =f"(w= z Wh. h²)
(8)
3

---

Published as a conference paper at ICLR 2026
Update with
CUP
Backward C
Policy Model Wa
Layers
Layers
Ri
A
Layer
Candidate Layer index
Fa
Fixed Fixed-point Layer
LoRA
Layer
Loyers
My-M
Layers
Layers
Sampling
Dense Language Model
Reword Model
Rewards
Advantages
Figure 3: Workflow of GPPO. The policy model samples groups of pruned layers, which are used to construct ELM as reward models. Each fixed-point layer is trained with LoRA, updated by super- vised loss, while group- nommlloore rewards guide the policy update.
The transformer layer f* inherits parameters from the first pruned layer f', and the initial value of (0) is zero. In addition, the learnable matrix Wa is random initialized and Wn is set as identity matrix. Consequently, the result of fixed-point layer at the first iteration is the same as the output of layer f', enabling good initial starting point.
Although the upper bound of iteration numbers is +oc in theory, the iteration converges within M steps in empirical practice. In this way, the proposed ELM achieves parameter compression over the M transformer layers, while incurring no additional computational complexity.
The optimization objective of the proposed fixed-point module is composed of two components: cross-entropy loss and a distillation loss, as
Cast = AeeCoe
+ AdisunCissut
-DEEDDC Ep .2")
where 'D denotes data samples from the training dataset. Aces Adisill are hyper-parameter of the cross entropy and distillation respectively. We utilize a distillation loss (MSB) align the output of with the output hidden states h + from the (L+M 1)-th layer fi+MM-1 of baseline transformer F. Additionally, cross-entropy loss is employed optimize the next-token prediction results.
Training fixed-point networks with implicit gradient in Eq. 4 is computationally expensive due to matrix inversion. To mitigate this, we adopt the Stochastic Jacobian-Free Backpropagation method for training the fixed-point layer. Specifically, we uniformly sample m U[0. M/2] steps of fixed- point iterations without gradients and (11,, M/2] steps with gradients, as illustrated in A.I
3.2 GROUP PRUNING POLICY OPTIMIZATION
Inspired by the success of policy optimization methods such PPO (Schulman et al. 2017) and GRPO (Shao et al. 2024), we develop Group Pruning Policy Optimization (GPPO), a lighteeight framework for automatically search for pruning intervals under the ELM formulation. GPPO follows the policy gradient paradigm. where learnable parameters E N-M define a policy model that represents the probability of selecting each candidate layer the first pruned layer. The resulting compressed equilibrium model serves as the reward model, with the policy distribution adapting maximize downstream performance as shown Fig.
f....,f -NNM M is treated as a candidate, and lightweight LoRA module is attached to reduce training overhead. layer index is sampled from wo, and the supervised fine-tuning loss Can (Eq. 9 is computed to update the reward model. At the same time, negative cross-entropy losses are accumulated as rewards.
Concretely, each transformer layer f
R (-c) - L. -
(10)
4

---

Published as a conference paper at ICLR 2026
Algorithm Group Pruning Policy Optimization (GPPO)
Require: Policy parameters Original language model F. Hyperparameters: Clipping threshold c, Policy update frequency p. Policy update iterations g
1: Rewards group R {}
2: for = 0, 1........ do
3: Sample first pruned layer ID Re, and construct ELM Fi
4: Compute loss Lin and rewards Lce (see Eq. 9
5: R.append(- -ove)
> Store negative cross-entropy as rewards
6:
Backpropagation((Cat
7:
Optimize(FFi
8:
if' i mod = 0 then
> Policy update phase
9:
for j = 0, 1, .......---........... do
10:
Compute policy loss:
Cpolicy f- Tou (TTE)
(see Eq. 11
(nckpropagatioon(Coltt)) 12: Optimize(0)
13:
end for
14: R.clear()
Reset rewards list for next cycle
15:
end
end for
17:
return
where p denotes the size of the reward set. The policy update follows To
CLIP (TT)
imin( Te
Å, clip(
c, 1 c)
(11)
Te.. with the advantage function defined as the normalized reward,
Tea
À R mean R)
(12)
std(R)
Training proceeds in cycles: rewards are collected over p steps, and then the policy parameters are up- dated for q optimization steps using the aggregated rewards. After convergence, the first pruned layer in- dex is arg max f. Finally, pruning layers start from arg max 0, and the equilibrium model is constructed according to the workflow described in Sec. 3.1 The paradigm of GPPO is illustrated in Fig. and the pseudo-implementane is shown in Algo.
Iter 3
Iter 2
Iter
of aentioon
Multi-Step cache
attention
3.3 ONE-STEP KV CACHE
With respect to the deployment of large language models at inference time, the memory overhead in- curred by the KV cache is a significant challenge. With the conventional LLM inference pattern, the fixed-point module is executed many times itera- tively to achieve convergence. In the k-th step, all previous KV caches are included in the calculation
One-Step KV cache
Fixed-pointt Layer
Seved oachs
Unsoved KV oache Skipped KV calcul
(k+1)
p
K 1...T T...T 2T+1 hTT1).
(13)
Figure 4: Illustration of cache strategies in
the fixed-point layer. One-Step KV cache re- quires only the cache from the last iteration, while Multi-Step requires all caches.
Consequently, the storage complexity of the Multi-
Step KV cache in the fixed-point module is O(T
5

---

Published as a conference paper at ICLR 2026
M). where T denotes the length of previous tokens and M is the maximum number of fxee-point iterations. The One-Step KV cache stores and reuses only the cache from the last round in fixed- point iterations of all preceding tokens, which is then applied during the fixed-point iterations of the current token. Since intermediate features 2(6) converge to the equilibrium state z", One-Step KV does not compromise the accuracy of the equilibrium language model and consistently converges to the approximate results the Multi-Step KV cache under the appropriate conditions.
In the k-th fixed point step of the T + token, the One-Step KV cache iteration is formulated as
(R) (k+1) :p (Ki.... V....T 2T+1 hT+1). T+1
(14)
Under this framework, the storage complexity of the KV cache becomes O(T). Moreover, it elim- inates issues related to KV cache missing when allocating different iteration steps among tokens, enabling the efficient deployment of fixed-point solvers to accelerate our proposed ELMs.
3.4 FIXED-POINT SOLVING ACCELERATION
Benefiting from the One-Step KV cache mechanism, our method can efficiently adopt an adaptive termination criterion for the fixed-point point iterations: the process is halted once the & norm of the residual between successive states, [1)p(z(()) h) (x))|1, ||2. falls below predefined threshold 8. The final output is then taken as the equilibrium state z". This approach allows variable computational budgets across tokens, depending their individual convergence speed.
In addition to the standard fixed-point (simple) iteration, we consider two classical acceleration tech- niques to further reduce the required iterations. Broyden's 's method (Broyden 1965) employs a quasi- Newton update that iteratively approximates the inverse Jacobian. Anderson acceleration (Walker & Ni 2011 uses linear combination of previous iterations extrapolate.
EXPERIMENTS
4.1
EXPERIMENTAL SETTINGS
Source Model. We evaluate method on popular open-source LLMs:
Swen2.5-7B-Instrun (Qwen et al. 2025) and Llama3.2-3B-Instuus (Dubey et al., 2024). All mod- els consist of 28 transformer layers. We use their finetuned counterparts baselines and compare against them to measure the performance drop caused by parameter pruning. Additional implemen- tation details are provided in A.7
We compare against representative pruning baselines. Sheared-LLLMM (Xia al. 2023) applies structured pruning over heads, FFN channels, hidden dimensions, and layers with to regu- larization and LoRA finetuning. Shortened-LLaMe
Baselines.
(Kim et al., 2024) prunes layers by perplexity. ReplaceMe (Shopkhoev al. 2025) and LLM-Streamline (Chen al. 2024) replace consecutive layers with lightweight modules based on cosine similarity. ReplaceMe utilizes linear transforma- tion for feature aligmment and ALM--treamling utilizes a transformer layer or an FFN module.
Benchmarks. We conduct on three downstream tasks, including: commonsense reasoning, mathe- matical problem solving, and code generation. We fine-tune the model on PIQA (Bisk et al. 2020). SIQA (Sap et al., 2019), OBQA (Mihaylov et al. 2018) and MMLU (Hendrycks et al.) 2020) to ver- ify commonsense reasoning; MetaMathQA (Yu et al., (202)) for mathematical problem solving; and OpenCoder (Huang et al. (2224) for code generation. Evaluation results are performed in zero-shot
settings among PIQA, SIQA, OBQA, MMLU, GSM8K (Cobbe et al. 2021). MATH (Hendrycks)
et al. 2021), HumanEval (Chen et al. 2021), and MBPP (Austin et al. 2021)
4.2 QUANTITATIVE ANALYSIS
Comparisons with State-of-the-Arts We present the performance of all benchmarks compared with sota method under the same settings. With pruning eight layers, i.e., M-8 (about 28.6% of the non-embedding parameters compressed), ELMs consistently outperform baseline methods in both compression efficiency and stability as shown in Table [ On average, they preserve 99% the precision of fine-tuned dense L.MSs across benchmarks and base models.

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison of pruning baseline methods on commonsense, math, and code benchmarks. The last columns reports the average accuracy across all benchmarks and the relative performance (RP) compared with dense models.
Code
Average
PIQA
SIQA
(BQQ
MMLU
GSMSK
MATH
Hanannvvil
MBPF
Dense
0.00%
54,8
72.4
68.9
Lhaaned-LLMM
37.6
02.6
61.55
Qwen2.5-1.8 5--
Instruct
Shortened-I L.MA
Cosinc)MeeCosne)
28.69 6%
28, 6%
75.6
71.8
34.4
74.8
28.8
41.5
0.00
63.0
0.00
24.5
64.0
1.83
2,00
65.6
60.1
20.7
87.3%
30.0%
ReplaceMec.5)
38.6%%
33.8
29.1
0.00
1.22
1.20
20.5
LLM Layer)
23.6%%
75.6
70.7
71.6
45.2
44.3
16.1
38.4
53.4
51.9
75.4%
58.0
24.4
44.6
41.6
71.8
74.4
70.6
68.1
LM
28, 3%
77.1
73.6
68.2
39.1
4
22.2
70.7
69.8
61.8
90.0%
Shurtened LLaMA
6%
K2.6
77.1
79.2
49.7
67.6
25.4
73.2
68.6
65.4
95.4%
Llana3
ReplaceMe
73.0
34.9
ReploceMe(LLS)
28.6%
K2.7
77.8
72.6
45.0
0.30
0.36
0.00
11.00
34.5
50.8%
LLM SteaSaiinnllleayyee
70044
J.M-SSccanlio(FFN
79,8
50.6
66.5
63.6
92.7%
ELM)Ouna
28.6%
84.
78.4
79.6
30.5
74.5
29.7
76.8
69.8
67.9
99.0%
Dense
0.00%
87.4
79.1
57.0
00.2
82.6
42.4
53.4
78.4
75.7
100,0%
Sheamd-H
28. 5%
81.8
77.0
78.4
47.3
68.0
25.4
61.7
72.6
66.5
87.95
LLMMA
Qwen2.5
Instruct
ReploceMetUS)
ReglaceMe(Cosingg
28.0%
28.65
37.4
57.4
49.9
49.9
41.6
41.6
32.1
32.1
0.00
0.00
0.00
0.00
11.0
11.0
11.8
11.8
25.5
25.5
33.61%
33.69
28.6%
72.7
68.5
72.0
47.4
35.4
15.8
40.2
53.4
50.1
66. 1
CLM(Ours)
28.6%
86.3
79.3
$7.0
59.2
82.2
$8.4
76.2
75.0
99.0%
Unlike baselines that collapse on mathematical reasoning and code generation tasks, ELMs retain at least 90% accuracy even on MATH. For example, on GSM8K, the ELM on Qwen2.5-7B-Insst achieves 99.5% accuracy of dense, outperforms Sheared-LLLaMA by 17.2%, Shortened-LLLaMA by 20.4% and L.MM-Stremmline by 34.9%, while ReplaceMe fails completely. Performance gains also hold across models. Whereas methods like L.r--Stremlline maintain high accuracy only on -lama3.2-3B-Instuu their performance drops sharply on Qwen2.5, In contrast, ELMs deliver consistent improvements, e.g., surpassing LLM-Streamline by 23.5% on Qwen2.5-1.5B-Img and 22.5% Qwen2.5-7B-Instte In-sructt in average. These results highlight the robustness and general applicability of our method.
Performance under Different Compression Ratios. We evaluate ELMs with varying numbers of pruned layers M=8, 10, 12, 14 on 1.5e2..5-1.5B-Instrree and J-3ma3.2-3B-Instuee for code synthesis and commonsense reasoning. As shown in Table ELM consistently outperforms pruning baselines across models and benchmarks under the same pruning ratio. With increasing pruning ratio, baselines suffer severe degradation, while ELM exhibits much milder decline, highlighting its robustness. On
ELM still retains 97.7% of the dense performance on MBPP with 50% layers pruned, outperforming Shortened-LLaMe by 8.6% and LLM-Streamline by 43.4%. For MMLU, ELM retains 98.65 6.6% accuracy, outperforming Mheared-LLaMi by 35.7%.
Table 2: Comparison of different numbers of pruned layers for code and commonsense tasks.
HumanEval
MBPP
MMLU
Model
Method
M
8
10
12
14
8
10
12
14
12
14
Sheared-LLaMA
75.0
68.3
62.2
50.0
69.0
67.6
65.2
60.4
37.6
40.5
38.2
32.1
Qwen2.5 1.5B
Shorteaed-LLaMA
64.0
59.1
53.0
46.3
65.6
61.8
60.2
58.2
41.5
39.4
36.9
34.6
Instruct
LLM- ELM(Ours)
S.M-Sincamlin((ayeer
38.4
32.3
22.0
18.9
53.4
47.6
41.2
37.2
45.2
39.0
36.1
34.1
74.4
71.3
66.5
58.5
70.6
69.4
67.4
65.4
54.0
53.0
50.2
46.2
Sheared-LLaMA.
69.0
67.5
66.9
65.5
69.8
67.3
66.8
67.1
39.1
40.3
39.3
38.1
Llama3 22-3B Instruct
73.2
67.7
64.6
68.6
69.6
49.7
50.1
49.5
L.M. Streamline(Layer
76.2
75.0
23.8
22.6
67.0
64.0
42.0
38.0
50.8
50.8
50.1
32.1
FL. (Qurs)
76.8
77.4
73.2
68.9
69.8
71.0
70.4
6R.4
50.5
50.8
50.7
50.3

---

Published as a conference paper at ICLR 2026
4.3 STUDIES ON GROUP PRUNING POLICY OPTIMIZATION
In this section, compare proposed Group Pruning Policy Optimization (GPPO) in Sec. 3.2 with statistical metrics of cosine similarity and perplexity. We randomly sample 36 instances from the training set to calibrate the cosine similarity and perplexity metrics. GPPO is trained for two epochs on each task. To illustrate the impact of pruning strategies, we report the results of a commonsense task (MMLU) and code generation task (MBPP) under different choices of l" for ELM as shown in Fig. S The results show that pruning strategies are critical and the optimal /* varies among downstream tasks. GPPO consistently identifies the best choice of (", matching the outcomes of enumeration experiments. Moreover, the learned values exhibit a stronger positive correlation with the performance of ELMs, confirming the effectiveness of our approach.
Commonsense
Code
MMLU
2.5
MBPP
2.5
Normalized Metric Value
52
GPPO
GPPO
Evaluation Score
50
PPL
PPL
15
688
COS
1.0
COS
10
0.5
66
0.5
46
0.0
42
-10
-1.0
-15
-1.5
16
20
12
30
Layer Index
Layer Index
Figure 5: Layer-wise experimental values on ELMs and normalized pruning measures for com- mon and code tasks. colored rectangle stands for the maximal value of each metrie, i.e., cosine similarity, perplexity and GPPO. The best pruning performance of ELM is signed red star.
4.4 STUDIES ON ONE-STEP KV CACHE
As demonstrated in Sec. 3.3 show the effectiveness of the One-Step KV in fixed-poine solving converge to the same results as those obtained with the Multi-Step KV cache. We conduct abla- tion studies ELMs with and without the One-Step KV cache under different pruning settings (M=8, 14). We employ simple iteration as the solver and set the number of iterations equal to M. reportsssprts both the accuracy of ELMs Qwen2.5-1.5B-IIsiiee ing benchmarks and KV cache consumption at 4K context length in 16-bit. The results show that the One-Step KV cache achieves nearly identical performance the Multi-Step variant, confirming the theoretical equivalence. Moreover, it reduces memory overhead by 25% and 46% for --8 and =114, respectively, thereby substantially lowering the memory footprint during inference.
across mathematical reason-
Table 3: Memory efficiency ELM under different M.
M
Method
Iters
Model(GB)
KV(MB)
GSM8K (Acc.)
MATH (Acc.)
Dense
3.09
114.8
72.4
32.2
ELM (w/o One-Step KV)
8
2.44
114.8
71.4
30.8
8
ELM (w/ One-Step KV)
8
2.44
86.0
71.8
30.5
ELM (w/o One-Step KV)
14
1.88
114.8
63.3
24.1
ELM (w/ One-Step KV)
14
1.88
61.6
63.6
24.0
4.5 STUDIES ON ACCELERATION SOLVERS
Leveraging the fixed-point property of ELMs with the One-Step KV cache, we apply existing numer- ical solvers to accelerate convergence in fixed Exeedpooint iterations and reduce computational cost. We evaluate this approach GSM8K with 1..e2.5--1.5B-Inaee under pruning settings (M=8, 14),
8

---

Published as a conference paper at ICLR 2026
considering simple iteration, Broyden's method, and Anderson acceleration. Table shows the per- formance and iteration counts under different values.
Overall, the accuracy exhibits minimal variation across different 8 values. Among different solvers, Broyden's 's method shows weaker performance when = due to larger errors in approximating the inverse Jacobian. Anderson acceleration achieves the most stable speedup across pruning depths, e.g., reducing computation cost by 38.6% over simple iteration when M is set to 14, and is 0.5 with negligible accuracy loss. Even with loose å (8-4.0), ELMs retains over 98% and 86% accuracy of the dense LLMs with 8 and 14 pruned layers, respectively. At the same time, ELMs reduce computation cost by 17% and 35%. These results demonstrate that ELMs not only shrink parameter size but also substantially reduce computational overhead, enabling faster inference.
Table 4: Performance comparison of solvers under different thresholds d GSM8K benchmark with 1...e...--.5B-InsS Rows indicate methods (simple, broyden, anderson), while columns correspond thresholds.
M
M 14
Method
Threshold 6
Threshold
0.00
0.50
2.00
4.00
10.0
100.0
0.00
0.50
2.00
4.00
10.0
100.0
simple
71.8
71.3
72.0
71.6
70.1
68.8
63.3
63.8
63.2
63.8
62.5
35.8
Acc.
broyden
69.8
70.5
69.7
70.1
70.1
68.9
63.2
62.9
63.5
61.0
59.0
36.2
anderson
71.6
71.0
71.0
71.1
70.4
68.8
63.4
64.3
62.7
62.6
60.3
35.5
simple
8.00
7.16
4.60
3.60
2.43
1.06
14.00
12.25
7.03
5.52
3.63
1.10
Iters
broyden
8.00
7.22
5.10
4.16
2.43
1.06
14.00
8.05
5.47
4.55
3.49
1.10
anderson
8.00
5.56
4.08
3.37
2.43
1.06
14.00
7.52
5.18
4.27
3.16
1.10
4.6 INFERENCE EFFICIENCY
Memory Consumption. We further evaluate the inference-time memory efficiency of ELM un- der realistic deployment settings. The primary memory cost of LLM inference stems from model parameters and the KV cache. Table.3]reports the memory consumption of ELM based on Qwen2.5- I.St-Instruct. Compared with Dense baseline, ELMs with M-8 and M 14 pruned layers reduce the parameter memory from 3.09 GB to 2.44 GB and 1.88 GB, achieving 21 % and 39% savings, respectively. When combined with our One- Step KV cache, ELM further decreases KV cache over- heads at a 4K context length from 114.7 MB to 86.0 MB (25% reduction) for =/=8 and to 61.6 MB (46% reduction) for 1414. These results confirm that ELM substantially lowers the total memory footprint and can benefit deployment on memory-com cennory-oonstrained platforms.
Speedup Threshold
(b) Accuracy Threshold
(c) Accuracy Speedup
Simple
Simple
Simple
Broyden
0%
Broydon
Broyden
1.294
Andierson
Anderson
Anderson
Speedup
1.20
Accuracy
98 0%
Accuracy
$8.0%
on 0%
37.0%
1.10x
66.0%
55.0%
10
1.00
1.10x
1.20x
1.30x
Threshold
Threshold
Speedup
Figure 6: Performance of solvers on ELMs based on Qwen2.5-1.. 5-.en255-1.BB-Instmmc across various thresh- olds. The decode speed and accuracy of GSM8K normalized with Dense.
Decoding Speedup. We also evaluate the computational efficiency ELM during decoding. Al- though ELM relies on fixed-point iterations, it achieves substantial decoding speedup due to two

---

Published as a conference paper at ICLR 2026
key reasons: (1) appropriate choices of the convergence threshold d and acceleration methods ef- fectively reduce iteration counts, and (2) the computational cost the solvers and convergence checks is negligible. As shown in Fig. 6 ELM with = accelerates compared with the dense model across different solvers with thresholds 820.5. With 8=4, the Anderson solver converges in only 2.4 iterations and decodes 87.8 token/s, while the dense model only reaches 76.4 token/s. Fig. (c) illustrates the accuracy-speeduy Pareto frontier. where 6=10 of Anderson solver provides favorable balance, achieving 28.3% speedup while maintaining 97% accuracy. Even under an ag- gressive setting of 8=100, ELM continues to deliver usable performance. The effect is attributed to SJFB training, which enables stable generation even with few iteration steps. These results demon- strate that fixed-point iteration is not decoding bottleneck: iteration counts remain small, solver overhead is minimal, and the accuracy-speedup trade-off can be smoothly tuned via the threshold ö.
RELATED WORK
Layer-Level Pruning in LLMs Research. Previous pruning approaches for LLMs have mainly focused on weight matrices (Frantar & Alistarh 2023; Sun et al. (023). attention heads, and hidden dimensions (Ma et al.. 2023] Xia et al.] 2023). but these often introduce structural ireeularitiis. More recently, layer-level pruning has emerged as a promising alternative due to its deployment efficiency across platforms. For example, Shortened-LLaMA (Kim al.) (024) removes layers based on metrics such as perplexity or Taylor expansion and employs LoRA fine fine-tuning for recovery. LLM--ttramliie (Chen al. 2024) and ReplaceMe (Shopkhoev et al., (025) prune consecutive intermediate layers and insert lightweight alignment modules: the former reintroduces a transformer or feed-forward block, whereas the latter applies a simple linear mapping. Our method follows this paradigm but employs novel alignment network, achieving substantially higher performance.
Looped Mechanisms in LLMs Research. Standard LLMs consist of stacked residual trans- former layers, while recent studies (Zhu et al., 2025 Chen et al. 2025 Bae et al. [025 Li et al.,
2025 Geiping et al. 2025) investigate looped architectures. For instance, Inner Thinking Trans-
routes every ween-numbeeed layer through learned loop, and Mixture- of-Recursion (Bae et al. 2025) generalizes this with diverse recursive designs. ELMs advance this line of work with two key innovations: (1) grounding looped computation in fixed-point theory, which enables train-free acceleration solvers and adaptive per-token iteration; and (2) pioneering the use of looped mechanisms for post-training L.M compression.
former (Chen et al. 2025
6 CONCLUSION
We introduced Equilibrium Language Models (ELMs), a novel LLM compression framework that replaces consecutive transformer layers with a fixed-point network and leverages a policy optimiza- tion framework GPPO to learn optimal pruning strategies. By exploiting One-Step KV cache and acceleration solvers, ELMs effectively reduce computational overheads while maintaining model fi- delity. Experiments demonstrate that ELMs retain over 99% of the performance of dense models average with 28% fewer parameters, and outperform state-of-the, the-art art pruning methods on complex generation tasks such as mathematical reasoning and code synthesis.
In addition, we discuss several directions for future work. Our experiments primarily focus on post- training on downstream tasks, leaving the generalization of ELMs in pre-training scenarios open for investigation. Furthermore, our current design only prunes the single consecutive layer interval. Extending ELMs more aggressive architectures such transformers composed entirely of fixed- point layers offers a promising avenue for future research.
REPRODUCIBILITY STATEMENT
We have made extensive efforts to ensure the reproducibility of our results. Details of the proposed model and algorithms are provided in Sec. 3 with additional related algorithms, derivations and proofs given in A All base LLMs and datasets used in our experiments are publicly available. The experimental setup, including hyperparameter choices, data processing, evaluation protocols, and baseline implementation details are described in Sec. and further elaborated in Appendix(A.) A.77 We provide the core implementation for reproducing our method in https://gihub..om/yy-
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Salch Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.1502 2024.
Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski. David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint SKKv::2088.07322 2021.
Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Learning dynamic recur-
Harutyunyan, Ziwei Ji, Aaron Courville, et al. Mixture-of- sive depths for adaptive token-level computation. arXiv preprint arXiv:2507.1052. 2025.
Shaojie Bai, Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in neural information processing systems, 32, 2019.
Xingjian Bai and Luke Melas-Kyriazi. Fixed point diffusion models. In Proceedings of the 2024.
IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
Loubna Ben Allal, Niklas Muennighoff, Logesh Kumar Umapathi, Ben Lipkin, and Leandro von
Werra. A framework for the evaluation of code generation models. httpp:: / com/ 2022.
bigcode-projecce /bi geode wwweevalaaionnnnneeeeeeeae
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Charles G Broyden. A class methods for solving nonlinear simultancous equations. Mathematics of computation, 19(9)):5775593, 1965.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph. Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint Si..::2100.033744, 2021.
Xiaodong Chen, Yuxuan Hu, Jing Zhang, Yanling Wang, Cuiping Li, and Hong Chen. Streamlining redundant layers to compress large language models. arXiv preprint arXiv:2403. 19135, 2024.
Yilong Chen, Junyuan Shang, Zhenyu Zhang, Yanxi Xie, Jiawei Sheng, Tingwen Liu, Shuohuan Wang, Yu Sun, Hua Wu. and Haifeng Wang. Inner thinking transformer: Leveraging dynamic depth scaling to foster adaptive internal thinking. arXiv preprint arXiv:2502.1388 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2/ 10. 1400..11100 14168, 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. Xiv--2207, 2024.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International conference on machine learning, pp. 10323-10337. PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh, Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.1733 2022.
Samy Wu Fung, Howard Heaton, Qiuwei Li, Daniel McKenzie, Stanley Osher, and Wotao Yin. Jfb: Jacobian-free backpropagation for implicit networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 6648-6656, 2022.
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv=2502.05/7/ 202..
11

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint 00t1::2000003300, 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint (rxXiv:203.03350 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1rXiv:1503.0253) 2015.
Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J Yang, Jiihhng Liu, Chenchen Zhang, Linzheng Chai, et al. Opencoder: The open cookbook for top-tier code large language models. arXiv preprint arXiv:241 I. 1.04905, 2024.
Yikun Jiang, Huanyu Wang, Lei Xie, Hanbin Zhao, Hui Qian, John Lui, et al. D-llm: A token adaptive computing resource allocation strategy for large language models. Advances in Neural Information Processing Systems, 37:1725-1727 1749, 2024.
Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kyu Song. Shortened Ilama: A simple depth pruning for large language models. arXiv preprint arXiv:2402.028344 11:1, 2024.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: (4f2. 69800, 2014.
Ziyue Li, Yang Li, and Tianyi Zhou. Skip layer or loop it? test-time depth adaptation of pretrained 1lms. arXiv preprint 2rXiv:2507.07996 2025.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang. Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of machine learning and systems, 6:87-100, 2024.
Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W Mahoney, and Yaoqing Yang. Alphapruning: Using heavy-tailed self regularization theory for improved layer-wisg wise pruning of large language models. Advances in neural information processing systems, 37:9117-9152 17-91 2024.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large 2023.
language models. Advances in neural information processing systems,
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02755 02789, 2018.
Reiner Pope. Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of machine learning and systems, 5:606-663 624, 2023.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu. Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL httpstt::: / / xixvvvvvv vv. orgg abs/ / 241 2. 5115
David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of -e--etsss: Dynamically allocating compute in transformer based lan- guage models. arXiv preprint 0222558 2024.
12

---

Published as a conference paper at ICLR 2026
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Common- sense reasoning about social interactions. arXiv preprint arXiv:1904.09725 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0633 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Dmitriy Shopkhoev, Ammar Ali, Magauiya Zhussip, Valentin Malykh, Stamatios Lefkimmiatis, Nikos Komodakis, and Sergey Zagoruyko. Replaceme: Network simplification via layer pruning and linear transformations. arXiv preprint arXiv:2505.02819, 2025.
Sharath Turuvekere Sreenivas. Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Ameya Sunil Mahabaleshwarkar, Gerald Shen, Jiaqi Zeng. Zijia Chen, Yoshi Suhara, Shizhe Diao, et al. Lim pruning and distillation in practice: The minitron approach. arXiv preprint arXiv:2408. 1176, 2024.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11698 2023.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Zhen Tan, Daize Dong, Xinyu Zhao. Jie Peng, Yu Cheng, and Tianlong Chen. Dio: Dynamic layer operation for efficient vertical scaling of llms. arXiv preprint arXiv:2407. 10000, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Homer F Walker and Peng Ni. Anderson acceleration for fixed-point iterations. SIAM Journal Numerical Analysis, 1755)1115- 2011.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared Ilama: Accelerating language model pre-training via structured pruning. arXiv preprint 10.i::2310006644 2023.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning, pp. 38087-38099. PMLR, 2023.
Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey knowledge distillation of large language models. arXiv preprint 40Xiv:2402.13116 2024.
Yifei Yang, Zouying Cao, and Hai Zhao. Laco: Large language model pruning via layer collapse. arXiv preprint urXiv:2402.1 187, 2024.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your mathematical questions for large language models. arXiv preprint 122i:2239..1284, 2023.
Ruike Zhu, Hanwen Zhang, Tianyu Shi, Chi Wang, Tianyi Zhou, and Zengyi Qin. The 4th dimension for scaling model size. arXiv preprint arXiv:2506. 18233, 2025.
13

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1
STOCHASTIC JACOBIAN-FRER BACKPROPAGATION
We present the pscudo code of SJFB method in Sec. 3.1 follows.
Algorithm Stochastic Jacobian-Free Backpropagation
Require: input hidden states h', target hidden states + Hyperparameters: Max iterations M
1: function TRAINSTEP(F))
2: - zero li')
3: for m7 sampled uniformly from to M/2 do
Not included in backpropagation
4:
< (=,, h²)
5: end for
6:
z.detach()
7:
for n. sampled uniformly from to M/2 do (=,(z, 1
8:
9:
end for
Compute loss (F,(F, h 2) (see Eq. 9 Backpropagation Caa)
11:
12: Optimize(Fi)
end function
A.2 GPPO
Baseline Metrics. We formulate the baseline metrics(cosine similarity and perplexity) in Sec
4.3
| -
Icos = arg max Exre
(15)
| [1
(ppL arg max Ex exp(
Σ log Pf pp (2t (x)),
(16)
where h is the hidden states, is input tokens and C is the calibration dataset. Fu.+M-1 represents LLM F with M consecutive layers pruned starting from and T is the length of context.
Stability. We visualize the loss curves of GPPO training based Qwen225-1.55-Insttee Instructt under code synthesis task. Fig. shows that GPPO successfully learns the best layer pruning index (?" 2) with increasing rewards.
Fea Over Steps
Centt Over Steps
200
Cca
Valle
S0mp
Rewards Overr Steps
Policy Values of Layers
Polcy NAlaa
S0pp
12op
Figure 7: GPPO loss curves, including the cross entropy, distillation, rewards and the value of policy parameters.
14

---

Published as a conference paper at ICLR 2026
Costs. We analyze the overhead introduced by GPPO in Table Given the calibration dataset with size 10k, training ELM with M pruned layers on Qwen2.5-1.5B-Imm -nS-tructttt for epochs requires 0.15 GPU-days with 17.5 GB memory. Enumerating all possible start layers would therefore cost 3.15 GPU-days, which clearly expensive. GPPO circumvents this issue through a lightweight policy network with LoRA-based updates: with only 2 epochs of training, it converges to the optimal start layer in 0.07 GPU-days, using 21.5 GB memory. Compared with heuristic metrics such as Perplexity and Cosine Similarity, computational costs similar but resulted configurations lead to substantially worse accuracy while GPPO provides an efficient and reliable solution.
Table 5: Training cost and pruning performance of GPPO compared with other metrics.
Method
Time (GPU-days)
Memory (GB)
Start Layer
MBPP
ELM training (single)
0.15
17.5
Enumeration
3.15
17.5
2
70.6
GPPO
0.07
21.5
2
70.6
PPL.
0.99
10.1
8
66.6
Cos.
0.03
8.7
65.4
A.3 ABLATIONS ON ELM
Ablation on Fixed-Point Module. We conduct ablation study on fixed-point layer module in com- monsense reasoning task. ELM(FFN) behaves significant performance degeneration compared with ELM(Layer), indicating the necessity of using transformer layer to compress the pruned module.
Table 6: Commonsense reasoning performance on different fixed-poine module designs.
PIQA
SIQA
OBQA
MMLU
ELM(FFN)
82.6
74.7
81.2
51.6
ELM(Layer)
83.1
76.9
83,8
54.0
Ablation on W, initialization. We present the distillation loss value the first 4k training steps on coding task. Results shows that initializing W,, identity significantly accelerates the convergence of ELM training compared with random and also achieves better performance in MBPP.
Table 7: Distillation loss and MBPP performance of different initializations of W,.
1k
2k
3k
4k
MBPP Acc.
random
210.000
0.824
0.459
0.398
0.355
69.8
identity
2.172
0.272
0.198
0.152
0.154
70.6
Ablation on SJFB. We compare the training of ELM with or without SJFB based on Qwen2.5- I.SB-Instruct in math task. The dataset size is 395A with 4 epochs. ELM without SJFB costs 7.1 GPU-days with 20.1 GB memory and ELM with SJFB costs 6.1 GPU-days with 17.5 GB, reducing about 14% training time and 13% memory cost.
Base LLMs with different depths. We provide experiments over commonsense reasoning task and code synthesis based Qwen2.5-3B Insttrcctt. which contains 36 layers and we prune 10 layers. Table 8 shows that ELM still achieves 98.7% % performance of Dense in average.
Table 8: Performance of ELM based on Qwen2.5-3B-Insttee with M = 10.
Ratio
PIQA
SIQA
OBQA
MMLU
GSM8K
MATH
HumanEval
MBPP
Avg.
RP
Dense
0.00%
85.7
87.0
61.0
77.4
37.5
87.8
75.8
100.0%
ELM
27.8%
85.3
80.5
84.8
60.5
78.7
36.6
86.0
73.0
73.2
98.7%
15

---

Published as a conference paper at ICLR 2026
A.4 CONSISTENCY OF ONE-STEP KV CACHE
Under appropriate circumstances in ELMs, inference with One-Step KV cache (Eq. 14) converges to the same result as Multi-Step KV cache (Eq. 13) We introduce the proposition of consistency between two methods follows.
Proposition If the fixed-point function yp satisfies that the norm derivatives corresponding to 0e de key and value tensors aK 8V is bounded, fixed-point iteration with One-Step KV cache (Eq. 14 converges to the same result as iteration with Multi-Step KV cache (Eq. 13p.
Proof. We denote 27+1 the convergent value with Multi-Step KV cache
lim 4p (K(,)
(k) T.TT
V1
1..T TT+ hr+1),
(b)
(17)
10-000
Let Ki..., Vi... represent the convergent value of KV cache:
k000 I...T Ki...r
lim K(A
(18)
lim ((k) Vi.... 2000
The sequence of results with One-Step KV cache is formulated
(k+1)
p (K..... VT... T hT+1).
(19)
T+1
Considering the fo-morm difference between two iteration sequences:
(k) 211 7 (| Vi...: 21t1. - 2
(k+1)
(20)
According to the Triangle Inequality, we have
|2 zi+1ll (llpp(Ki... V (R) ( sp (K(K (k) ((()) (k)
(k+1)
I..To
27 hT+1)l
(21)
(K T.
(k) (((b) (k)
I...T TT hT+1) xT+1ll
For the first term, we use first-order Taylor expansion
(k) |I((K....r T, VI...T, 2I41, 2..,,,244,, hr41))+)++)))) (K(( (k) (k)
2
1..27 1-. 2T+1 hr+1))l
=
0K
0e
(K T...T K ( || +
1V (V v( ||| + P((k)
02
(22)
<
Bi
K Ki... (b) ( + Te (V........ ((*)))] + R(k)
JK
1=rll) represents the remainder term. Given the condition that the norm of derivatives 5K ΦV is bounded and the convergency of KV cache in Eq. 18. the limitations of the first term zero:
where ((k)
a(I Ki...
K (2) o( Vi
de By
lim IIp(Ki..... Vi...r *T hTT1) - (K((( Ty ( k-yo
(k) TT41 hT+
(23)
For the second term, due to the convergency Eq. we can give the limitations
lim
|
T: T+1 hT+1) - +1 =
(24)
T.
Overall, the limitations of the difference is
lim
(k+1) T+1
2 0,
(25)
which means that iteration with One-Step KV cache (Eq. 14 converges to the same result as iteration with Multi-Step KV cache (Eq. 13).
16

---

Published as a conference paper at ICLR 2026
A.5
ALGORITHMS OF SOLVERS
Simple iteration. The simple solver loops the fixed-poing function until meets the equilibrium state.
Algorithm 3 Simple iteration
Require: Function pt h). init 20)), tol max iters M Ensure: Equilibrium ="
1: k +-00
2: while k <, M do
3: x(++1] t ((2(() h))
4: if (x(11 - 2 2 < then
5:
return x" = z (k+1)
6: end if
7: k <- ++1
8: end while
9: return z" = (M)
Broyden's method. The Broyden solver is a quasi-Newton approach for solving nonlinear fixed- point equations, where the Jacobian (or its inverse) iteratively approximated by low-rank updates. In our implementation, we approximate the inverse Jacobian matrix and omit the step-size search procedure, e.g., line search, as typically involves uncontrollable number of function evaluations. We adopt a fixed step size of 1 by default.
Algorithm 4 Broyden's 's Method
Require: Function g(2) p(2, h) 2, init 20) max iters M, tol 8
Ensure: Equilibrium z*
1: Initialize U.V 0
P low-rank factors of inverse Jacobian
2: 6+00 0
3: g (z)=(0))
4: while k < M 1 do
5: if Ig
2 < then
6: return n" = (1)
7: end if
8: Compute direction:
p = --(-1+UV (8)
9:
Update variable:
> default step size as
x (k+1) = (() + p(())
10:
Evaluate new residual: g (k+1) = g(2(h+1))
Define differences:
Az= (k+1) -(())
Ag = = (k+1) (()
12:
Compute rank-1 vectors:
(1) = --I++UV A=
(4) Az (-1 UV >g
(n(R))TA) Ag
13:
Update low-rank factors:
U U, ( V - V. (
14: k- k 1
15: end while
16: return z* = (M)
17

---

Published as a conference paper at ICLR 2026
Anderson acceleration. The Anderson solver accelerates convergence by updating the current state as linear combination of previous iterates. The coefficients a are obtained by solving regularized optimization problem. To compute them, employ Lagrange multiplier formulation that yields a linear system, and fall back to least-squares solution when the system becomes singular. The length of history states for interpolation is set to 5.
Algorithm Anderson acceleration
Require: Function p( h), init = 20"), tol 6, max iters M, history length m Ensure: Equilibrium
1: 4-20 =0)
2: h --0 0
3: while k < M do
4: if < 2 then
5:
(k+1) t- (02(1)) h)
D no history yet
6: else
7:
Let my f min(m, k)
8:
Form difference matrices
P E Ruxma
G - (+-+ (8-m)))
2(6) z (A-11)
9:
Solve optimization problem:
min acR
|Ga subject to 1" a =
10:
Leads analytical form:
0
A
1
1
G
G
0
11:
2' - Xxn (1=(k-it1)
b. interpolation
12:
2k++1) x- p(z h))
13:
end
14:
if [k(++) - =14) 2| then
return x" x (k+1)
15:
16:
end
17: k< ++1
18:
end while
return x" (M)
A.6 VISUALIZATION ADAPTIVE ITERATIONS
Iteration Steps per Token for Different Solvers
smtzle: brayden
Iteration Steps
10-
Step
Theredore
Figure 8: Iteration steps per token for different solver methods. Color intensity represents step count, with darker shades indicating more iterations required.
18

---

Published as a conference paper at ICLR 2026
We visualize generation example showing the number of iteration steps per token under the ap- plication of One-Step KV cache (Sec. 3.3) and fixed-point solvers (Sec. (Sec.B.4), (.4), using Qwen2.5-1.5B. Instruct the base model. The pruning depth is set M 14, and the convergence threshold is = 0.5. We represent the step counts as colored rectangles, where different colors indicate different numbers of iterations. As shown in Fig. [ the solvers adaptively allocate different iteration steps across tokens, and the acceleration methods substantially reduce the overall computation compared simple iteration. Consistent with the results in Table 4 Anderson acceleration achieves the best performance. In contrast, Broyden's s method exhibits limited robustness, as its quasi-Newton update relies on low-rank Jacobian approximations without explicit step-size control, which often leads to unstable convergence or suboptimal fixed points in practice.
A.7 EXPERIMENTAL DETAILS
The details of our experiments are elaborated as follows.
Datasets and Prompts. Our experiments conducted on three downstream tasks.
Commonsense reasoning. The training dataset is composed of training sets from PIQA, SIQA, OBQA, MMLU and includes 154k instructions in total. The prompt is set follows.
prompt-"N
Choose the correct answer with the following question. An
Mathematical solving. The training dataset is from MetaMathQA, composed of 398k math instruc- tions generated from the training sets from GSM8K and MATH. The prompt is set as follows and we adjust the output to produce final answers after 'The answer is:" in the end for convenience of extracting answers.
prompt="*"
Please sclve the following problem. step by step. When you reach the answer, please output the answer after "Te answer is: at the end of the response. An
Code generation. The training dataset from the educational innstruct and package_ insttuctt part of OpenCoder's Stage 2 SFT dataset, composed of 289k Python coding instructions. The prompt is set as empty to adapt to the content evaluation benchmarks HumanEval and MBPP.
The prompt template follows the default template of Qwen-2.5 & Llama-3.2 models in the library t trans formers. The content of templates are as follows. Variables inst ruct ion and output in the template denote specific content corresponding to each data entry.
Qwen
<lim_ start >sssttm
You are a helpful assistant < <nin_ I (prompt)
{instructicn) in
<lim_start passistant
{output [ |
Llama
<| [lbeginnef__re htt.o//eeeeeetppatttttaaeaeaeeee >sssttsmsssssmsic
You are helpful
assistant. eot id
>user< end header_id/>
19

---

Published as a conference paper at ICLR 2026
[prompt] finstruction]</eott start_heeetttttteeeereere >ase |
[output)</ect_iie
Baselines. We reimplement baseline methods mentioned in Sec. to align all the experimental conditions with our method.
Dense represents the finetuned original LLMs with LoRA for each downstream tasks.
Sheared-LLaMAAXia et al., 2023 is representative structural pruning method, which prunes at- tention heads, the intermediate channels of FFN, the hidden dimension and layers. optimizes learnable to regularization mask for subnetwork searching and applies LoRA to finetune the pruned models.
Shortened-LLaMA Kim et al. 2024) utilizes Perplexity (PPL) to prune unimportant layers and apply LoRA to finetune the pruned models. The metric evaluated based on the cross entropy:
0pr = Exne exp(
Σ log Po p (x))x)))))
(26)
T
where C represents the calibration dataset and Fi represents the transformer with layer / removed. We pruned layers according to the descending order of Opp..
Replaceee(Shopphierr al. 2025 replaces intermediate consecutive transformer layers with linear option. ReplaceMe(LS) utilizes least-square to solve the fa-distance of feature alignment and ReplaceMe(Cosine) utilizes adam method to solve the cosine distance. It prunes the layer interval with cosine similarity and the first pruned layer index is defined as Eq. 15
LLM- Streamline(Chen et al. 2024 replaces intermediate consecutive transformer layers with a lightweight network. LLM AM-SStraalinn(Layer utilizes transformer Layer and LLM- Streamline(FFN) utilizes feed-forwar network. It aligns the feature with di-sttancce and prunes layers with cosine similarity (Eq. 15).
To clearly demonstrate the differences between baselines and our method, we take the experiments in mathematical solving task on 1-.52.5-------sstii as an example and the implementation details are summarized in Table
Training. For finetuning, we apply the consistent training settings across different baselines for fair comparisons. Specifically, we utilize Adam optimizer (Kingma Ba 2014) with a linear warmup on learning rate for the first 10% training epochs and cosine annealing schedule for the rest. The epochs finetuning are shown in Table The maximal learning rate is 2 x 10 and the minimal and linearly increase Ace from O for
is 10-6 For syper--arameetrs, the last epochs. The context length of tokens is 2048. We set the clipping grad norm to 0. for stable training. The batch size on per device is 1. For pruning phase with training, our method and Sheared-LLaMA keep the same settings, which cost 2 epochs. GPPO mask is initialized as zero tensors and the size of reward set p (the frequency of updating the reward model) is 10 and the frequency of updating the policy model is 2.
set Ausaill
Evaluation. For commonsense reasoning tasks, we generate answers with greedy decoding. The answer is marked as correct if the model produces the same option with target in lower case. For mathematical solving tasks, we generate answers with greedy decoding. For GSM8K, the answer marked as correct if the numerical difference with target is less than 10-5 For MATH, the answer is marked as correct the answer string is the same as target. For code generation tasks, we generate 10 outputs with temperature as 0.9 and top.p as 0.9, We use bigcode-evalaation-anness (Benes Allal et al. 2022) to evaluate the pass@ metric of outputs. To alleviate the misalignment in the format of generated code contents, make simple post processing for benchmark HumanEval, that is preserving existing code contents (import necessary pacakges) in the prompt before concatenating the prompt and generation codes.
20

---

Published as a conference paper at ICLR 2026
Table 9: Implementation details of baselines in mathematical solving task on Qwen2.5-1.5B- Instruct.
Shoriened- LLaMA
Dease
Sheared-LLaMA
ReplaceMe
LLM-Steaalliee
ELMrOurs)
Praning Metric/Modules
mask
Perplexity
Cosine Sim.
Cosine Sim.
GPPO mask
Pruning Calibration
Data Size
FULL
36
3k
FULL
Pruning Epochs
Pruning Layers
6.7.9.10]
12.13,14.15
12.13.44155
6,7.8.9
Lineur Transfommation
Finetuning Module
LaRA
LoRA
LoRA
Layer/ PFN
Layer
Medule Setting
Parameter Size(M)
Finetuning
Distill Loss (MSE / Cos)
Distill Loss (MSE) SFT
Objective
SFT
SFT
SPT
Distill Loss (MSF)
Finctuning Data
Size
FULL
FULL
FULL
FULL
FULL
Finetuning Epochs
A.8 GENERATION EXAMPLES
We sample several examples from mathematical solving and code synthesis to demonstrate the gen- eration quality of ELM. The base model is Qwen2.5-1.5B- Insttruuttt and the number of pruning layers M 8.
GSM8K
Instruction:
Gunter is trying to count the jelly heans in jar. He asks his friends how many they think are in the jar. One says 80. Another says 20 more than half the first one. A third says 25% more than the first one. What is their average guess?
ELM:
The first friend says there 80 jelly beans in the jar.
The second friend says there 20 more than half of the first friend's guess, so 80/2 20 60 jelly beans.
The third friend says there are 25% more than the first friend's guess, so 80 (80 0.25) 80 + 20 100 jelly beans.
To find the average guess, add up the guesses and divide by the number of guesses. The total number guesses 3, the average guess (80 + 100) 3 240 80 jelly beans. ##iH 80
The answer is: 80
Dense:
The first friend says there are 80 jelly beans in the jar.
The second friend says there are 20 more than half of the first friend's guess, so 80/2 + 20 = 60 jelly beans.
The third friend says there 100 jelly beans. To find the average guess, we add up all the guesses and divide by the number of guesses. The total number of jelly beans guessed is 80 60 100 = 240.
% than the first friend's guess, so 80 (80 0.25) 80
Since there are 3 guesses, the average guess is 240/3 80 jelly beans.
#### 80
The answer is:
21

---

Published as a conference paper at ICLR 2026
ННН MATH
Instruction:
Given ||v| = and |||| 8, find the largest possible value of
projw vll.
ELM::
The projection of v onto w is given by
proj
|w W.
The magnitude of this projection is
projw vll
v
|w|]
w
v w |w |/|2
||w]|
Since ||w|] 8, this hecomes
projw v/1 |v w|
v w
8
8²
8
The largest possible value of |v w| |v] ||w| || projw v/l The answer is:
5
40, the largest possible value of
5
Dense:
The projection of w is given by
V. w
proj v
1/w/12 W.
The magnitude of this projection is
proj vll
w
|v w
|
1
||
8
Since ||v|| = 5 and ||w|| = 8, the maximum value of v w Therefore, the largest possible value of || projw v/| is 40 The answer is: 5
5- 40. |v|l
#* HumanEval
Instruction:
def specialFilter (nums):
"*"Write a function that takes an array of numbers as input and returns
the number of elements in the array that are greater than 10 and both first and last digits of a number are odd (1, 3, 5, 7, 9). For example:
specialFilter (115, -13, 14, -151) specialFilten [33, -2, -3, 45, 21, 109])
ELM:
def specialFilter (nums):
def isSpecia Number (n)
str_n str (n)
return n 10 and int (str_n[0]) !- o and int (stx_nl-11)
2 !=
return (isSpeciaa (x) for in nums)
22

---

Published as a conference paper at ICLR 2026
Dense:
def specialFilter (nums): count for num in nums: if num > 10:
str_num str (nam)
if int (str_num(0]) != and int (str_num (-1]) !=
0:
count +=
return count
#*# MBPP
Instruction:
Write function to search element given array by using binary search.
Tese case:
assert binary searcc((1,2.33355g3 6) == False
ELM:
def sinary_search(arr x)
low 0
high = 1en (arr) -
mid
while low <= high: mid - (high + low) // 2
if [mid]
low mid 1
elif [mid] x:
high = mid
else:
return True
return False
Dense:
def seaacy, low
high len (arr) mid -
while high:
mid (high + low) /1
if arr [mid] < low mid
elif [mid] x: high mid
else:
return True
return False
A.9 DISCUSSION ON PRUNING PATTERN
In the current paper, we focus on the simplest and most interpretable configuration: compressing single continuous layer group into one fixed-point layer. This choice is aligned with prior observa-
23

---

Published as a conference paper at ICLR 2026
tions that intermediate layers tend to be more compressible. and it allows to clearly analyze the behavior of fixed- point compression without introducing additional degrees of freedom. However, more flexible pruning patterns, such as pruning multiple disjoint layer groups, valuable exten- sions. To better understand this possibility, we conducted preliminary experiments and analysis on ELM and GPPO.
Single Group vs, Multiple Groups. We compare two pruning strategies under the same total prun- ing budget M-14: I.prune one continuous group: 2.prune two disjoint groups (each with M-7).
To ensure fairness, we still use one fixed-point layer to compress all pruned layers. Experiments conducted coding tasks. Our preliminary results in Table TO show that multi-group prun- ing performs worse than single-group pruning under the current ELM formulation. This is expected because when the pruned layers become disjoint, the fixed-point layer must jointly approximate mul- tiple heterogeneous regions of the network, making the training significantly harder. This explains the consistent performance drop across both HumanEval and MBPP.
This observation also supports our choice of focusing single continuous layer groups in the main paper, which is stable and interpretable starting point for evaluating fixed-point compression.
Table 10: Different pruning patterns on ELM.
M
Pruned Layer Groups
HumanEval
MBPP
[2.3,445.66..88..
19.10.1111.1334411555
56.7
58.4
14
[2,3,445,6,7,88, 110111,12118,1,,5516
47.0
57.0
14
58.5
65.4
Extension of GPPO. The GPPO algorithm in the paper is specifically designed for single-group pruning, consistent with our modeling focus. However, the framework is flexible enough to be extended. In response, we give simple analysis of GPPO under multi-gron -grup pruning strategy. Considering pruning layers in total, we can list each pruning strategy as [[4,, Mi)....... 14,, Ma]]. where k is the total number of separate pruning layer groups, 1; is the start layer index and M, is group size, subject to 4 + M, I 1++1,4 4 + Mg I N. If there are n valid strategies under the constraint,
define policy network 0 E I= producing distribution over these candidates. LoRA adaptation can be applied per-strategy or per-layer depending on scale. In principle, GPPO optimize the policy network similarly the single-group case.
Enabling full vubitrary-pattens search requires substantially more design choices, such as how to parameterize multi-region compression and avoid combinatorial explosion. These go beyond the scope of this paper, but represent a promising direction for future exploration.
A.10 THE USE OF LARGE LANGUAGE MODELS (LLMS)
LLMs were used solely as auxiliary tools for polishing the writing of this paper. Their role was lim- ited to improving grammar, refining wording, and suggesting formatting adjustments for tables. No part the research ideation, methodology, experiments, or analysis relied on LLMs. We understand that the authors bear full responsibility for all content of the paper, including any text generated or refined with the assistance LLMs, and understand that such tools are not eligible for authorship.
A. LICENSE OF ASSETS
We do not utilize any close-source assets in this paper.
can be down-
and the license can be download from https://www.llatt wwwaeaccooooaaaeaaeee wwm/lamam//icens//////////ee//// Qwen series model can be download from Huggingface
load from https://www. http://www.llama.ccom
and the license is Apache. All training datasets and evaluation benchmarks are obtained from Hug- gingface. The license of PIQA is Academic Free License v3. The license of SIQA, MBPP is CC BY-NC 4.0. The license of OBQA, MetaMathQA is Apache. And license for MMLU, GSM8K, MATH, OpenCoder, HvmanEval is MIT acknowledge.
24


Reviews:
[{"review_id": "3BaS1k4Mc8", "reviewer": "Reviewer_4aYQ", "summary": "This paper proposes a compression method for LLMs called \"Equilibrium Language Models\" (ELMs). The approach replaces a continuous block of layers in the model with a single lightweight equilibrium layer, which computes its output through iterative solving, thereby achieving parameter compression. To support this framework, the paper introduces three auxiliary components: 1) a reinforcement learning-based \"Group Pruning Policy Optimization\" (GPPO) method for automatically selecting the layer blocks to be replaced; 2) a \"single-step KV cache\" mechanism designed to reduce memory overhead during iterative inference; and 3) the application of existing numerical solvers to accelerate convergence. Experiments demonstrate that this method maintains performance close to the original model across multiple benchmarks after pruning approximately 28% of the parameters, outperforming several baseline approaches.", "questions": "Same as weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "A New Perspective on Compression\nThe paper rethinks model pruning as an implicit depth problem. This offers a new way to approach this field, showing that we can use iterative computation to replace a large number of parameters.\n\nStrong Empirical Results \nThe paper shows excellent results in balancing parameter reduction and accuracy, outperforming strong existing baselines. This demonstrates the great potential of the ELM framework.\n\nEfficient Memory Optimization\nThe \"One-Step KV Cache\" is a simple but effective innovation. It directly solves the huge memory overhead that the iterative method could cause during LLM inference. This is especially valuable for long sequence generation.", "weaknesses": "Lack of Key Inference Latency Evaluation\nThe authors claim the method is friendly for edge deployment, but they don't provide any real-world speed data (like tokens/sec) to support this. The iterative process, the solver, and the convergence checks all add extra overhead. The final latency might even be worse than the original model. Without this data, the practical value of the paper is questionable.\n\nMissing Analysis of GPPO's Practicality and Cost\nAutomatic search is nice, but its cost is critical to determine if it's feasible. The authors need to quantify the search cost of GPPO (e.g., how many GPU-days it takes) and discuss its cost relative to the final model training. \n\nLimited Pruning Strategy\nThe method is limited to pruning a single, contiguous block of layers. This is a strong assumption and may not be the best approach. The authors should discuss why they made this choice (e.g., technical limitations or to simplify the problem) and explore the possibility of extending it to more flexible pruning patterns, like multiple, non-contiguous blocks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ffzWbTW79M", "reviewer": "Reviewer_6rpi", "summary": "This paper proposes ELMs, a model compression framework for LLMs. Instead of pruning individual weights or layers, ELMs replace groups of Transformer layers with a fixed-point network. This approach reinterpretes deep sequential computation as solving for an equilibrium state. The paper introduces: 1. Group Pruning Policy Optimization (GPPO): a reinforcement learning–based policy that learns optimal layer intervals for conversion to fixed-point modules; 2. One-Step KV Cache: a technique that stores only the final iteration’s cache to reduce memory cost during inference without sacrificing accuracy; 3. Adaptive Solvers (Broyden/Anderson acceleration): used to speed up convergence of fixed-point iterations.\n\nExperiments on Qwen2.5-1.5B, Qwen2.5-7B, and Llama3.2-3B across commonsense reasoning, math, and code benchmarks show that ELMs prune ~28% of parameters while maintaining ≈99% of the dense model’s accuracy.", "questions": "1. How does the choice of fixed-point iteration solver (simple, Broyden, Anderson) affect latency in real-time inference on GPU/edge devices?\n2. Can GPPO be applied jointly to multiple layer groups (non-contiguous intervals), or is it limited to a single interval per model?\n3. What is the training cost of GPPO, and how is it comparing to enumerating method?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper reformulates deep computation as solving for an equilibrium state, which is quite novel.\n2. The approach consistently outperforms all baselines, including recent strong pruning methods. The method still reaches good performance on hard reasoning tasks (e.g., GSM8K, MATH, HumanEval), where most compression methods fail.", "weaknesses": "1. The GRPO approach seems to only search for the start of the layers to be pruned under a fixed number of pruned layers. In this case, the search space is not very huge. How is the computation cost comparing to going through all of the possible start layers? Also, GPPO adds a non-trivial training loop with policy optimization, which might not be required for simpler pruning methods. \n2. Evaluations are limited to post-finetuned LLMs. It remains unclear how ELMs generalize to instruction-tuned multi-task setups.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ULLK3v0fH4", "reviewer": "Reviewer_Vx6z", "summary": "This paper proposes Equilibrium Language Models (ELMs), a new family of Transformer-based language models designed to reduce inference-time memory usage and improve computational efficiency without sacrificing performance.", "questions": "- What happens when the pruned layers are not contiguous? For example, instead of pruning 12 consecutive layers, what if we prune two separate groups of six layers distributed across the model? \n\n\nI am open to discussing this further during the rebuttal and will be happy to increase my score if my concerns are addressed.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Well-written and clearly structured; the mathematical formulations are concise and supported by clear intuition.\n\n- Novel modeling approach: Reformulating decoding as a fixed-point equilibrium problem is original and elegant.\n\n- Parameter efficiency: Demonstrates substantial parameter pruning without requiring fine-tuning.", "weaknesses": "- While the paper emphasizes parameter pruning (~28%), it does not report key deployment metrics such as inference latency, memory consumption, or speedup over dense baselines. This makes it difficult to verify the real-world efficiency gains, especially since equilibrium solvers may incur higher per-step computation even if they reduce parameter count.\n\n- Although the paper presents an elegant formulation of Equilibrium Language Models (ELMs) and demonstrates strong pruning performance, it does not discuss the cost, stability, or convergence of obtaining these models. In particular, the Group Pruning Policy Optimization process likely introduces significant computational overhead, as it involves iterative search and training steps that may require multiple retraining cycles.\n\n- In Table 2, ELMs are not consistently better than all baselines under the same compression level. For instance, LLM-Streamline (Layer) outperforms ELMs on MMLU at M=8, and Sheared LLM performs better on HumanEval. This suggests that ELMs may not be universally superior. A more systematic comparison across different  M values and tasks is necessary to understand when ELMs are advantageous or potentially weaker than existing methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7uwR0duvge", "reviewer": "Reviewer_wdiM", "summary": "This submission proposes Equilibrium Language Models (ELMs), a novel compression framework for Transformer-based large language models targeting edge deployment. The core idea is to replace groups of consecutive Transformer layers with a lightweight fixed-point network, reinterpreting deep computation as solving for the equilibrium state of a fixed-point system. To enable ELMs, the authors introduce two key components: (1) Group Pruning Policy Optimization (GPPO), a reinforcement learning-based method to automatically identify optimal layer pruning intervals; and (2) One-Step KVCache, which reduces memory overhead by storing only the final iteration’s cache without accuracy loss. Experiments on Qwen2.5 and Llama3.2 models show that ELMs prune 28% of parameters while retaining 99% of the accuracy of dense fine-tuned LLMs across commonsense reasoning, mathematical problem-solving, and code generation tasks.", "questions": "1. What happens if the inherited Transformer layer is replaced with a simple FFN? Or if W_h is not initialized as an identity matrix? Such ablations would clarify which components drive performance retention.\n\n2. How do you choose the convergence threshold for One-Step KVCache? \n\n3. The paper mentions using Stochastic Jacobian-Free Backpropagation (SJFB) to reduce training cost , How does ELMs’ training time/GPU memory compare to baselines?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work breaks from traditional LLM pruning (e.g., weight-level, head-level, or heuristic layer pruning) by leveraging fixed-point network theory (from Deep Equilibrium Models, DEQs) to replace layer groups. \n\n2. GPPO avoids the limitations of heuristic pruning metrics (cosine similarity, perplexity) by framing layer interval selection as a policy optimization problem. \n\n3. One-Step KVCache is a critical optimization for edge deployment", "weaknesses": "1. All experiments use LLMs with exactly 28 layers (Qwen2.5-1.5B/7B, Llama3.2-3B). It remains unclear how ELMs perform on models with different depths.\n\n2. The paper highlight the significance for edge deployment, however, the paper shows no data on latency results for edge devices. \n\n3. While One-Step KVCache reduces memory, the submission only reports KV cache size (Table 3) but not total inference memory (including model parameters, activations). Edge devices often have strict RAM limits (e.g., 4GB on low-end phones), so total memory data is essential.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)