Liger: Linearizing Large Language Models to Gated Recurrent Structures
Authors: Disen Lan
Weigao Sun
Jiaxi Hu
Jusen Du
Yu Cheng
Venue: ICML 2025 poster (2025)
Keywords: Large Language Model
Linear-Complexity Sequence Modeling
LoRA
Primary area: deep_learning->large_language_models

Abstract: Transformers with linear recurrent modeling offer linear-time training and constant-memory inference. Despite their demonstrated efficiency and performance, pretraining such non-standard architectures from scratch remains costly and risky. The linearization of large language models (LLMs) transforms pretrained standard models into linear recurrent structures, enabling more efficient deployment. However, current linearization methods typically introduce additional feature map modules that require extensive fine-tuning and overlook the gating mechanisms used in state-of-the-art linear recurrent models. To address these issues, this paper presents \textbf{Liger}, short for \underline{\textbf{Li}}nearizing LLMs to \underline{\textbf{g}}at\underline{\textbf{e}}d \underline{\textbf{r}}ecurrent structures. Liger is a novel approach for converting pretrained LLMs into gated linear recurrent models without adding extra parameters. It repurposes the pretrained key matrix weights to construct diverse gating mechanisms, facilitating the formation of various gated recurrent structures while avoiding the need to train additional components from scratch. Using lightweight fine-tuning with Low-Rank Adaptation (LoRA), Liger restores the performance of the linearized gated recurrent models to match that of the original LLMs. Additionally, we introduce Liger Attention, an intra-layer hybrid attention mechanism, which significantly recovers 93\% of the Transformer-based LLM performance at 0.02\% pre-training tokens during the linearization process, achieving competitive results across multiple benchmarks, as validated on models ranging from 1B to 8B parameters.

Liger: Linearizing Large Language Models to Gated Recurrent Structures
Disen Lan
12*
Weigao Sun" 1
Jiaxi Hu
Jusen Du 14a
Yu Cheng 5 220
Abstract
PIDA
74
Transformers with linear recurrent modeling of- fer linear-time training and constant-memory in- ference. Despite their demonstrated efficiency and performance, pretraining such non-standard architectures from scratch remains costly and risky. The linearization of large language mod- els (LLMs) transforms pretrained standard models into linear recurrent structures, enabling more effi- cient deployment. However, current lincarization methods typically introduce additional feature map modules that require extensive fine-tuning and overlook the gating mechanisms used in state- of-the-art linear recurrent models. To address these issues, this paper presents Liger, short for Linearizing LLMS to gated recurrent structures. Liger a novel approach for converting pre- trained LLMs into gated linear recurrent models without adding extra parameters. It repurposes the pretrained key matrix weights construct diverse gating mechanisms, facilitating the formation of various gated recurrent structures while avoid- ing the need to train additional components from scratch. Using lightweight fine-tuning with Low- Rank Adaptation (LoRA), Liger restores the per- formance of the linearized gated recurrent models to match that of the original LLMs. Addition- ally, we introduce Liger Attention, an intra-layer hybrid attention mechanism, which significantly recovers 93% of the Transformer-base based LLM per- formance at 0.02% pre-training tokens during the linearization process, achieving competitive re- sults across multiple benchmarks, as validated on models ranging from 1B 8B parameters.
MMLU
6B4 ARC-
20,8
50.27
30
0L822% Tolono
53.9
012
ARC-c
E30
70
27
20
Hells.
0.94
Uge GLA-3
Canal sigg (2
Figure I. Liger Performance and Efficiency. Our proposed Liger recovers nearly 93% performance of Llama-3.2- 3.2-a and outper- forms pretrained gated recurrent models at only 0.02% the pre-training tokens cost.
1. Introduction
Large language models (LLMs) have demonstrated excep- tional performance across various natural language process- ing tasks (Chintala, 2023; Team, 2023; Zhu et al., 2024; Qu et al., 2024). However, the Transformer-ban ture (Vaswani et al., 2017) used in modern LLMs, with its reliance on softmax attention, suffers from quadratic compu- tational complexity. This inefficiency results in significant speed and memory challenges, particularly during pretrain- ing on long sequences. During inference, the Key- Value (KV) cache (Kwon et al., 2023) grows linearly with the input sequence length, leading to reduced inference speed and high memory usage, which severely limits the capabil- ity of these models for handling long-sequence tasks (Sun et al., 2024a). In contrast, models based linear recurrent modeling (Katharopoulos et al., 2020; Yang et al., 2023; Qin et al., 2024b; Sun et al., 2025; Du et al., 2025) pro- vide linear-time training and mussant-memory inference, offering substantial efficiency benefits and positioning them- selves as promising candidates for the next generation of foundational architectures (MiniMax al., 2025).
architec-
"Shanghai Al Laboratory "South China University of Tech- nology 'The Hong Kong University of Science and Technology (Guangzhou) "Najjing University "The Chinese University of Hong Kong. Interns Shanghai AI Laboratory. E Correspond- ing Authors: Weigao Sun <sunweigao @ outlook. coms, Yu Cheng <chengyu@ess @sse www.eeeww.e.eeeeeeeeeeeeeeeeeeee edu.kedu. Weigao Sun is the Project Lead.
While pretraining L.MSs using architectures based on linear
The source code is available at https: /tl::///iii /ttpt//gttb///////////////t com/ LtM.rraeLLM///rrrrrr..
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
and the models are available https: httss://h/gggggaaa/eeee co/ col cowlectiiooooooooooooooooooooool l innarrmoe- hub.

---

Liger: Lincarizing Large Language Models to Gated Recurrent Structures
recurrent modeling reduces costs due their linear training complexity, the high expenses of pretraining from scratch associated with large model sizes and datasets still remain a major obstacle to their adoption and practical use. This challenge has hindered the advancement of linear recurrent models. Linearizing pretrained LLMs like SUPRA (Mercat et al., 2024), MambainLllama (Wang et al., 2024) and LoL- CATs (Zhang et al., 2024a), as an emerging new direction, allows the transfer of weights from an existing pretrained model to one with linear recurrent modeling architectures at a small fraction of the original pretraining cost. The linearization approach is promising post-training tech- nique to enable efficient pretrained model deployment while preserving their performance. Gating mechanisms (Qin et al., 2024a; Sun et al.. 2023) play a crucial role in lin- ear recurrent models by controlling memory retention and forgetting, with their effectiveness widely demonstrated in such architectures. However, incorporating gate modules as additional components requires both transferring weights from pre-traine trainaiie LLMs and training these gating modules from scratch. This process not only increases the cost of lin- earization but also creates larger architectural divergence from Transformer-basse LLMs. This divergence may hinder the effective approximation of softmax attention, limiting the performance of gated linear recurrent models (Zhang et al., 2024d). Moreover, existing linearization methods often overlook the detailed design considerations of gated linear models, and the newly added modules fail to lever- age the pre-trained weights of LLMs, further reducing the efficiency of linearization.
gating modules without introducing additional parame- ters, obtaining gated recurrent L.LMs with the benefits of constant-memory inference.
We propose Liger Attention, an intra-layer hybrid attention mechanism that combines sliding window softmax attention with linear recurrent modeling. This simple yet effective design retains the essential soft- max non-linearity, accelerating the linearization pro- cess while maintaining the capabilities of pre-trained LLMs and ensuring linear-time inference efficiency.
We apply Liger to linearize the latest Llama-3 series, ranging from 1B to 8B parameters. Experimental re- sults show that Liger outperforms existing linearization methods (like SUPRA (Mercat et al., 2024). Mambaln- Llama (Wang et al., 2024) and LoICAAS (Zhang et al., 2024a)), in terms of both efficiency and its ability to preserve the original performance of pre-trained LLMs.
2. Preliminary
Transformer with Softmax Attention. Given the input sequence X = T2,,.. ..,....,........................ TT) E TRxx, with sequence length T and dimension D, vanilla transformer (Vaswani et al., 2017) adopts standard softmax attention:
Q.K. V -XQQ XWK. XWV
QK T VD
(1)
0 = Softmax ((
O MMV
In this paper, we present Liger, which stands for Linearizing large language models to gated recurrent structures, novel approach for linearizing LLMs. Liger repurposes the weights from pre-trained Transformer-basee I.LMs and introduces a novel method for constructing crucial gating mechanisms in gated recurrent structures using the key pro- jection. This approach avoids the complex attention trans- fer process found in existing linearization methods. After transforming the weights and constructing the gating mech- anisms, Liger requires only lightweight fine-tuning of the linearized gated recurrent model parameters through LoRA autoregressive training. By introducing Liger Attention, this efficient process restores further improves the model's 's per- formance with minimal linearization cost, achieving com- petitive results across a range of language modeling and understanding benchmarks while benefiting from the linear- time inference efficiency of the recurrent architecture.
where WQ. WK. Wv DDD are learnable parameters for input sequence X projection and M E TxT is mask matrix for causal modeling by preventing future information leakage in autoregressive generation task. The above paral- lel form of softmax attention in Eq. is applied for efficient training and can be rewritten in the following recurrent form during inference stage:
4:: ke, U2 = Iz WQ D. WK. & We
explq,kk T DDD;
(2)
O, Σ Σi expla.h /VD)
The standard softmax attention is highly reliant on the grow- ing KV Cache (Chou et al., 2024b; Wang et al., 2024) to recall the history "memory" for sequence modeling, which results in quadratic complexity and costly memory require- ments especially in long context setting.
Our contributions can be summarized as follows:
Linear Attention. Linear transformer (Katharopoulos et al., 2020; Qin et al., 2023) approximates softmax self-attention as the dot product of the kernel feature mapping and utilizes associative property of matrix products to calculate the self- attention weights, achieving efficient linear-time sequence
We introduce Liger, a novel method for adapting pre- trained Transformer- based LLMs into gated recurrent structures. This approach efficiently repurposes re- dundant weights from pre-trained models to construct
2

---

Liger: Lincarizing Large Language Models to Gated Recurrent Structures
modeling and constant memory consumption. Concretely, the linear attention can be formulated as follows:
3. Methodology
In this section, we will introduce our proposed Liger for linearizing large language models to gated recurrent struc- tures. We also design simple yet effective hybrid attention form, namely Liger Attention, and build the Liger archi- tecture based on it, including intra- and inter-layer hybrid architectures.
(kg)
Ot E ((q))() (kk;))
(3)
(q)) Ei=1 ((kis) '/a
o ( Σi=1 (kki
Let S; = Ei= (ks) T 0/ and # = Ei= (ks)), the above formulation in Eq. 3 can be rewritten in the recurrent form as an RNN (Katharopoulos et al., 2020):
3.1. Gate Construction by Key Projection
The parameter space of large language models (LLMs) ex- hibits intrinsic structural redundancy (Yu et al., 2024; Agha- janyan et al., 2020), a phenomenon attributed to the over- parameterization inherent in deep neural architectures. This redundancy motivates a principled approach to reformulat- ing LLMs as gated linear recurrent architectures: rather than introducing new parameters, strategically repurpose subsets of pre-trained LLM weights to serve dual roles as gating modules.
S. S + ((ke) vt:
o(q))s
Zt = =t-1 + @bke) T Ot (qt)
Although linear attention with causal mask matrix cannot use matrix associativity reduce the parallel form training complexity from quadratic to linear, its chunk-wise parallel form allows hardware-efficien sub- quadrataiiicc and partially parallel training (Yang et al., 2023; Sun et al., 2024a; 2025; Qin et al., 2024a).
Building on the design principle of gated linear recur- rent structures for optimal softmax attention approxima- tion, we propose reallocating the key projection matrix WK dual roles concurrently perform its canonical linear transformation and gating mechanism. Formally, the gating mechanism is derived via transformation of G, = fkk) = f(x, (w), where f-) operates on the pro- jected key embeddings. This parameter-sharing paradigm ensures compatibility with pre-trained weights while clim- inating the need for auxiliary trainable gating parameters, thereby preserving computational and memory efficiency.
Gating Mechanism. While linear attention (or linear recur- rent structures) are widely recognized for their linear-time computational efficiency, they have historically exhibited notable performance gap compared to standard softmax attention. To address this limitation, recent advances in linear recurrent models have incorporated gating mecha- nism, which is a critical architectural component enabling dynamic, context-aware information retention through in- put and forget gates. This mechanism allows models to selectively preserve or discard historical information, sub- stantially enhancing their expressiveness through constant memorization capacity. The integration of gating mecha- nism has consequently become a prevalent design paradigm in state-of-the -he linear attention variants (Yang et al., 2023; Peng et al., 2023; Qin et al., 2024c). A representative im- plementation, Gated Linear Attention (GLA) (Yang et al., 2023), demonstrates this principle through its mathematical formulation:
In practical implementations, gating mechanisms can be instantiated through diverse transformation strategies. Our approach employs parameter-fiee Pooling(-) operation to derive gate values, circumventing the need for additional trainable parameters. This design preserves compatibil- ity with pre- pre-trained LLM weights, enabling direct reuse of existing parameters for gate construction without archi- tectural modification. Empirical evaluations demonstrate that this parameter-efficient strategy achieves competitive performance compared to conventional trainable gating pro- jections (e.g., linear or nonlinear parametric layers), while maintaining computational efficiency and reducing opti- mization complexity.
G. S
3.2. Liger: Linearizing LLMs to Gated Recurrent Structures
At its core, Gated Linear Attention (GLA) fundamentally augments conventional linear attention through the integra- tion of gating mechanism. This modification serves generalized framework that can be systematically extended to diverse linear recurrent architectures by reparameterizing the gating term G, Specifically, G, governs the tempo- ral decay dynamics, enabling broad and flexible adaptation across variant gated linear recurrent structures.
Prior methodologies for linearizing transformer-based large language models (LLMs) typically rely on auxiliary com- ponents, such feature mapping layers, to approximate softmax attention mechanisms (Zhang et al., 2024a). No- tably, LoL.CATs (Zhang et al., 2024a) propose two-stage fine-tuning paradigm to mitigate this limitation: First stage:
3

---

Liger: Lincarizing Large Language Models to Gated Recurrent Structures
Output
Liger: Linearizing Large Language Models to Gated Recurrent Structures
Standard Softmax Attention
Gated Recurrent Structures
Head
Norm
MatMul
MatMul
+
MLP
Gated Recurrent
Softmax
Linearization
Norm
Bt
Pooling
Pre-trained LLM
MatMul
MatMul
Attention
St-1
X
Norm
Wy
We
LoRA Finetune
Frozen
Trainable
Embedding
Input
(a) Parallel Training Form
(b) Recurrent Inference Form
Figure 2. Overall Framework of Liger. We linearize the Transformer- large language model (LLM) architecture into gated linear recurrent model by 1) Replacing Softmax Attention with a Gated Recurrent Memory module, and 2) Employing LoRA to fine-tune the Liger architecture while frozen most original weight parameters. The Liger architecture enables efficient chunk-wise parallel training also enjoying cheap linear recurrent inference.
Model
Gate Parameterization
Pooling for Gate Construction
Gated Linear Attention (Yang et al., 2023)
G. = a
a: = o (Pooling(k.))
Mamba2 (Dao & Gu, 2024)
G.
ar exp( poftplus(Pooling(ke
mL.TM (Beck et al., 2024)
G = 0r
Q: a (Pooling(k:))
Gated Retention (Sun et al., 2024b)
G. 0;
ae (Pooling(k:))
HGRN2 (Qin et al., 2024c)
G: a. 1
Q: + (1 (Poolingg)))
RWKV6 (Peng et al., 2024)
G: a:
a: exp( exp (oo(Poolnnk(k)))
Gated Slot Attention (Zhang et al., 2024c)
G. a;
((Pooling(kk))
Table Gated Linear Recurrent Structures with Variations Gate G, Parameterization. Gating mechanism can be constructed
through pooling to reuse the key projection of pre-trained LLM.
Attention Transfer phase trains newly introduced modules (e.g., kernel approximations) while freezing pre-existing parameters, followed by the Second stage: employing Low- Rank Adaptation (LoRA) to fine-tune attention layers. How- ever, previous linearization approaches including LoLCATs incurs two critical constraints:
end-to-end trainability.
To advance the linearization of pre-trained large language models (LLMs) into gated recurrent neural architectures, we propose a parameter-effictte strategy that employs the canonical Softmax operation for feature mapping. Unlike existing approaches that rely trainable modules such T2R (Mercat al., 2024) or Hedgehog (Zhang al., 2024b) to approximate attention dynamics, our method uti- lizes Softmax to inherently normalize query and key rep- resentations. This normalization ensures bounded magni- tude the query-key product space, critical property for faithfully replicating the numerical stability conventional softmax attention within linearized frameworks.
The dependency supplemen- tary feature mapping and gating modules to replicate soft- max attention outputs precludes direct reuse of pre-trained LLM parameters, necessitating non-trivial architectural modifications.
Architectural Overhead:
Optimization Fragility: The sequential training paradigm introduces brittleness, as end-to-end fine-tuning is infeasi- ble due to the interdependency between the frozen base model and the auxiliary components.
By eschewing auxiliary trainable components, our design eliminates architectural dependencies on attention trans- fer mechanisms. This divergence from sequential training paradigms (e.g.. frozen base models with incremental mod- ule updates) enables fully end-to-end fine-tuning without
These limitations hinder extensibility to modern linear recur- rent architectures incorporating gated mechanisms, which require seamless integration with pre-trained weights and

---

Liger: Lincarizing Large Language Models to Gated Recurrent Structures
compromising compatibility with pre-trained LLM weights.
Output
The resultant gated recurrent architecture, formulated as:
Head
MLP
MLP
Norm
q: 2 W k; - Wk. v, 2; Wv
G2 Pooling(k:)
(6)
Norm
Attention Transformer Block
Softmax
ligar Attantion
With the generated gate G, the recurrent update rule and the followed output computation will be:
Transformer Block
Embeedding
S. G. o S.-1 + ((k,) )o
Input
O, - (s
Figure 3. Liger Hybrid Architecture. Liger adopts intra-hybrid
Here all the trainable parameters Wq. WK. Wv are in- herited from the pre-trained LLM. This method of gating mechanism construction
Liger Attention and inter-hybrid model architecture by stacking a layer standard attention Transformer blocks every a few (e.g. layers of Liger Transformer blocks.
be extended to various gated linear recurrent structures, as shown in Table 1. Prior linear recurrent models employ learnable feature mapping func- tions (denoted as ((-)) to compute similarity representations between query (q,) and key (k:) vectors, which introduce superfluous trainable parameters while generating output distributions that deviate from the canonical Softmax atten- tion distribution inherent in pre-trained LLM. Such distri- butional discrepancies consequently degrade the efficacy of linearization by impairing compatibility with the original attention mechanisms. We found that feature mapping (-)) can be effectively approximated via simple normaliza- tion operation (Softmax(-) (-) in our implementation), thereby eliminating the requirement for computationally intensive at- tention transfer or distillation procedures while maintaining fidelity to the target attention distribution. Eq. 7 preserves the expressivity of softmax attention while inheriting the computational efficiency of linear recurrent models. This unification of architectural simplicity and functional fidelity addresses key limitations in prior linearization methods.
we apply Low-Rank Adaptation (LoRA) (Hu et al., 2021) specifically to the linear recurrent layers of large language models (LLMs), focusing on the fine-tuning of the weight matrices WQ: WK. K. Wv. Instead of training all model parameters, LoRA decomposes the adaptation term AO into two low-rank matrices B and A, such that AO = BA, where B E Pxr, A E RTXD, and r < D with r typically set to a small value, such as 8. Our empirical results demonstrate that LoRA consistently outperforms full-rank fine-tuning, offering a more efficient and effective approach for LLM linearization.
3.3. Liger Attention: A Hybrid of Sliding Window Attention and Gated Recurrent Modeling
Building upon previous works that combine softmax atten- tion with linear attention (Katharopoulos et al., 2020; Arora et al., 2024; MiniMax et al., 2025), we propose an intra- layer hybrid attention mechanism, termed Liger Attention. This method integrates a hybrid form of Gated Recurrent Modeling (GRM) and Sliding Window Attention (SWA) (Beltagy et al., 2020) with narrow softmax attention win- dow size, by blending their outputs in a weighted manner. Specifically, the formulation is given by:
Following the initialization of the model with pre-trained LLM weights and its architectural reconfiguration into gated linear recurrent framework, we employ next-token pre- diction as the fine-tuning objective to recover performance in the transformed architecture. Formally, we parameterize the adapted weights as θ = O, + Aθ, where AO denotes the incremental adjustments required to align the original transformer-basee parameters with the gated linear recur- rent structure. The optimization objective minimizes the cross-entropy loss L for autoregressive prediction over input sequences 71
O: LigerAtun(qe ki, ve)
(9)
a GRM(q, k,, + SSAA (q. k..
where GRM denotes Gated Recurrent Modeling (and its variants) in Eq. 3 and SWA refers to Sliding Window Atten- tion, which is variant of softmax attention expressed in Eq. I formulated in Eq. 10. The parameters and 3 control the relative contributions of each attention mechanism. We found that setting the sum of the two parameters a and / to is particularly critical for linearization (simply sets to 0.5 each in our implementation), which can better approximate the original attention output distribution.
L=
Σ log (e(
(8)
This approach circumvents the need for auxiliary training stages (e.g., attention transfer) by directly optimizing the gated linear recurrent architecture mnndto-end, thereby pre- serving the computational efficiency and parameter effi- ciency inherent to the original L.LM.
5

---

Liger: Lincarizing Large Language Models to Gated Recurrent Structures
Training Tokens (B)
PiQA
ARC-e
ARC-c
Hella. acc oec_nomm
Wino.
MMLU
Avg.
Avg.
Model
acc
acc 1
aec_norm
(S-shot)
t
(no MMLU)
Mistral-7B
8000
80.6
80.7
53.9
81.1
74.3
62.6
72.2
74.1
SUPRA-Mistal-7H
100
80.4
75.9
45.8
77.1
70.3
34.2
64.0
69.9
LoLCATs-Mistrl-7B Attn. Trf.
0.02
79.8
79.3
51.7
48.3
74.2
23.0
59.4
66.7
LoLCCAT--Mistal-78 LoRA
0.02
77.3
74.9
45.1
40.9
67.9
23.0
54.8
61.2
Lol.CATs-Mistra--7
0.04
79.7
78.4
47.4
58.4
71.0
23.7
59.8
67.0
Liger-GLA-MMitral-22 (Ours)
0.02
80.1
78.7
49.3
76.3
70.1
36.3
65.1
70.9
Llama-3-88
15000
79.4
80.1
53.2
79.2
72.9
65.3
71.7
73.0
SUPRA-Llama-3-88
20
78.9
75.1
46.5
71.7
65.8
40.9
63.2
67.6
Mambu2-Llama-3-99
74.1
48.0
70.8
43.2
61.9
65.6
Mamhu2-I -amm-22--- 50%
81.5
78.8
58.2
79.5
$6.7
71.0
73.9
Lol.CATs--llma-3--0 Atin. Trí.
0.02
78.4
79.3
51.9
51,6
73.4
23.5
59.7
66.9
LoLCCAT--Lamma--53 LoRA
0.02
72.4
72.6
44.3
34.6
68.0
23.0
52.5
58.4
LoLCAAT--Llama---8
0.04
80.1
80.4
53.5
63.4
72.9
42.1
65.4
70.0
Liger-GLAA-lamm---8BB (Ours)
0.02
80.3
81.1
52,5
76.3
72.0
43,4
67.6
72.4
Table Lincarized LLMs Comparison. Liger outperforms other linearization method language modeling and understanding tasks with less training tokens across Mistral-7B and Llama-3-8B LLM architectures.
RQI: Can Liger linearize the pre-trained LLMs and re- cover performance more effectively compared with other linearization methods?
6, = SWA(q;: k1, vi)
Ei explq. /VD)u
(10)
Σ E explqg /VD)
R Q2: Can Liger serve universal and scalable lineariza- tion method for different L.M architectures?
where denotes the window size (set to 64 in our default implementation) for limiting the length of the lookback window for the input token. Liger Attention demonstrates strong performance in sequence modeling tasks while main- taining efficient linear complexity of (TW +D*).
RQ3: Does Liger genuinely achieves linear/subquadratie time complexity and constant memory inference?
RQ4: How effective is Liger to its key components?
3.4. Liger Architecture and Its Hybrid
4.1. Experimental Setups
The overall architecture of our proposed Liger is presented in Fig. Following the popular LLM architecture like Llama (Dubey et al., 2024), we retain the Pre-Norm layers and MLP layers with residual connection (He al., 2016), only change the softmax attention layers with Liger atten- tion without introduction of any trainable modules like feature mapping. For the each Liger blocks including time mixing layer and token mixing layer, the forward process can be formulated as:
Models and Datasets. We select two popular LLM archi- tectures: Mistral-7B (Jiang et al., 2023) and Llama-3-8B (Dubey et al., 2024) as base model for linearization. We opt for GLA (Yang et al., 2023), a general gated linear recurrent model structure, as the basis of the Liger and its hybrid architecture for lincarization. We use 50,000 high quality instruction samples of cleaned Alpaca dataset (Taori et al., 2023) during linearization process to improve instruction- following ability and recover LLM performance in language modeling tasks.
H = LigerAttn(Norm/ X)) + X
(11)
0 MLP(Norm(H)) +H
Implementation Configurations and Details. All experi- ments are implemented Py Torch and conducted on single NVIDIA A800 80GB GPU. We opt for AdamW optimizer with a learining rate of 1e -3, By default, the LoRA rank is set to 8 and alpha is 2, which means we only use 100,000 cleaned Alpaca in- struction samples (around 0.02B tokens) for gate reccurrent model lincarization. We pad the input sequence to 1024 tokens with mini batch size 1, and set the global batch size to 8 by gradient accumulaltion, following the settings in LoLCATS (Zhang et al., 2024a).
As presented in Fig. 3, we also attempt to add one softmax attention block after stacking number of Liger (or gated linear recurrent) blocks construct layer-wise hybrid model architecture.
8. The finetuning epochs is
4. Experiments
In this section, we conduct extensive experiments to answer the following research questions (RQ)):
6

---

Liger: Lincarizing Large Language Models to Gated Recurrent Structures
Training Tokens (B)
PIQA
ARC-e
ARC-c
Hella. acc_norm f
Wino.
MMLU
Avg.
Model
acc
acc_none [ee_roomm
(S-shot)
(no MMLU) t
(Transformer)
Mistral-7B
8000
80.6
53.9
81.1 79.2
74.3
62.6
Llama-3-88
15000
79.4
80.1
53.2
72.9
65.3
71.7
73.0
(LInearSubquadratic)
Mamha-7B
1200
81.0
77.5
46.7
77.9
71.8
33.3
64.7
71.0
RWKY-6-Wordd7BB
1420
78.7
76.8
46.3
75.1
70.0
69.4
69.4
LransNormerLLM-778
1400
80.1
75.4
44.4
75.2
66.1
43.1
64.1
68.2
Hawk-7B
300
80.0
74.4
45.9
77.6
69.9
35.0
63.8
69.6
Grifin-7B
78.6
72.6
39.3
(Hybrid)
StripedHyena-Nouie
76.4
66.4
26.0
Zumbu-7B
1000
81.4
74.5
46.6
80.2
76.4
57.7
69.5
71.8
Zumbu2-7B
2100
81.0
80.3
56.4
81.5
77.2
64.8
73,5
75.3
(Linearized)
Liger-Gl G----LLL----a--- (Ours)
0.02
80.3
81.1
52.5
76.3
72.0
43.4
67.6
72.4
Miger-GLA-Llamm--- (Ours)
0.02
80.6
80.7
52.7
76.9
71.4
44.4
67.8
72.5
Table Performance Comparison Pre-trained and Linearized LLMs on Common-sense Reasoning and Knowledge Benchmarks. Results span Transformer-base (Mistral-7B. Llama-3-8B), lincar/subquadratie (Mamba, RWKV), hybrid (Zamba), and our linearized Liger-GLA GLA variants language modeling and understanding tasks. Our Lincarized Liger models achieve competitive performance with only 0.02B training tokens, demonstrating efficient adaptation to gated linear recurrent architectures.
LLaMM-2 80 FA Time
Avg.
Avg.
Model
(no MMLU) f
LLATA, Verooy
mory Usagge
15.0
Ligen Momery
Llama-3.2-1B
55.1
59.9
2000
GLA-IB
46.9
$1.1
1000
LoLLCATT-LLama-322-1B
51.1
56.7
1300
GPU
Liger-GLA-LLlmma-3.-11
$2.9
59.0
Llama-3.2-3B
68.1
Decoding Longth
GLA-3B
49.1
53.8
LoL.CATs--Llaa-322-33
55.6
62.0
Figure 4. Decoding Latency Time and GPU Memory Usage of Each 8B Models, We variate the decoding length from IK 32K with fixed batch size of 16 on single A800 80GB GPU to evaluate the models' efficiency. Liger enjoys linear-time inference with constant GPU memory usage.
Liger- Ligr-GLLA-Lamma3322-3
60.7
66.5
Llama-3-8B
71.7
73.0
LoLCAAT--Llama-3-88
62.2
70.0
Liger-GLA- GLA--LLma----BBBB (Ours)
67.6
72.4
Table 4. Scalability Analysis of Linearized Llama-3 Architee-
tures across Model Sizes (1B to 8B), Liger demonstrates consis- tent scaling laws, outperforming LoLCAT' by +6.8-1 6-11 1.55 abso- lute average metrics while preserving 83-98% of base model capabilities with only 0.02B adaptation tokens.
4.2. Main Results:
Liger can recover pre-trained LLMss performance more effectively (1)
To validate the effectiveness of our proposed method, we conducted experiments series of language modeling and understanding tasks, including PiQA (Bisk et al., 2020), ARC-easy (ARC-e), ARC-challenge (ARC-c) (Clark et al., 2018), HellaSwag (Hella.) (Zellers et al., 2019), Wino- Grande (Wino.) (Sakaguchi et al., 2019) and MMLU (Li et al., 2023). The results are reported Table 2 and evaluations were done using am-evaluation-hanae al., 2024). Liger, utilizing only 0.02B tokens, achieves linear recurrent model that recovers 90% of Mistral's perfor- mance and 93% of Llama-3's performance with only 0.085% model parameters LoRA finetuning, Our method signifi- cantly outperforms other linearization baselines, incluing SUPRA (Mereat et al., 2024) and Mamba Llama (Wang al., 2024), which still need billions of tokens for linear
recurrent architecture conversion fine-tuning, and LoLCATs' linearization approach, which requires a two-stage process and twice the number of training tokens.
We also compared Liger with other pre-trained models, in- cluding the Transformer models Mistral (Jiang al., 2023) and Llama-3 (Touvron et al., 2023), linear/subquadratic models such as Mamba (Gu & Dao, 2023), RWKV-6 (Peng et al., 2023), TransNormerl LLM (Qin et al., 2023), Hawk and Griffin (De et al., 2024), as well as hybrid models like StripedHyena (Poli et al., 2023), Zamba and Zamba2 (Glorioso et al., 2024). As shown in Table 3, our pro- posed method outperformed nearly all of the pre-trained
(Gao

---

Liger: Lincarizing Large Language Models to Gated Recurrent Structures
Gated Linear
Gated Memory Formulation
Form of Gate G
Avg.
MMLU
Output Formulation
Recurrent Variants
0-shot
5-shot
Liger-GLA
S, G, & _ + k v,
o: = 9,S.
G. E R2
72.4
43.4
Liger-HGRN2
S. G.S. G.) Vi
= q.S.
G é IR
69.5
36.2
K. G.KrI+ G
k
Liger-GSA
V, Softmax q.)
G. E RM
70.5
41.2
V. G.V.-1+(0 G.)
Table Gated Linear Recurrent Model Variants with Liger. Liger can be applied to the efficient lincarization of various lincar recurrent structures with gating mechanism and achive high quality performance recovery.
linear/subquadratic models and achieve competitive perfor- mance compared with transformer-based and hybrid LLMs.
Model
Validation PPL.
Avg.
Avg.
Variants
+
1
(no MMLU) f
Liger-GLA
2.96
67.6
72.4
4.3. Liger a Efficient and Scalable Hybrid Structure QRQ2& & RQ))
Gate Proj.
3.16
63.8
68.8
Feat. Map.
9.04
43.5
40.2
Pure LA
3.00
66.1
71.5
We conducted experiments to compare efficiency in terms of decoding latency speed and GPU memory consumption of
w/o LoRA
3.23
61.7
68.1
w/o SWA
3.75
54.2
60.2
Llama-3-8B without (w/o.) Flash-Attention-2 (FA2), Llama- 3-8B with (w/)) 2 (FA2) and Liger-GLA-8P on single A800 80GB GPU. We set a fixed batch size of 16 and variate the decoding sequence length from IK to 32K with the fixed prefix input length of 128. As presented in Fig. we observe that Liger achieves linear-time decoding complexity while maintaining constant memory usage.
w/o GLA
3.01
66.2
72.0
Table 6. Ablation Study of Liger on Gated Linear Attention. We linearize Llama-3-8B into Gated Linear Attention (GLA) to evaluate the key components of Liger. We report Validation per- plexity (PPI..) on cleaned alpaca dataset after Liger lincarization and the average performance on language modeling and under- standing tasks.
We evaluate efficiency across three scales of the Llama-3 series (IB, 3B, 8B), comparing vanilla transformers, GLA, LoLCATs, and our LLA. As shown in Table 4, Liger consistently outperforms both GLA and LoLCATs while preserving 93% of the base Llama-3's performance on av- erage. Notably, GLA-IB substantially underperforms all methods (46.9% average), highlighting the necessity of our parameter-efficient adaptation strategy. The performance gap between Liger and vanilla Llama-3 narrows with model size (A = 4.8% at IB- + A = 1.8% at 8B), indicating improved architectural compatibility at scale.
individually. The results of these experiments are detailed in Table 6, demonstrating the effectiveness of proposed components in Liger.
5. Related Work
Linear Recurrent Models and their Hybrid. To address the challenges of quadratic complexity computation cost in standard softmax attention, many linear recurrent mod- els are proposed to achieve efficient training and inference. -ata-dependent gating/decay has been proved as an effec- tive mechanism control the memory changes and improve sequence modeling expressiveness (Yang et al., 2023; Peng et al., 2024; Qin et al., 2024c; Zhang et al., 2024c; Du et al., 2025). Recently, some layer-wise hybrid model architec- tures have been proposed to compensate for the lack of memory capacity in the linear recurrent models, such as StripedHyena-Nout (Poli et al., 2023), Jamba (Lieber et al., 2024), Zamba (Glorioso al., 2024), Hymba (Dong et al., 2024), Titans (Behrouz al., 2024) and Minimax-01 (Min- iMax et al., 2025). Actually, all of these hybrid attention architectures introduce extra modules (e.g. feature mapping or gate modules) that need to be trained from scratch, which increases the complexity the model architecture design and may lead suboptimal softmax attention approxima-
We conduct experiments on gated linear recurrent structure variations including GLA (Yang et al., 2023), HGRN2 (Qin et al., 2024c) and GSA (Zhang et al., 2024d), with the results presented in Table 5, demonstrating the extensibility of Liger on various gated linear recurrent structures.
4.4. Liger Framework Analysis (RQ4)
To verify the key components of Liger, we conducted abla- tion studies on the model structure. Specifically, we exper- imented with using gates generated from randomly initial- ized gate projections (Gate Proj.) instead of pooling, and adopting pure linear attention without considering gating mechanism (Pure LA). Additionally, we incorporated learn- able feature map modules (Feat. Map.), similar to Zhang et al. (2024b). We also evaluated the effects of removing LoRA (w/ LoRA), GLA (w/o GLA) and SWA (w/o SWA)
8

---

Liger: Lincarizing Large Language Models to Gated Recurrent Structures
tion. In addition, integration of softmax and linear attention shows great potential
Impact Statement
intra-layer hybrid attention paradigm, such as Agent Attention (Han et al., 2024), Based (Arora et al., 2024), GSA (Zhang et al., 2024c). However, these inter-layer hybrid attention forms have linear recur- rent modules that only focus on long-term modeling and ignore local information (Arora et al., 2024), and lack a gating mechanism and the approximation of the attention output distribution by controlling the hybrid ratio (or hybrid weights) (Han et al., 2024; Arora et al., 2024), which is particularly critical in the linearization process.
This work represents a notable advancement in artificial intelligence and machine learning, particularly in lineariz- ing the pretrained Transformer-basee models into gated recurrent structures. Liger enables the processing of much longer sequences compared to existing methods while sig- nificantly accelerating computation, making it highly benefi- cial for tasks like natural language understanding, genomic sequence analysis, and time-series forecasting. However, the enhanced capabilities and efficiency introduced by Liger also raise ethical and societal considerations, such as the potential for misuse in generating persuasive but misleading content or in surveillance applications. Nevertheless, the contributions of Liger to reducing computational overhead and energy consumption in training large models may also bring positive environmental impacts.
Linearizing Large Language Models. Linearizing or fine- tuning (uptraining) transformers to linear-RNNs could sig- nificantly reduce the cost of training a brand-new large-scale linear recurrent model architecture by distilling knowledge from pre-traaee pre-trained LLMs. Most linearization methods pro- posed to uptrain transformer-based LLMs into linear-RNNs by introducing extra feature mapping modules (Kasai et al., 2021; Mercat et al., 2024; Chen et al., 2024) or adding a loss (Zhang et al., 2024b; Bick et al., 2024; Zhang et al., 2024a) to approximate softmax attention. However, these methods have introduce extra modules that cannot reuse the existing pre-trained LLM weights and need be trained from scratch, which may not match the output of the orig- inal attention and increases the complexity of the model architecture, leading to suboptimal attention approximation.
References
Aghajanyan, A., Zettlemoyer, L., and Gupta, S. In- trinsic dimensionality explains the effectiveness of lan- guage model fin-ttniing 2020. URL https:://arr/tt o org/ abaaabss11111 /201 13255.
Arora, S., Eyuboglu, S., Zhang, M., Timalsina, A., Alberti, S., Dylan Zinsley, J. Z., Rudra, A., and Ré, C. Sim- ple linear attention language models balance the recall- throughput tradeoff. arXiv preprint 186686,002.1866,, 2024.
6. Conclusion
This paper introduces Liger, novel method for lineariz- ing Transformer-based LLMs into gated linear recurrent structures. By leveraging the key matrix weights of pre- trained standard-structur models and repurposing them to construct the gating mechanisms, Liger avoids the need for additional parameters and extensive fine-tuning, mak- ing a cost-effective and efficient linearization approach. The use of end-to-end LoRA fine-tuning restores model performance while minimizing linearization costs. Further- more, the introduction of Liger Attention enhances nearly 93% performance recovery with only 0.02% pre-training tokens, making Liger a competitive solution across range of language modeling and understanding tasks. Our results demonstrate the effectiveness of Liger on models ranging from IB to 8B parameters, offering a promising path to- ward more efficient deployment of large-scale LLMs with linear-time inference and constant memory usage.
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., and Hochreiter, S. xlstm: Extended long short-term mem- ory, 2024. URL httpt: /tttts://arxiv org/ abs/ 2205. 04517.
Behrouz, A., Zhong, P., and Mirrokni, V. Titans: Learning to memorize at test time, arXiv preprint arXiv:2501.00663 2024.
Beltagy, I., Peters, M. E., and Cohan, A. Longformer: The long-document transformer, 2020.
Bick, A., Li, K. Y., Xing, E. P., Kolter, J. Z.,, and Gu, A. Transformers to ssms: Distilling quadratic knowledge to subquadratic models, 2024. URL https:
org/ abs /2408 10189.
Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural language. In Thirty- Fourth AAAI Conference on Artificial Intelligence, 2020.
Acknowledgements
This work is supported by the Shanghai AI Laboratory.
Chen, H., Liu, Z., Wang, X., Tian, Y., and Wang, Y. Dijiang: Efficient large language models through compact ker- nelization, 2024. URL htips: / /arxivv org/ abs/ 2403.19928.
9

---

arXiv Liger: Lincarizing Large Language Models to Gated Recurrent Structures
Chintala, S. Gpt-4 moe, June 2023.
URL
Glorioso, P., Anthony, Q.. Tokpanov, Y., Whittington, J.,
https: //x. wtt.c//o//s///c soumi http://summthhhhhttttaaacc saichintal/sstatuu 1671267150101721099
Pilault, J., Ibrahim, A., and Millidge, B. Zamba: A compact 7b ssm hybrid model, 2024. URL https:// arxiv. azxiv.org//ie ors/abs////eoooo..reo.gooeeeeeer /t0p:/2455.1122
Chou, Y.. Yao, M., Wang. K., Pan, Y., Zhu, R., Zhong, Y., Qiao, Y., Wu, J., Xu, B., and Li, G. Metala: Unified optimal linear approximation to softmax attention map. arXiv preprint a. 1.0041,, 2024a
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:3312.00752,222 2023.
Chou, Y., Yao, M., Wang, K., Pan, Y., Zhu, R., Zhong, Y., Qiao, Y.. Wu, J. Xu, B., and Li, G. Metala: Unified optimal linear approximation to softmax atten- tion map, 2024b. URL https::////// http///axvvv..//////.e/e..e orgoorg//o abs/ 2411. 10741.
Han, D., Ye, T., Han, Y., Xia, Z., Pan, S., Wan, P., Song, S., and Huang, G. Agent attention: On the integration softmax and linear attention, 2024. URL https: / //aariiv..oo www.iv.ogg/aaee abs 2312.08874.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/ 1800.0544557 2018.
2018.
Dao, T. and Gu, A. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.2 20600, 2024.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models, 2021. URL https: //arxiv.//xxxv.. org/ abs
De, S., Smith, S. I., Fernando, A., Botev, A., Cristian- Muraru, G., Gu, A., Haroun, R., Berrada, L., Chen, Y., Srinivasan, S., Desjardins, G., Doucet, A., Budden, D.,
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.- A., Stock, P., Scao, L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023. URL https: //arxiv. org/ abs /2310.23100000 012.3100088....
Teh, Y. W., Pascanu, R., Freitas, N. D., and Gulcehre, C. Griffin: Mixing gated linear recurrences with lo- cal attention for efficient language models, 2024. URL https: /wtpv//aavvv/v..////v/e.././.ava org/ abs/ / 2402 19427.
Dong, X., Fu, Y., Diao, S., Byeon, W., Chen, Z., Maha- baleshwarkar, A. S., Liu, S.-Y.. Van Keirsbilck, M., Chen, M.-H., Suhara, Y., al. Hymba: A hybrid-head ar- chitecture for small language models. arXiv preprint 1:Xiv22411133776 2024.
Kasai, J., Peng, H., Zhang, Y., Yogatama, D., Ilharco, G., Pappas, N., Mao, Y., Chen, W., and Smith, N. A. Fine- tuning pretrained transformers into rnss, 2021. URL http:://aaxxxv.../a/.ososos
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are rans: Fast autoregressive transformers with lincar attention. In International Conference on Machine Learning, pp. 5156-5165. PMLR, 2020.
Du, J., Sun, W., Lan, D., Hu, J., and Cheng, Y. Mom: Linear sequence modeling with mixture-of-memoon preprint 2rX.v:2502.
2025.
Dubey, A.. Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. arXiv preprinl arXiv:2407.21783, 2024.
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, H., Gonzalez, J., Zhang, H., and Stoica, Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Sym- posium on Operating Systems Principles, pp. 611-626. 2023.
Gao, I.,, Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, F., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024. URL https: http///eeodddoe/ www.eoooooooooeoeeeeeeeeeeeeeeee wtt.records///////ooooooo 12608602.
Li, H., Zhang, Y., Koto, F., Yang, Y., Zhao. H., Gong, Y., Duan, N., and Baldwin, T. Cmmlu: Measuring massive multitask language understanding in chinese, 2023.
Lieber, O.,, Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedi- gos, I., Safahi, E., Meirom, S., Belinkov, Y., Shalev- Shwartz, S., et al. Jamba: A hybrid transformer mamba language model. arXiv preprint arXiv:2403. /9887, 2024.
Gershman, S. J., Ficte, I., and Irie, K. Key-value memory in the brain. arXiv preprint arXiv:2501.0200 2025.
10

---

Liger: Lincarizing Large Language Models to Gated Recurrent Structures
Mercat, J.. Vasiljevie, I., Keh, S., Arora, K., Dave, A., Gaidon, A., and Kollar, T. Linearizing large language models, 2024. URL https: /tte:///xxxii orgoorgggaoooo abs/ 2405. 06640.
ited sequence lengths in large language models, arXiv preprint arXiv:2401.04653 2024a.
Qin, Z., Sun, W., Li, D., Shen, X., Sun, W., and Zhong, Y. Various lengths, constant speed: Efficient lan- guage modeling with lightning attention. arXiv preprint arXiv:2405. 17381, 2024b.
MiniMax, Li, A., Gong, B., Yang, B., Shan, B., Liu, C., Zhu, C., Zhang, C., Guo, C., Chen, D., Li, D., Jiao, E.,
Li, G., Zhang. G., Sun, H., Dong, H., Zhu, J., Zhuang, J., Song, J., Zhu, J., Han, J., Li. J., Xie, J., Xu, J., Yan, J., Zhang, K., Xiao, K., Kang. K.. Han, L., Wang. L., Yu, L., Feng, I.., Zheng, I.., Chai, I.., Xing, I.., Ju, M., Chi, M., Zhang, M., Huang, P., Niu, P., Li, P., Zhao, P., Yang, Q. Xu, Q., Wang, Q.. Wang, Q., Li, Q., Leng, R., Shi, S., Yu, S., Li, S., Zhu, S.. Huang. T., Liang. T., Sun, W., Sun, W., Cheng, W., Li, W., Song. X., Su, X., Han, X., Zhang, X., Hou, X., Min, X., Zou, X., Shen, X., Gong, Y., Zhu, Y., Zhou, Y., Zhong, Y., Hu, Y., Fan, Y. Yu, Y.. Yang, Y., Li, Y., Huang, Y., Li, Y., Huang, Y., Xu, Y., Mao, Y., Li, Z.., Li, Z.,, Tao, Z., Ying. Z., Cong, Z. Qin, Z., Fan, Z. Yu, Z., Jiang, Z., and Wu, Z. Minimax-01: Scaling foundation models with lightning attention, 2025, URL https: /twpavaavvvvv/vv////v///eee/aaiv org/ abs / 22002251.0000 05311.
Qin, Z.,, Yang, S., Sun, W., Shen, X.,, Li, D., Sun, W., and Zhong, Y. Hgrm2: Gated linear mns with state expansion. arXiv preprint (tfi::2004079904 2024c.
Qu, X., Dong, D., Hu, X., Zhu, T., Sun, W., and Cheng, Y. Llama-moe v2: Exploring sparsity of Ilama from per- spective of mixture-of-expprts with post-training. arXiv preprint arXiv:2411. 157708 2024.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv: /907. 10641, 2019.
Sun, W., Qin, Z., Li, D., Shen, X., Qiao, Y., and Zhong, Y. Linear attention sequence parallelism. arXiv preprint M.XXv:2404.028800 2024a.
08313.
Peng, B., Alcaide, E., Anthony, Q. Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Der- czynski, L., Du, X., Grella, M., Gv, K., He, X., Hou, H., Kazienko, P., Kocon, J.. Kong, J., Koptyra, B., Lau, H., Lin, J., Mantri, K. S. I., Mom, F., Saito, A., Song, G., Tang. X., Wind, J., Wožniak, S., Zhang, Z., Zhou, Q. Zhu, J., and Zhu, R.-J. RWKV: Reinventing RNNs for the transformer era. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computa- tional Linguistics: EMNLP 2023, pp. 14048-14077, Sin- gapore. December 2023. Association for Computational Linguistics. doi: 10. 12016665/11222...nnnngg....
Sun, W., Lan, D., Zhong, Y., Qu, X., and Cheng, Y. Lasp-2: Rethinking sequence parallelism for linear attention and its hybrid. arXiv preprint arXiv:2502.07563. 2025.
Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang. J., and Wei, F. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.0862 2023.
Sun, Y., Dong, L., Zhu, Y., Huang, S., Wang, W., Ma, S., Zhang, Q. Wang, J., and Wei, F. You only cache once: Decoder-decoder architectures for language mod- els, 2024b. URL https: //tp://aaavvv///a///aa org/ 2205 05254.
936. URL https: htt////aaanntttlooooooooo org/ findings-emn p. 936.
Peng. B., Goldstein, D., Anthony, Q. Albalak, A., Al- caide, E., Biderman, S., Cheah, E., Ferdinan, T., Hou, H., Kazienko, P., et al. Eagle and finch: Rwky with matrix- valued states and dynamic recurrence. arXiv preprint 204::22404408892., 2024.
Taori, R. Guirajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction model. https: /ttl:///gittii thub. com/ com/tatsu-labe stanford_alpaca,
Ilama
2003 2023.
Poli, M., Wang, J., Massaroli, S., Quesnelle, J., Carlow, R., Nguyen, E., and Thomas, A. Stripedhyena: Moving be- yond transformers with hybrid signal processing models. URL https://githu. comm/hogeherrommpteesssipplleeee 2023.
Team, I. Internlm: A multilingual language model with progressively enhanced capabilities, 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models. arXiv preprint 2xxvv:232..13977 2023.
Qin, Z., Li, D., Sun, W., Sun, W., Shen, X., Han, X., Wei, Y., Lv, B., Luo, X., Qiao, Y., et al. Transnormerllm: A faster and better large language model with improved transnormer. arXiv preprint vrXiv:2307.14999 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E.,, and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Qin, Z., Sun, W., Li, D., Shen, X., Sun, W., and Zhong, Y. Lightning attention-2: A free lunch for handling unlim-

---

Liger: Lincarizing Large Language Models to Gated Recurrent Structures
Wang, J., Paliotta, D., May, A., Rush, A. M., and Dao, T. The mamba in the llama: Distilling and accelerating hybrid models, 2024. URL fttps:///axivee org/ abs/2408. 15237.
Yang, S., Wang, B., Shen, Y., Panda, R., and Kim, Y. Gated linear attention transformers with hardware-efficient train- ing. arXiv preprint arXiv:2312.06633 2023.
Yu, L.., Yu, B., Yu, H., Huang, F,, and Li, Language models are super mario: Absorbing abilities from homol- ogous models free lunch. In Forty-first International Conference on Machine Learning, 2024. URL https: //openreview net/ forum? widdidd ffai10X.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sen- tence? In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, 2019.
Zhang, M., Arora, S., Chalamala, R., Wu, A., Spector. B., Singhal, A., Ramesh, K., and Ré, C. Lolcats: On low- rank linearizing of large language models, 2024a. URL https: /ttpxaxxvvvoo////// org/ abs 2410 10254.
Zhang, M., Bhatia, K., Kumbong, H., and Ré, C. The hedgehog the porcupine: Expressive linear attentions with softmax mimicry, 2024b. URL https: arxiv.
org/ abs//ab//at 2402 04347.
Zhang, Y., Yang. S., Zhu, R., Zhang, Y., Cui, L., Wang, Y., Wang, B., Freda Shi, Bailin Wang, W. B., Zhou, P., and Fu, G. Gated slot attention for efficient linear-time sequence modeling. arXiv preprint arXiv:2409.0711 2024c.
Zhang, Y., Yang, S.. Zhu, R., Zhang, Y., Cui, L., Wang, Y., Wang, B., Shi, F, Wang, B., Bi, W., al. Gated slot attention for efficient linear-time sequence modeling. arXiv preprint 240.v:2409.07146 0466,0071666000 2024d.
Zhu, T., Qu, X., Dong, D., Ruan, J., Tong. He, C., and Cheng, Y. Llama-moe: Building mixture-of-experts from Ilama with continual pre-training. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 15913-15923, 2024.
12

---

Liger: Lincarizing Large Language Models to Gated Recurrent Structures
A. Datasets and Benchmarks
We linearize Liger on Cleaned Alpaca dataset (Taori et al., 2023) and evaluate on downstream language tasks using um-evaluation-hamae (Gao et al., 2024).
Cleaned Alpaca (Taori et al., 2023): The cleaned Alpaca dataset is a structured dataset designed for instraction-tuning of language models, containing 52,000 instructions and corresponding outputs generated by OpenAT's text-davinci-O0 engine. Each entry in the dataset includes an "instruction" field that specifies the task for the model, an optional "input" field providing context or additional information, and an "output" field with the model's response. The dataset is formatted in JSON and is intended enhance the ability of language models to follow instructions effectively.
PiQA (Bisk et al., 2020): The PiQA (Physical Interaction: Question Answering) dataset is designed to assess physical commonsense reasoning, containing 3,084 samples for testing. Each instance includes a "goal" field representing the question, two "solution" fields with potential answers, and a "label" indicating the correct solution. The dataset focuses on everyday situations requiring physical commonsense.
ARC-Easy & ARC-Challenge (Clark et al., 2018): The ARC (A12 Reasoning Challenge) dataset is a collection of 7,787 genuine grade-school level multiple-choice science questions. It is divided into two subsets: ARC-Easy and ARC-Challenge. The ARC-Easy subset contains relatively straightforward questions that test basic knowledge, while the ARC-Challenge subset includes more complex and difficult questions that require advanced reasoning abilities
HellaSwag (Zellers et al., 2019): The HellaSwag dataset is a comprehensive collection of narrative reasoning tasks designed to evaluate model's ability to predict the next event in sequence. It consists of 10,125 examples, each containing context and four possible endings, with one correct and three incorrect options. The dataset is derived from the ActivityNet Captions corpus and is structured to test the model's understanding of narrative coherence and common-sense reasoning based on the given context.
WinoGrande (Sakaguchi et al., 2019): The WinoGrande dataset is a large-scale collection of 44,000 problems designed to evaluate commonsense reasoning, inspired by the Winograd Schema Challenge but enhanced scale and difficulty. Each problem is structured as fill-n-the-blank task with two options, requiring models choose the correct answer based contextual understanding. WinoGrande is significantly more challenging than the original Winograd Schema Challenge, with state-of- of-the----ee-rr models achieving lower accuracy, highlighting its effectiveness in testing true commonsense understanding.
MMLU (Li et al., 2023): The MMLU (Massive Multitask Language Understanding) dataset is a comprehensive benchmark designed to evaluate Al models' general knowledge across a wide range of subjects and languages. It comprises 57 distinct categories, spanning clementary-Level knowledge to advanced professional topics such as law, physics, history, and computer science. The dataset has been translated into 14 languages using professional human translators, ensuring high-quality and accurate translations. This multilingual approach aims to improve the inclusivity and effectiveness of Al models across different linguistic communities.
B. Experiment Details
Our 8B
model linearization experiments are conducted on single NVIDIA A800 80G GPU. With batch size I and gradient accumulation over 8 batches, Liger method takes around 4 hours and 27GB GPU memory usage for 2 epochs end-to-end linearization, instead of any multi-stage training. All our experiments were conducted and evaluated using a fixed random seed of 0 to ensure reproducibility.
LoLCATs
Liger
C. Model Weight Changes
AW ll:
3,68c+06
2.91c+06
We analyze the model weight changes after model architecture linearization. Let W be the sum of each weight parameter of LLM, we calculate the model weight changes AW in Liger compared with Lol L.TS.. As presented in Table 7, we observe that Liger has a smaller number of changing parameters than LoLCATs, which mitigates catastrophic forgetting of pretrained knowl- edge during linearization process, thereby enhancing both training efficiency and model effectiveness.
AW
/1/W
4.82%
3.85%
AW lla
3,93c+04
1.27c+03
AW |12/W
179.29%
7.63%
Table 7. Model Weight Changes after Lin- earization. Liger has a smaller number of chang- ing model weights than LoLCATs.
13

---

Liger: Lincarizing Large Language Models to Gated Recurrent Structures
PiQA
ARC-e
ARC-c
Hella.
Wino.
MMLU
Avg.
Avg.
Model
ace ?
acc t
acc_norm
acc_norm f
acc
acc (5-shot)
(no MMLU) 1
Liger-GLA--B---Poooin
80.2
78.6
$1.6
76.7
71.6
41.7
66.7
71.7
Liger-GLA-&BB- B-LLA----K-Poliin K-Pooling
80.3
81.1
52.5
76.3
72.0
43.4
67.6
72.4
Lig Beoor-GA--B--- V-Pooling
80.7
77.4
50.6
76.4
70.4
43.7
66.5
71.1
Table 8. Results on Gate Construction from Different Components. We compare the -gA--G'A-SBB's performance of gate module obtained by pooling from query (Q). key (K) and value (V) matrices, the gate construction from key matrix outperforms others.
D. Gate Construction from Different Components
We consider constructing gating mechanisms from different components of the model. We compare the Liger-GLA-88B's performance of gate module obtained by pooling from query (Q), key (K) and value (V) matrices, and the results are shown in Table We observed that the gate construction from key matrix outperforms than others. The key matrix usually serves as memory indexing in transformer or linear model (Gershman et al., 2025), which is similar to the gate that determines which part of the memory should be retrieved or forgotten. this view, is intuitive choose Key for gate construction. MetaLA (Chou et al., 2024a) also points that the linear recurrent model needs meet the "least parameter approximation" condition achieve the optimal linear attention design. In this case, the parameters of key matrix redundant and can be used to construct the gate, which also provides motivation and theoretical support for gate derived from Key.
E. Full Results
Sequence Length
Llama-3-8B w/o FA2
Llama-3-8B w/ FA2
Liger-8B
Time
Memory
Time
Memory
Time
Memory
IK
37.92
17.50
29.36
17.26
47.83
16.37
2K
102.54
19.75
62.52
19.29
94.41
16.37
4K
312.98
24.25
151.51
23.35
185.79
16.37
8K
1062.65
33.26
436.04
31.48
367.78
16.37
16K
3882.36
51.26
1449.20
47.73
734.91
16.37
32K
OOM
OOM
1465.52
16.37
Table 9. Detailed Results on Inference Efficiency in terms of Decoding Latency Time and GPU Memory Usage. We present the decoding latency time (seconds) and the GPU memory usage (GB) during inference stage compared with Llama-3-8B without (w/o), with (w/) Faash-Atenttion22 (FA2) and Liger-8B. OOM denotes of memory.
PIQA
ARC-e
ARC-c
Hella.
Wino. acc 1
MMLU
Avg.
Avg.
Model
acc
acc f
acc_nome f
acc_norm
acc (5-shot) f
(no MMLU) 1
Liger-GLA
80.3
81.1
52.5
76.3
72.0
43.4
67.6
Liger-HGRN2
79.2
76.8
48.5
74.4
68.8
36.2
64.0
69.5
Liger-GSA
79.5
78.5
49.4
74.5
70.5
41.2
65.6
70.5
Table 10. Full Results Different Gated Linear Recurrent Model Variants with Liger. Liger he applied to the efficient linearization of various linear recurrent structures with gating mechanism and achieve high quality performance recovery.
14

---

Liger: Lincarizing Large Language Models to Gated Recurrent Structures
PIQA
ARC-e
ARC-c
Hella.
Wino.
MMLU
Avg.
Avg.
Model
acc
acc f
acc norm 1
acc norm
acc f
ace (5-shot)
(no MMLU)
Llama-3.2-IB
74.1
65.4
36.4
63.8
60.0
31.0
55.1
59.9
GLA-IB
69.9
55.2
27.6
48.9
53.9
25.9
46.9
LoI.CAT--Laaa-322-1B
74.1
63.7
36.4
51.2
58.2
23.1
51.1
56.7
Liger- GLA-LLama-3322
75.0
65.4
35.7
59.8
59.1
22.4
52.9
59.0
Llama-3.2-3B
76.4
74.7
46.0
73.6
69.9
56.2
66.1
68.1
GLA-3B
71.5
59.2
30.0
53.0
55.3
25.6
49.1
53.8
LoLCATT-LLama332--BB
76.7
72.0
42.3
51.9
66.9
23.6
55.6
62.0
Liger-GLA-LLama-3222
77.9
74.0
43.9
70.3
32.1
60.7
Llama-3-8B
79.4
80.1
53.2
79.2
72.9
65.3
71.7
73.0
LoLCATs-Llama-3--
80.1
80.4
53.5
63.4
42.8
65.5
70.0
Lige--GLALLlmma-3 (Ours)
80.3
81.1
52.5
76.3
72.0
43.4
67.6
72.4
Table 11. Full Results on Scalability Analysis of Linearized Llama-3 Architectures across Model Sizes (1B 8B). Performance comparisons between vanilla Llama-3, GLA, Lol.CATs, and our Liger-GI. variants on language modeling and reasoning tasks. Liger demonstrates consistent scaling laws, outperforming LoLCATs by +6.8 11.5% absolute average metrics while preserving 83 98% of base model capabilities with only 0.02B adaptation tokens.
Validation PPL.
PIQA
ARC-e
ARC-c
Hella.
Wino.
MMLU
Avg.
Avg.
Model
t
acc
acc
(5-shot) 1
t
(no MMLU)
Liger-GLA
2.96
80.3
81.1
52.5
76.3
72.0
43.4
67.6
72.4
Gate Proj-
3.16
79.1
75.9
49,6
71.8
67.3
39,2
63.8
68.8
Feat. Map.
46.3
24.2
33.7
50.4
40.2
43.5
Pure LA
3.00
79.9
79.8
52.0
75.3
70.6
38.8
66.1
71.5
w/o LoRA
3.23
78.7
75.6
47.4
74.0
64.8
29.5
61.7
68.1
w/o SWA
3.75
75.0
68.3
39.1
63.4
55.3
26.4
54.2
60.2
w/o GLA
3.01
80.5
52.4
72.4
37.0
66.2
72.0
Table 12. Full Results on Ablation Study. We linearize Llama-3-8B to Gated Linear Attention (GLA) to evaluate the key components of Liger. We report Validation perplexity (PPL.) cleaned alpaca dataset after Liger linearization and the average performance on language modeling and under-standing tasks.
15


Reviews:
[{"review_id": "bZ4iMPecVA", "reviewer": "Reviewer_xf9Q", "summary": "This paper studies the problem of how to convert pretrained standard attention-based LLMs into more efficient hybrid models, a research area that recently emerges along with significant progress on sub-quadratic model architectures. It proposes a methodology to convert current Mistral/Llama models to constant-memory hybrid models that combine sliding window attention and gated recurrent neural networks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer xf9Q,\n\nWe sincerely thank you for your careful comments and thorough understanding of our paper! Here we give point-by-point responses to your comments and questions.\n\n> `Q1`: The baseline that only uses sliding window attention is missing.\n\nThanks for your careful comment. We have supplemented the experiment with sliding window attention (SWA) only.\n\n||PiQA|ARC-e|ARC-c|Hella.|Wino.|MMLU|Avg.|Avg.(no MMLU)|\n|-|-|-|-|-|-|-|-|-|\n|GLA only|75.0|68,3|39.1|63.4|55.3|26.4|54.6|60.2|\n|SWA only|79.8|80.5|52.4|75.1|**72.4**|37.0|66.2|72.0|\n|Liger Attention|**80.3**|**81.1**|**52.5**|**76.3**|72.0|**43.4**|**67.6**|**72.4**|\n\nNotably, whether SWA only or GLA only is used, the performance will decrease, while Liger Attention achieves the best performance.\n\n> `Q2`: It's unclear whether gating is still necessary in this distillation setting\n\nWhile transformer-based LLMs suffer from computational inefficiency due to unbounded historical storage in the KV-cache, linear recurrent models face challenges in dynamically \"forgetting\" obsolete information without gating mechanism. Prior work in linear recurrent models [1-4] underscores the critical role of gating in enabling adaptive forgetting, suggesting its necessity even in linearization or distillation setting.\n\nHowever, existing linearization methods often omit gating or introduce randomly initialized gating modules that need extra training, which incur added training costs and suboptimal performance (e.g., Table 6 shows performance drops when learnable gates are added). In contrast, Liger incorporates gating and optimizes its construction without any parameters introduction and achieves better performance, which proves that gating is necessary for linearization.\n\nIt is worth noting that although most linearization methods are based on distillation setting and adopt multi-stage training with further fine-tuning. In fact, our proposed Liger **does not** adopt any distillation setting. It uses only **single stage of LoRA fine-tuning** to complete the adaptation of the architecture conversion with only **0.059%** of the parameters need to be fine-tuned on **0.02B** training tokens and restore **over 93%** of the LLM performance. The linearization cost significant reduce and it can work on various linear recurrent architectures with gating mechanism, which reflects the simplicity and versatility of Liger.\n\n> `Q3`: Hopefully more discussion on the parameterization of G\n\nThe parametric way for $G$ construction is introducing extra learnable parameters, which dynamically adjusts the flow of information through learnable weights and biases (e.g. MLP projection). It is data-adaptive but requires more computing resources and training data compared with the non-parametric way. In the scenario of LLM linearization, we seek to obtain a powerful linear LLM faster at a lower cost under resource constraints. \n\nOur ablation study (Table 6 in our manuscript) show the following results, where Gate Proj refers to construct the gate projection module in parameteric way by introducing extra parameters, which is implemented as the low-rank MLP used in GLA[1].\n\n||Valid PPL$\\downarrow$|Avg$\\uparrow$|Avg(no MMLU)$\\uparrow$|\n|-|-|-|-|\n|Liger-GLA|2.96|67.0| 71.7|\n|Parametric Gate Proj.|3.16|63.8|68.8|\n\nThe results reveal that Liger can achieve better performance by using a simple pooling operation in non-parameteric way to construct the gating module, which is consistent with the concept discussed above about parameterization of $G$.\n\n> `Q4`: Provide details of the pooling operation along the resulting shape of G_t\n\nThe form of $G_t$ depends on the specific gated recurrent model employed. Most gated linear models adopt a dynamic time-varying vector gate instead of a simple scalar. In the specific implementation, our pooling operation uses `torch.AdaptiveAvgPool1d` over the $D$ dimension, which can construct a specified gate form through average pooling or interpolation without introducing any learnable parameters, benefiting the linearization process. For example, the gate of GLA: $G_t \\in \\mathbb{R}^{D}$ is a vector, while the gate of GSA is $G_t \\in \\mathbb{R}^{M}$, where $M$ is the number of slots. We can adopt the simple pooling operation to transform the shape from the LLM weight matrix to construct the corresponding gate vector.\n\n___\n\n[1] Songlin Yang, et al. Gated Linear Attention Transformers with Hardware-Efficient Training. ICML, 2024. \\\n[2] Zhen Qin, et al. HGRN2: Gated Linear RNNs with State Expansion. COLM, 2024. \\\n[3] Yutao Sun, et al. Retentive Network: A Successor to Transformer for Large Language Models. arXiv, 2023. \\\n[4] Yuhong Chou, et al. MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map. NeurIPS, 2024.", "claims_and_evidence": "The major claim is that the proposed recipe for obtaining a constant-memory model out of existing pretrained models can recover 93% performance. This claim is clearly supported by the main results.\nThe efficiency claim of the resulting model is also well supported by benchmarking decoding efficiency in terms of GPU memory usage and latency.", "methods_and_evaluation_criteria": "The central problem towards converting a pretrained LLM to another architecture lies in the parameterization choice, and the proposed method aims to address this problem. The evaluation is based on some standard benchmarks for evaluating LLMs.", "theoretical_claims": "NO.", "experimental_designs_or_analyses": "While the main results clearly support the major claim of performance gain and efficiency benefit, I believe the experiments should be further improved to understand the core piece of the method. Specifically, it would be great if there were more experiments on the choice of target architecture (i.e., the constant memory model) and the choice of parameterization to address the following concerns:\n\n1. The baseline that only uses sliding window attention is missing, which makes it hard to infer the effect of the linear attention part.\n2. The premise of using gated linear attention is that gating is crucial for RNN. However, I believe it's unclear whether gating is still necessary in this distillation setting (i.e., converted from pretrained weights) and hybrid architecture (sliding window + linear attention).\n3. Hopefully more discussion on the parameterization of G (e.g., a parametric way as opposed to the current non-parametric way).", "relation_to_broader_scientific_literature": "This paper connects to the broad research effort on sub-quadratic/efficient model architectures. The results here could also shed light on the connection between softmax-attention and linear attention in terms of parameterization.", "essential_references_not_discussed": "NO.", "other_strengths_and_weaknesses": "No", "other_comments_or_suggestions": "Figure captions should be improved. E.g., table 2 is hard to read with many model variants.", "questions_for_authors": "It's unclear to me whether G_t is a scalar or vector. Can you provide details of the pooling operation along the resulting shape of G_t.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p3Ii11kNpG", "reviewer": "Reviewer_2WYJ", "summary": "This paper proposed a novel method, Liger, for transforming pretrained Transformer-based LLMs into gated linear recurrent models, repurposing key matrix weights to create gating mechanisms. \nIt utilizes Liger Attention, an intra-layer hybrid attention mechanism involving Gated Recurrent Modeling (GRM) and Sliding Window Attention (SWA). The resulting model retains 93% of the original Transformer model’s performance while requiring only 0.02% of the pretraining tokens.\nUsing lightweight LoRA finetuning, Liger successfully restores the performance of linearized models, achieving state-of-the-art efficiency and accuracy across multiple benchmarks on models ranging from 1B to 8B parameters.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 2WYJ,\n\nWe are truly grateful for the time you have taken to review our paper and your insightful comment. We address your questions in the following.\n\n> `Q1`: Discussion on intra-layer hybrid attention with similar designs\n\nWe carefully read these works for further comparison and discussion.\n\nHymba uses a intra-layer hybrid attention which combines softmax attention and SSM via independent parameterizations: $o_t=\\beta_1\\text{norm}(M_{attn}x_t)+\\beta_2\\text{norm}(M_{ssm}x_t)$, where $M_{attn}$ and $M_{ssm}$ employ separate projection matrices $W_Q/W_K/W_V$ vs. $W_A/W_B/W_C$. Liger Attention eliminates this duplication by: $o_t=\\alpha\\text{GRM}(q_t,k_t,v_t)+\\beta\\text{SWA}(q_t,k_t,v_t)$, which shares matrices between sliding window attention (SWA) and gated recurrent modules (GRM), avoiding the need for introducing extra learnable parameters. This parameter efficiency directly supports better linearization while retaining generality—GRM subsumes (but is not limited to) SSM. \n\nTitans' hybrid form also similarly uses independent parameters to construct linear and attention modules respectively, and is essentially an inter-layer hybrid rather than intra-layer, because the output of its memory module (linear module) will be used as the input of the attentnion module, and vice versa.\n\nWe would respectfully cite and supplement the discussion of related works.\n\n> `Q2`: Significant degradation in performance on MMLU\n\nWe have noticed that most linearization methods have significant degradation on MMLU. We hypothesize that the performance gap partially stems from MMLU's evaluation design, which assesses both knowledge recall in pretrained weights and context-based answer retrieval through its 5-shot multiple-choice format[4]. While this setup requires models to identify correct answer indices from contextual prompts, prior research demonstrates that linear recurrent architectures exhibit **inherent limitations** in such retrieval tasks[5].\n\nDespite this, we still try our best to alleviate this problem. We  linearize Liger from Llama-3-8B-instruct with stronger instruction following ability. Besides, we investigate whether we can improve MMLU performance from stronger LLM (e.g. Qwen2.5).\n\n||MMLU|\n|-|-|\n|Llama-3-8B|65.3|\n|Liger-GLA-8B|43.4|\n|Llama-3-8B-Instruct|65.7|\n|Liger-GLA-8B-Instruct|47.6|\n|Qwen2.5-7B|74.2|\n|Liger-GLA-Qwen2.5-7B|55.1|\n\nAlthough there is still a significant drop on MMLU, we find that Liger can benefit from the instruct model to improve 5-shot performance, and can also benefit from more powerful model (Qwen2.5) for such task (only 1.6 lower than Mamba2-Llama3-8B 50% Attn), further narrowing the gap with the original LLM.\n\n> `Q3`: Why does the gate need to be derived from the Key instead of other elements\n\nThe key matrix usually serves as memory indexing in transformer or linear model[1], which is similar to the gate that determines which part of the memory should be retrieved or forgotten. In this view, it is intuitive to choose Key for gate construction. MetaLA[2] also points out that the linear recurrent model needs to meet the \"least parameter approximation\" condition to achieve the optimal linear attention design. In this case, the key matrix is ​​redundant and can be used to construct the gate, which also provides motivation and theoretical support for gate derived from Key.\n\nWe have supplemented experimental comparisons of gate derived from Q and V:\n\n||Avg|Avg(no MMLU)|\n|-|-|-|\n|Liger-Q-pool|66.7|71.7|\n|Liger-K-pool|67.6|72.4|\n|Liger-V-pool|66.5|71.1|\n|Liger-Gate-init|63.8|68.8|\n\nWe can observe that the effect of constructing the gate module from any Q, K, V is ​​significantly better than randomly initialized gate, and the overall performance of gate derived from Key is ​​more outstanding, which is consistent with the conclusion in [2][3] that the parameters of the Key Matrix are more redundant and can act as gating mechanism.\n\n> `Q4`: How much do the LoRA weights change\n\nLoRA is only used to finetune QKV that only 0.059% of total weights need to be trained. Let $W$ be the sum of each weight parameter of LLM, we calculate the change in Liger's LoRA weights compared with LoLCATs:\n\n||LoLCATs|Liger|\n|-|-|-|\n|$\\|\\|\\Delta W\\|\\|_1$|3.68e+06|2.91e+06|\n|$\\frac{\\|\\|\\Delta W\\|\\|_1}{W}$|4.82%| 3.85%|\n|$\\|\\|\\Delta W\\|\\|_2$|3.93e+04|1.27e+03 |\n|$\\frac{\\|\\|\\Delta W\\|\\|_2}{W}$|179.29%|7.63%|\n\nWe observe that Liger has a smaller number of changing parameters than LoLCATs, which makes Liger less pretrained knowledge destruction, more efficient and effective.\n\n___\n\n[1] Gershman S. J., et al. Key-value memory in the brain. arXiv, 2025 \\\n[2] Chou Y., et al. MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map. NeurIPS, 2024 \\\n[3] Qin Z., et al. HGRN2: Gated Linear RNNs with State Expansion. COLM, 2024 \\\n[4] Hendrycks D., et al. Measuring Massive Multitask Language Understanding. arXiv, 2020 \\\n[5] Waleffe R., et al. An Empirical Study of Mamba-based Language Models. arXiv, 2024", "claims_and_evidence": "- This work asserts that the proposed method can adapt a pretrained transformer-based LLM into a gated recurrent structure with high token efficiency. This assertion is well supported by the method and empirical results.\n- This work asserts that the proposed Liger Attention is an intra-layer hybrid attention mechanism that combines sliding window softmax attention with linear recurrent modeling. This assertion is well supported by the method and empirical results.", "methods_and_evaluation_criteria": "This work evaluates the proposed method regarding task accuracy and latency.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "One of the contributions is the intra-layer hybrid attention, referred to as Liger Attention. A similar design is also found in [1,2].\n\n[1] Dong, X., Fu, Y., Diao, S., Byeon, W., Chen, Z., Mahabaleshwarkar, A.S., Liu, S.Y., Van Keirsbilck, M., Chen, M.H., Suhara, Y. and Lin, Y., 2024. Hymba: A hybrid-head architecture for small language models. arXiv preprint arXiv:2411.13676.\n[2] Behrouz, A., Zhong, P. and Mirrokni, V., 2024. Titans: Learning to memorize at test time. arXiv preprint arXiv:2501.00663.", "other_strengths_and_weaknesses": "see Questions", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- It appears that all methods experience significant degradation in performance on the MMLU task, including the \"Mamba2-Llama-3-8B 50% Attn.\" model. It would be beneficial to conduct further analysis to understand why \"Liger-GLA-Llama-3-8B\" does not perform well on MMLU as well.\n\n- Why does the gate need to be derived from the Key instead of other elements?\n\n- How much do the LoRA weights change after fine-tuning?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uRnz8KNsLE", "reviewer": "Reviewer_5t3M", "summary": "Transformer Language models based on linear recurrent structures  (Katharopoulos et al., 2020; Yang et al., 2023; Qin et al., 2024b) are substantially more efficient than regular transformers due to their linear dependency on sequence length and constant memory requirements. Prior work - SUPRA (Mercat et al., 2024), MambaInLlama (Wang et al., 2024) and LoL-CATs (Zhang et al., 2024a) - has shown that is is possible to initialize these models from existing transformer based LLMs and linearize them. However, this process requires costly fine-tuning steps and results in loss of performance. One solution to improve performance is via introducing a gating operation in the recurrence. This paper presents an approach, called Liger, to speedup the model transfer and improve performance by repurposing existing key vectors for the recurrent gating operation. Lora fine-tuning is employed to fine-tune these parameters. The paper also presents an approach to interpolate gated-recurrent attention with a sliding-window softmax attention.  The liger attention block is used along with the original transformer attention block in the model. Results are presented on Mistral-7B and Llama3-8B llms. On Mistral-7B, the approach outperforms SUPRA trained on 500x training tokens and Lolcat which is trained on a similar number of training tokens. On Llama3-8b, the approach outperforms Lolcat trained on a similar number of training tokens and Supra trained on 100x training tokens and some versions of Mamba trained on 100x training tokens. However it underperforms a variant of Mamba trained on 100x training tokens with 50% attention. It achieves a linear time decoding speed when examined on sequences ranging from  1k to 32k. Ablations are shown on each of the critical components such as gates generated from random projections. Lora, feature map, SWA.\n\n## update after rebuttal\nThe authors have addressed the clarity issues that I raised in my initial review. Hence increasing the scores after the rebuttal.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 5t3M,\n\nThanks for your thorough reviews and we appreciate the attention to detail and would fix the writing errors in the revision.  We give point-by-point responses to your comments based on your questions.\n\n> `Q1`: Is pooling performed across the time dimension?\n\nNo, the pooling operation is performed accross hidden dimension. We perform linearization by pooling the model's $K \\in \\mathbb{R}^{T \\times D} $ into $G \\in \\mathbb{R}^{T \\times D_G}$, where $D_G$ depends on the gating module of the specific model used, such as $G \\in \\mathbb{R}^{T \\times S}$ in GSA, where $S$ is the number of memory slots. This form can achieve the effect of dynamic gating without introducing any learnable parameters during the linearization process. It is simple and has been demonstrated to be particularly effective in experiments.\n\n> `Q2`: 'GLA denotes Gated Recurrent Modeling' -> should GLA be GRM instead?\n\nYes, thanks for your careful review. Here GLA should be GRM instead. GRM represents a general form of various gated recurrent modeling methods and their variants, and GLA is an instance of GRM.\n\n> `Q3`: What does Liger-GLA mean? Does this not include Sliding window attention?\n\nLiger-GLA means using Liger proposed in our paper for linearization of LLM to gated recurrent structures, which adopts Liger Attention, a intra-layer hybrid attention including Sliding Window Attention and using GLA as the instance of GRM.\n\n> `Q4`: What do learnable feature map modules mean?\n\nThe original softmax attention is calculated as $O=\\text{softmax}(QK)V$, and its complexity is $\\mathbb{O}(T^2)$ quadratic over the length of the sequence. Linear attention is achieved by removing the softmax operation and replacing it with **feature map modules** $\\phi$ acting on $Q$ and $K$ and reducing the computational complexity to $\\mathbb{O}(T)$ through the right multiplication technique. The specific formula is $O=\\phi(Q)\\phi(K)V$, so that the product of $\\phi(Q)\\phi(K)$ can approximate the output of softmax attention. For example, [1] setting feature map to $\\phi(x)=\\text{elu}(x)+1$ introduces non-negativity similar to softmax attentioin to achieve better performance.\n\n**Learnable feature map modules** introduce learnable parameter for $\\phi$ optimization, which makes the model adaptive to data and improves expressiveness to better approximate the effect of softmax attention. For example, the feature map module design in [2] adopts: $\\phi(x)=\\exp(\\mathbf{W}^\\top x+b)$.\n\nIn our proposed Liger, we set $\\phi(x)$ which simply uses the normalization function, which can ensure that the distribution of the product of $\\phi(Q)\\phi(K)$ is still non-negative and approximates the original $\\text{softmax}(QK)$ output distribution, and there is no need to introduce any learnable parameters, which is very helpful for LLM linearization and demonstrated effective in our experiments.\n\n___\n\n[1] Angelos Katharopoulos, et al. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. ICML, 2020. \\\n[2] Michael Zhang, et al. The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry. ICLR, 2024.", "claims_and_evidence": "The claims are supported by evidence.", "methods_and_evaluation_criteria": "Yes they make sense.", "theoretical_claims": "No theoretical claims are made in the paper.", "experimental_designs_or_analyses": "Yes, the designs are sound.", "relation_to_broader_scientific_literature": "Language models based on linear recurrent structures  (Katharopoulos et al., 2020; Yang et al., 2023; Qin et al., 2024b) are substantially more efficient than transformers due to their linear dependency on sequence length and constant memory requirements. Prior work - SUPRA (Mercat et al., 2024), MambaInLlama (Wang et al., 2024) and LoL-CATs (Zhang et al., 2024a) - has shown that is is possible to reuse existing transformer based LLMs by linearizing them. However, this process requires costly fine-tuning steps and results in loss of performance. One solution to improve performance is via introducing a gating operation in the recurrence. This paper presents an approach, called Liger, to speedup the model transfer and improve performance by repurposing existing key vectors for the recurrent gating operation. Lora fine-tuning is employed to fine-tune these parameters. The paper also presents an approach to interpolate gated-recurrent attention with a sliding-window softmax attention.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n* Presents an approach to speed up the conversion of a transformer based LLM into linear recurrent structures and improve performance using gating.\n* Reports results on Mistral-7B and Lllama3-8B showing that the approach yields improvements in performance compared to prior linearization approaches while training on similar or substantially fewer tokens.\n* Presents an ablation study showing the importance of each of the components added.\n\nWeaknesses:\n* Some parts of the paper are unclear. See questions below.", "other_comments_or_suggestions": "* Figure 4: It would be useful to show results from GLA and LolCat as well.\n* L41: resurrent -> recurrent\n* L84: rely -> reliant", "questions_for_authors": "* L208: Eqn 6: Is pooling performed across the time dimension? Please clarify.\n* L273: 'GLA denotes Gated Recurrent Modeling' -> should GLA be GRM instead?\n* Table 2/4: What does Liger-GLA mean? Does this not include Sliding window attention?\n* L417: What do learnable feature map modules mean? It would be good to explain briefly.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}]

Author rebuttal:

Meta-review: This paper presents an approach for converting pretrained softmax attention Transformers into gated linear attention Transformers, which can inherit the benefits of linear attention without having to pretrain everything from scratch. The paper outlines a series of well-motivated and empirically-tested recipe for doing this. The initial reviews raised some questions with regard to other baselines (e.g., non-gated linear attention) as well as clarity. These concerns were largely addressed in the rebuttal.

This is a well-executed and solid paper that should be of interest to the growing community of researchers working in this space, and thus I recommend that this paper be accepted.
Decision: Accept (poster)