RWKV-7 "Goose" with Expressive Dynamic State Evolution
Authors: Bo Peng
Ruichong Zhang
Daniel Goldstein
Eric Alcaide
Xingjian Du
Haowen Hou
Jiaju Lin
Jiaxing Liu
Janna Lu
William Merrill
Guangyu Song
Kaifeng Tan
Saiteja Utpala
Nathan Wilce
Johan S. Wind
Tianyi Wu
Daniel Wuttke
Christian Zhou-Zheng
Venue: COLM 2025 (2025)
Keywords: Goose
LLM
RWKV
RWKV-7
RWKV7
Linear
Linear Attention
SSM
subquadratic
TL;DR: RWKV-7 is a new sequence modeling architecture with constant memory usage and inference time per token, SoTA performance on multilingual tasks, and near SoTA English LLM performance at 3B scale, with dramatically less training than top 3B models.

Abstract: We present RWKV-7 "Goose", a new sequence modeling architecture with constant memory usage and constant inference time per token. Despite being trained on dramatically fewer tokens than other top models, our 2.9 billion parameter language model achieves a new 3B SoTA on multilingual tasks and matches the current 3B SoTA on English language downstream performance. RWKV-7 introduces a newly generalized formulation of the delta rule with vector-valued gating and in-context learning rates, as well as a relaxed value replacement rule. We show that RWKV-7 can perform state tracking and recognize all regular languages, while retaining parallelizability of training. This exceeds the capabilities of Transformers under standard complexity conjectures, which are limited to $\mathsf{TC}^0$. To demonstrate RWKV-7's language modeling capability, we also present an extended open source 3.1 trillion token multilingual corpus, and train four RWKV-7 models ranging from 0.19 billion to 2.9 billion parameters on this dataset. To foster openness, reproduction, and adoption, we release our models and dataset component listing at https://huggingface.co/RWKV, and our training and inference code at https://github.com/RWKV/RWKV-LM; all under the Apache 2.0 License.

Published as a conference paper at COLM 2025
RWKV-7"Gooser with Expressive Dynamic State Evolution
Bo Peng
Ruichong Zhang Tsinghua University zhangre24 (etpnnrcc2@mmallle 4mmails ttsinghua edu. cn
Daniel Goldstein EleutherAl, Recursal Al dan@ ran@recursal.a
EleutherAI,
RWKV Project (Linux
Foundation Al & Data) pengbo8 63@ gmail 1.. com
Eric Alcaide
Xingjian Du University of Rochester
Haowen Hou Guangdong Laboratory of Al and Digital Economy (SZ)
Jiaju Lin Pennsylvania State University
Jiaxing Liu Zhejiang University
Janna Lu Recursal A1, George Mason University
EleutherAI USI, IDSIA
William Merrill
Guangyu Song EleutherAl Tano Labs
Kaifeng Tan Shenzhen University
Saiteja Utpala EleutherAl
Nathan Wilce EleutherAI, Recursal
New York University
Johan S. Wind University of Oslo
Tianyi Wu Beijing Normal University
Daniel Wuttke EleutherAl, Denigma
Christian Zhou -hhnng EleutherAl
Abstract
We present RWKV-7 "Goose", a new sequence modeling architecture with con- stant memory usage and constant inference time per token. Despite being trained on dramatically fewer tokens than other top models, our 2.9 billion parameter language model achieves new 3B SoTA on multilingual tasks and matches the current 3B SoTA English language downstream performance. RWKV-7 introduces a newly generalized formulation of the delta rule with vector- valued gating and in- in-context learning rates, well relaxed value replace- ment rule. We show that RWKV-7 can perform state tracking and recognize all regular languages, while retaining parallelizability of training. This exceeds the capabilities of Transformers under standard complexity conjectures, which are limited TC To demonstrate RWKV-7's language modeling capability, also present an extended open source 3.1 trillion token multilingual corpus, and train four RWKV-7 models ranging from 0.19 billion 2.9 billion parameters this dataset. To foster openness, reproduction, and adoption, we release our models' and dataset component listing Hugging Face, and our training and inference code3 on GitHub; all under the Apache 2.0 License.
Introduction
Autoregressive Transformers (Vaswani al., 2017) have recently dominated sequence modeling tasks, enjoying excellent in-context processing and highly parallelizable training due to their use of softmax attention. However, softmax attention incurs quadratic computational complexity and memory usage with respect to sequence length due to its linearly expanding key-value cache. For
"Equal first authorship. Others listed alphabetically.
Model weights https: https:://uuginggfceeee.m .ot.eeevec../VVvv /RNVV
Dataset components listed at https : //uggi ng face. co/ RWKV
Source code htttps://githhee http:/ttwwbcoo.wc///./aaoo wwwp///w////ww--......

---

Published as a conference paper at COLM 2025
Name
State Evolution
Scalars
FD
DD
RWKV-4
8t 81-1 + t o Vti
X
K
X
5 = o 8 te
RetNet
S, = WS,-1 +
W
X
X
RWKV-5
5, = = ()-idd))(())
X
X
Mamba
s, - eexp(-(w) 1) 1 exp((A) (w, o T
V
X
RWKV-6 & GLA
s, - diag( w1) of kc
HGRN-2
5 S 10--dia(( w1) + +1; (1- w;) Kr
X
X
Mamba-2
S, = W,S; 1
Wt
T a
S, S a, a,Vi(S Kz, U1)
Longhorn
S, S @ (I- - a k. (a;x,) 2 Ks
Gated DeltaNet
S, W W,
a,k; kj) + 0g ! kt M, a;)
Wn. at
Titans
a,)
Wn. at
X
S, = W; W
k, 11)
Generalized A Rule
S, S, (diag( + b,) b+ v Kr
RWKV-7 (ours)
S, S, (diag( -) (a, @ kx)) +
Table Recent RNN architectures used for language modeling.
IS (Large Statc): matrix-valued states, or state size at least times larger than the model dimension. FD (Flexible Decay): the dimension of the decay term w or W, is not smaller than the model dimension.
DD (Dynamic Dependence): the decay term is a function over the input
GE (Generalized Higenvalue): evolution matrix admits eigenvalues outside of the interval [0, 1J.
Shown with mini batch size for simplicity.
short sequences, much this cost can be covered by modern GPU parallelism techniques (Dao, 2023), but Transformer inference becomes increasingly costly sequence lengths grow.
This limitation has inspired significant research into the design of recurrent neural network (RNN) architectures with compressive states that afford linear computational complexity and constant memory usage, while still allowing highly parallel training. Two of the most commonly proposed alternatives that satisfy these requirements et al., 2020b; Sun et al., 2023; Peng et al., 2024b; Yang et al., 2023a) and State Space Models (Gu sophisticated, with many recent proposals
linear attention variant models (Katharopoulos
Dao, 2023). These architectures have grown incorporating some form of the delta rule, as embodied by parallelized DeltaNet (Schlag et al., 2021; Yang al., 2024c). Such models have achieved impressive downstream performance results: since RWKV-4 (Peng et al., 2023), RNN models have shown increasing potential to rival Transformers when given equivalent model size and training compute, while dramatically reducing inference costs.
We present new architecture, RWKV-7 "Goose", which generalizes the delta rule for use in sequence modeling. First, we add a vector-valued state gating mechanism, enhancing expressivity and providing implicit positional encoding. Second, we expand the in- in-context learning rate (see Section 2) from scalar to become vector-valued, allowing the model to selectively replace state data on a channel-wise basis. Third, we decouple the keys at which the delta rule removes from and adds the state. Finally, place these innovations within modified RWKV-6 architecture, inheriting important features such as token-shiff shift, bonus, and a ReLU2 feedforward network. We also introduce expanded 3.1 trillion token RWKV World v3 corpus designed for enhanced English, code, and multilingual task performance. We use this architecture and corpus to train state-of-te of-the-art open- source language models, upgraded from preexisting RWKV-5/RWKV-6 checkpoints.
Our main contributions are as follows:
The RWKV-7 "Goose" architecture, which dramatically improves downstream bench- mark performance over RWKV-6 and demonstrates state-of-the- the-art multilingual perfor- mance at 3B scale and near SoTA English language performance, despite being trained many fewer tokens than the top models in its class.
2

---

Published as a conference paper at COLM 2025
The RWKV World v3 public dataset, comprised 3.1 trillion tokens of publicly available multilingual data.
Public release of four pretrained RWKV-7 World v3 language models, ranging from 0.19
to 2.9 billion parameters trained on 1.6 to 5.6 trillion tokens.
Public release of three pretrained RWKV-7 Pile language models, using the GPT-NeoX tokenizer (Black et al., 2022), ranging from 0.17 to 1.47 billion parameters, useful for comparative study with other architectures.
Proofs that the generalized delta rule employed in RWKV-7 can solve problems outside of TC under the widely held complexity conjecture that TC NC². This includes solving an Sş state tracking problem known to be in NC¹ using only single layer, and recognizing all regular languages using only constant number of layers.
A method for upgrading the RWKV architecture without pretraining from scratch, producing increasingly competitive trained models at reduced computational expense.
Background
Linear attention's major advantage over softmax attention is that it can be formulated as a RNN with constant running time per token and constant memory usage (Katharopoulos al., 2020a), while softmax attention takes O(N) time per token and O(N) memory with regard to sequence length. Despite this dramatic efficiency improvement, linear attention has its own significant drawbacks including fixed-size state that is added to but never truly erased (Schlag et al., 2021; Han et al., 2024; Fan et al., 2025; Yang et al., 2024b).
Delta Rule. DeltaNet (Schlag et al., 2021) sidesteps the problem of additive state by partially replacing the value stored the current key with the same amount of new value, allowing the model to both take away old memories and add new ones on per-key basis. It reformulates the state update as an explicit online learning problem where the goal is retrieve the correct value as output for given key as input. DeltaNer was the first to apply the foundational Error Correcting Delta Rule (Widrow al., 1960) key-value compressive states, akin to those stored in the RNN formulation of linear attention. This update rule is equivalent to single step of stochastic gradient descent, training the state S, at test time to output the desired values v, for the keys k, as inputs using loss L = 2 [1(S,kk - vz)l and gradient dz
as SK T k-v v-T k, leading to a recurrent update formula of S, 1 - akF k,) av; k,, where a is scalar learning rate. The ideas behind this internal state update can be traced back fast weights (Schmidhuber, 1992).
There has been significant recent interest in improvements to DeltaNet, in order to bring its efficiency and downstream performance in line with Transformers while still capturing the speed and memory benefits of Linear Attention. Parallelizing DeltaNet (Yang et al., 2024c) showed that DeltaNet used diagonal plus low-rank (DPLR) state evolution like S4 (Gu al., 2022), and could be parallelized across the time dimension, creating path to efficiently train such models. Our work further extends that parallelization to cover the generalized delta rule formulation introduced herein, as well as the specific formula of RWKV-7
Concurrent Work. Concurrent work with our own has focused architectural improvements beyond DeltaNet while still using the delta rule or variations thereof. Longhorn (Liu et al., 2024) employs an update rule that approximates closed-form solution globally optimal update objective, applied on otherwise unchanged Mamba architecture. Gated Delta Networks (Yang et al., 2024a) applies gating W, to the DeltaNet state, essentially multiplying the transition matrix by data- deeendentt scalar per head. This combines the DeltaNet update rule with the scalar decay found in some modern RNNs like RetNet and Mamba-2. TTT (Test-Tine Time Training) (Sun et al., 2024) and Titans (Behrouz al., 2024) also both apply scalar decay, but eschew per-step gradient descent update rules in favor of a batched multi- malti-timestep approach. Titans also adds momentum the otherwise classical SGD update applied to the state.
Grazzi et al. (2024) demonstrated the potential for increased expressiveness that comes from allowing the state transition matrix to contain negative eigenvalues. We show result signifi-
3

---

Published as a conference paper at COLM 2025
cantly beyond this, proving that RWKV-7 and generalized delta rule can recognize all regular languages using only a small constant number layers.*
3 Architecture
Head
Time Mix
LayerNorm
Readout
ReLU MLP
WKV7 Kernel
LayerNorm
x
M
Weight Prepare
Time Mix
LayerNorm
Token Shift
LayerNorm
Embedding
Figure 1: RWKV-7 overall architecture. See Appendix F for details.
+
wky,
wky,-1
diag(w;)
KG
(a,Ok)
Figure 2: A simple illustration of the update mechanism of single head RWKV-7's state. Note that the actual state size is 64 64 per head, not 4 4.
Unlike the other work described above, RWKV-7 generalizes the delta update rule into an extended formula S, S,-1 (diag( (,)) + 2 b,) of k, to increase expressivity (see Table 1). This is still diagonal plus rank one update rule, which admits efficient forms of parallelization (Yang et al., 2024c). Here, W; more expressive data- deitt-ddpeedeett vector- vector-valued decay, unlike the scalar decays featured in the other works previously described. Our use of 2, and b, in this extended formula permits flexible approach to state update, while retaining the important reduction in non-SRAM memory bandwidth usage that comes from using small deeendenn vectors instead of large matrices. One example of this flexibility is our of different removal and replacement keys.
RWKV-7's extended delta rule replaces data-dependent vector-valued amounts of the state, allow- ing each key channel in the state vary independently. We parameterize 2, R and b, k o a, where has elements in (0, 1). This keeps the update stable (see Appendix C), while maintaining increased expressivity. We show ablations for these improvements Appendix M.2.
RWKV-7 has a non- diagonal and dependee transition matrix, allowing it to represent more complex functions than its predecessors. (Merrill al., 2024) fact, we demonstrate that RWKV-7 possesses expressive power surpassing that of TC under standard complexity conjectures and can recognize all regular languages with constant number of layers. One new component of this power, not present in the original delta rule, is the ability to represent the "copy" state transition (Lemma 3). This is key element in our proof. As a result, we have the following theorems:
Theorem 1. RWKV-7 can solve a problem which is NC² -complete under AC reductions.
Theorem 2. For any regular language, there exists a 4- layer RWKV-7 model that recognizes it.
For now, we choose to allow only part of the range of possible negative eigenvalues in our pretrained large language models due experimentally observed training instabilities.
4

---

Published as a conference paper at COLM 2025
See Appendix D for proofs and details.
We replace the main RWKV-6 (Peng el., 2024b) diagonal transition matrix with our extended delta rule and make several other changes to the RWKV architecture, observing significant modeling improvements. These include updates to the channel mixing module and the token shift module. We remove the data dependency of token-shift and the receptance gating of channel mixing, both of which contribute to faster training and inference. We increase the use of low-rank projections generate more of our intermediate calculations, striking balance between the total number of model parameters, training and inference speed, and downstream performance.
4 Method
In this section, matrices, and vectors without subscript f are trainable parameters. The first subscript denotes sequence position and second subscript denotes layer index, where necessary. We the con- vention that all vectors are row vectors unless explicitly transposed, so all matrices operate on the right side, therefore a an outer product and ab" an inner one. We use the square subscript to denote placeholder for variable names and use the I1 sign for cumulative matrix multiplication. See Appendix G for a pseudocode implementation these formulas.
D to denote the model dimension. Bold capital letters represent trainable
4.1 Time Mixing
Weight Preparation Along the lines of (Peng al., 2024b), we introduce the following notation templates for common operators the model, using the square subscript to denote a variable:
lerp(a, b, x) = a (b @ x, loramlpn(f..) xx, bias) = f(xA_) (Ap if bias else 0),
(1)
(2)
Unless explicitly stated, all vectors appearing in this section are dimension D.
We extend the of low-rann rank MLP 2-layer MLP with small hidden dimension compared input and output), abbreviated as loramlp, to implement data dependency using minimal parameters.
The replacement key k, value v, decay w, removal key x, in-context learning rate a, receptance and rwkv gate parameters are computed as follows (outputs annotated with
x 1e)p(x;)xx11. (r v, gl a, sigmoid(loramlp, (Identiy, x², bias-True)),
token shifted inputs
(3)
b-in-context learning rate
(4)
k, x² We.
key precursor
(5)
K, &
removal key
(6)
E, = k o ae)
replacement key
(7)
V, (lapidloooommmlllllnittttt bias=True)),
value residual gate
(8)
1, x <WWr.
value precursor
(9)
layer 1
04 lerp(v', 1/ vel. layer 2
value
(10)
d, loramlp, ( (tanh, x bias=True),
decay precursor
(11)
W. exp(- e sigmoid(d.)),
-decay (12)
r, x wr.
receptance
(13)
gr foramml.(ssgmmii.... x bias=False)
prwky gate
(14)
& is learned parameter representing the removal key multiplier, which transforms the original key into a version to be removed from the state. In practice, & lies in a range of approximately (-5.3,9.4).
is a learned parameter representing the replacement rate booster, which adjusts the amount added back to the state after the transition matrix is applied.
5

---

Published as a conference paper at COLM 2025
Unlike r, k and which the main carriers of information, g,d,v and a act like gates which control the amount of information allowed to pass.
For comprehensive statistics of &, and biases of d, observed in the released RWKV-7 model, including extremum values, mean measurements, and distribution trends, see Appendix O.
For the computation of x,", we removed data dependency of linear interpolation from RWKV-6 to improve training speed.
We adapted the idea of Value Residual Learning Zhou al. (2024) for the computation of vú, which has shown to improve the final language modeling loss. V, represents the value residual mix, which interpolates between the layer zero and current layer value precursors: 1r,0 and Pt
We also updated the formula for computation of w,, restricting all entries in (exp(-e-U.5), 1) in favor a smaller condition number for diag(w;)) which maintains better training stability, and was beneficial to accuracy of the backward pass.
The k the formula can be regarded as a "normalized key", design to ensure that the state of wkv contains columns of O(1) size. Normally, expect k k, @ (1 - wr), as employed in RWKV-6c (see Appendix F), so that wkv, rows are linear interpolations between wkv,-1 and v/ kr controlled by Wt- However, further enhance expressivity, we decide decouple W, and a; We further decouple a, from the amount actually added to the state, allowing the replacement rate booster interpolate the amount added between the normal in-context learning rate and 1.0. Importantly, all of these modifications operate on a per-channel basis. The numerical range of RWKV-7's wkv entries are generally stable as in RWKV-6c, unlike RWKV-6, where entries of the states can accumulate to thousands (see Appendix N for state visualization).
The Weighted Key Value State Evolution After weight preparation, we reshape (r, W, k. V,K, a),, splitting them to h heads, with each head sized D/h. We always assume that h is a factor of D and heads are equally split. All operations in this section are shown per-head. Before mixing in the time dimension, is normalized per head:
(15) The wkv (Weighted Key Value) is multi-headed matrix-valued state of fast weights that un- dergoes dynamic evolution. The evolution of wkv is crucial for encoding context information by learning at test time to map keys to values. We start by defining the WKV time mixing as the recurrence relation
Ł k ||
wkvo = 0,
(16)
wkv; w wkv;-1 (ddiag(w))) - o
(17)
Compared to RWKV-5 and RWKV-6, the wkv in this paper is transposed to ensure consistency with RWKV-7's code. The wkv, attention calculation can alternatively be written parallel manner:
wkv; Σ
N (diagtw)) (a E (DDIM)((D
(18)
j=/+1
The recurrent transition design has parallels with Schlag et al. (2021), but crucially the transition matrix
G = diag(w;) R (a, o I- at o diag(w,)) = (I-2R(RR R,, diag(w,)
(19)
w,
is no longer a Householder matrix but a scaled approximation it, as k, * =,, k, This mimics Householder matrix but with expanded dynamics, while still having all eigenvalues in stable range of [- and allows the network decay information contrasts with the case of Householder-like matrix with learning rate (I - av v), a E [0, 1], as used in Schlag et al. (2021); Yang et al. (2024c) where all eigenvalues are one except for the last one corresponding to 1- Given these properties, we refer to W, as "in-context weight decay" and to a, as non-ootext learning rate" (ICLR). The RWKV-7 transition matrix, therefore, allows for both dynamic state evolution and approximation to forget gate at the same time. See Appendix C for the details on the eigenvalue of the transition matrix, and when the transition matrix is guaranteed to be stable.
subspaces if necessary.
6

---

Published as a conference paper at COLM 2025
WKV Bonus and Output All operations in this section shown per-head unless otherwise specified. Receptance, which acts like the query found in transformers, is applied to the WKV state, and the result is normalized. An added bonus, the amount of which is weighted by p. allows the model to place extra attention on the current shifted inpur token without requiring it to store that token in the state.
(rr (r. (p)))))))
bonus
(20)
LayerNorm(r, wkk))
aattention result
(21)
Finally, the heads recombined via reshaping that p1 E ER., gated, and transformed into the output as follows:
0r o ,
(22)
4.2 MLP
The MLP module of RWKV-7 is no longer identical the Channel Mixing module of previous RWKV-4,5,6 architectures (Peng et al., 2024b). We remove the gating matrix W,, making it a two-layer MLP. In compensation for the removed gating parameters satisfy the equi parameerr condition, we set the hidden dimension to be 4 times the size of model dimension.
k lerp(e 1 E R
(23)
o, ReLU(K))
(24)
RWKV World v3 Dataset
We train our models on the new RWKV World Dataset, new multilingual 3.119 trillion token dataset drawn from a wide variety of publicly available data sources. This dataset aims to help close the gap with the amount of data used to train modern LLMs, which may consume many as 15 18 trillion tokens (Qwen et al., 2025; Grattafiori et al., 2024). We select the data to approximate the distribution of our previous World datasets, including English, multilingual, and code, while slightly enhancing Chinese novels. We describe the composition of our dataset in Appendix B.
6 Pretrained Models
We have pretrained and publicly released seven Apache 2.0 licensed RWKV-7 models:
Trained on Pile: RWKV7-Pile of sizes 0.1B, 0.4B, and 1.4B 2. Trained on RWKV World V3: RWKV7-World-3 of sizes 0.1 0.4B, 1.5B, and 2.9B
Due to compute budget constraints, we used a new method to convert and train these models from upgraded pre-existing RWKV-5 and 6 checkpoints, See Appendix E for details.
7 Experiments
Speed and Memory Usage We test training speed and memory usage of RWKV-7, RWKV-6, and Flash Attention (Shah et al., 2024) kernels on H100 SXM. Due linear scaling, RWKV-7 surpasses Flash Attention's speed around 4096 sequence length. The new RWKV-7 kernel is also about 3x faster than the official RWKV-6 kernel. See Figure and Appendix for details.
LM Evaluation Harness Benchmarks We evaluated RWKV-7 alongside several new open models which are state-of- the-art state-of-the-art their parameter count ranges on series of common English and multilingual benchmarks using LM Evaluation Harness v0.4.8 (Gao et al., 2023) in fp32, shown in Figures 4 and 7 and Tables and 3.
The RWKV-7-Wone models establish
multilingual state-of- tee-the-rr t for their size, expand- ing upon wwwkv--World models already strong capabilities on multilingual benchmarks and significantly outperforming SmolL (Allal et al., 2025), 1lama-3.2 (Grattafiori et al., 2024), and
7

---

Published as a conference paper at COLM 2025
120
RWKKV77
100
RWWKV-6
v3
Time (mS)
60
20
500
1k
2k
6x
2k
4K
6k
10k
12k
14k
16k
Sequence Length
Sequence Length
Figure Kernel forward-backward time sequence length (H100)
65
80
Qwen2.5 7B
RWKV7-World3 2.9B
60
RWKV7-World3 2.9B
70
1.5B
Avorage Accuracy (%6
1.50
3B
1.58
1.78
55
1.58
Mamba2-Pile 2.7B
0.48
1./B
60
RWKV7-Pile 1.47B
.3
360M
0.58
50
@wen2.5 0.58
780M
0:40
0.19B
360M
370M
Sm 135M
50
45
SmolLM2 135M
0.198
130M
40
40
1E+20
1E+21
1E+22
1E+23
1F+20
1F+21
1F+22
1E+23
Training FLOPs (log scale)
Training FLOPs (log scale)
Figure 4: FLOPS vs. average benchmark accuracy for multilingual (left) and English (right). Multi- lingual benchmarks omit Pile models due to lack of multilingual training data in the Pile.
Qwen-2.5 (Qwen et al., 2025). find that RWKV-7 generally able match the English perfor- mance of Qwen2.5 despite being trained on less than one third as many training tokens. In Figure 4 we plot FLOPs used for training versus average accuracy across the same sets of common bench- marks, demonstrating very dramatic multilingual Pareto improvement versus the transformer models. We theorize that the difference would be even greater were we less compute constrained and able to train from scratch, instead of from pretrained checkpoints of earlier RWKV versions.
Mechanistic Architecture Design We evaluate RWKV-7 the Mechanistic Architecture Design (MAD) benchmark (Poli et al., 2024), suite of synthetic token manipulation tasks designed to probe architectural capabilities in sequence modeling, as shown in Table
RWKV-7 achieves the highest average score across all six tasks, outperforming previous architec- tures. demonstrates perfect accuracy on In- Context and Noisy Recall tasks, matching DeltaNet while setting a new ofatee-f-the--ee art for Fuzzy Recall. RWKV-7 also shows strong performance in memorization and selective copying, suggesting effective combination of attention- and recurrent model strengths.
Recent Internet Data Evaluation Despite careful data cleaning, benchmark data leakage remains a challenge, potentially compromising the validity these evaluations. To complement traditional benchmarks, we evaluated RWKV-7 and other leading open-soure models using temporally novel internet data, from after the models' training periods, removing data leakage concerns. See Table 5 and Appendix J.3 for full discussion and results.
8

---

Published as a conference paper at COLM 2025
Model
Tokens
Imb.o
hella
piqa
arcE
areC
glue
WG
sciq
mmlu
avg
(Name)
(T)
acc|
acc_n|
acc|
acc|
acc|
"acc1
acc]
acc]
acc|
acc1
RWKVS-World1-O.
0.6
38.4
31.9
61.4
44.2
19.9
45.5
52.9
76.3
23.1
43.7
SmolLM2-135M
2.0
42.9
43.1
68.4
64.4
28.1
49.0
53.0
84.0
25.8
51.0
RWKV7-World..8
1.6
48.1
42.1
67.3
39.3
25.5
48.1
52.7
86.3
25.4
50.5
RWKV5-World2-0.AM
1.1
54.0
40.9
66.5
54.0
24.0
50.0
53.2
86,9
23.8
50.4
SmoLLM2-360M
4,0
53.8
56.4
72.1
70.4
36.5
50.7
59.0
91.2
26.3
57.4
Qwen2.5-0.58
18.0
52.5
52.1
70.2
64.6
29.5
54.7
56.4
57.9
RMKV7-World..9-044B
3.1
58.6
56.8
72.9
68.7
31.9
49.4
89.7
26.1
57.1
RWKV6- Wor1d2.1-1.66
67.4
61.1
74.4
64.3
31.0
51.0
60.7
89.5
25.1
1lama3.2-1B
15.00
63.0
63.7
74.5
65.5
31.3
49.7
60.7
91.4
32.1
59.1
Smoll M2-1.77
11.0
67.7
71.5
77.0
77.7
44.7
51.5
66.
93.3
50.3
66.6
Qwen2.5-1.58
18.0
67.7
75.8
75.5
65.0
63.4
TWWKV-World3-1.58
5.6
69.5
70.8
77.1
78.1
44.5
62.4
68.2
94.3
43.3
67.6
RWKV6- RWKV6-World2., World2.1-38
76.4
71.2
56.3
66.3
92.2
1lama3.2-3B
$15.0
70.5
73.6
76.7
74.5
42.2
50.7
69.9
95.7
56.5
67.8
Qwen2.5-3B
18.0
67.1
73.5
78.6
77.4
45.0
70.2
68.5
96.2
65.7
71.4
5.6
73.4
76.4
79.7
81.0
61.8
55.0
71.5
glue is the average accuracy of 8 subtasks: mnli, mnli mismatch, mrpc, quli, qqp. rte, 8812 and wnli
1amm3322 and 38 pruned and distilled from Lamma311888 (Grattafiori al., 2024)
Table 2: English focused benchmarks, including LAMBADA (Imb.o) (Paperno et al., 2016), Hellswag (hella) (Hampel, 1974), PIQA (Bisk et al., 2020), A12 ARC (arcE, arcC) (Bhakthavatsalam et al., 2021), GLUE (Wang et al., 2018), Winogrande (WG) (Sakaguchi et al., 2001), SciQ (Welbl et al., 2017), MMLU (Hendrycks al., 2021). MMLU 5-shot, others 0-shot
Model
Tokens (T)
Imb.m "ppll
Imb.m acc1
pawsx acc|
xcopa acc|
xnli acc|
xsClz acc|
xwin acc
avg acc|
(Name)
RWKV5- WWKV5-World1--o -0.
0.6
270
22.0
48.6
53.0
36.1
51.7
59.5
45.1
SmolLM2-1333 135M
2.0
1514
18.6
51.2
52,2
34.9
50.6
61.7
44.9
RWKV7-0.1B
1.6
114
31.6
46.1
53.3
37.6
52.6
64.1
47.5
RWKV5-World--0.4B
1.1
66
36.8
49.5
54.0
38.5
54.1
65.6
49.8
M201LMM--300
4.0
389
25.8
51.4
51.7
36.0
51.2
67.8
47.3
Qwen2.5-0.5B
18.0
108
32.9
52.6
54.4
38.6
53.9
67.8
50.0
RWKV7-World3-0.BB
3.1
52
39.6
48.7
55.4
40.3
55.3
72.9
52.0
RWKV6-
WWXV6-World2.1-1.63
28
52.5
58.1
41.4
76.5
55.7
Llama3.2-11
115.0
52
39.0
53.9
55.3
41.2
56,6
72.2
53.0
MmolLM2-1177
11.0
85
37.1
56.5
53.1
38.1
54.1
72.8
52.0
Qwen2.5-1.5B
18.0
49
40.0
55.3
57.4
40.6
57.7
75.8
54.5
RWKV7-World3-1.5B
5.6
25
48.4
54.8
59.7
43.7
61.4
79.8
58.0
RWKV6- W
51.0
53.4
60.2
42.7
61.3
78.8
57.9
Llama3.2-33
115.0
30
45.9
59.9
58.5
44.2
60.6
79.2
58.1
Qwen2.5-3B
18.0
36
43.5
53.3
59.0
38.5
59.6
79.8
55.6
HWKV7-World3-2.9B
5.6
18
52.9
58.2
63.1
45.4
64.7
82.4
61.1
"The perplexity is the geometric mean, rather than arithmetic average, across 5 languages
"llama322- 2B and 3B were pruned and distilled from Llama3.1 (Grattafiori al., 2024)
Table 3: Multilingual benchmarks, including LAMBADA Multilingual (Imb.m) (Gao et al., 2023),
XCOPA (Ponti et al., 2020), XNLI (Conneau et al., 2018), X)SooryCll (xsCIz) (Lin et al., 2022), XWinogrande (xwin) (Tikhonov & Ryabinin, 2021). D. All 0-shot.
Additional Evaluations Please see Appendices L, and M for extended evaluations, multi- modal experiments, and ablations.
8 Conclusions
We introduced RWKV-7, novel RNN architecture that achieves state-of-the-ee performance for its size across a wide range of benchmarks, rivaling even highly optimized models such as Qwen2.5 despite being trained many fewer tokens. As RNN, RWKV-7 maintains high parameter

---

Published as a conference paper at COLM 2025
Model
Compress
Fuzzy Recall
In-Context Recall
Memorize
Noisy Recall
Selective Copy
Avg
RWKV-7
44.5
43.2
100
89.1
100
98.8
Transformer
51.6
29.8
94.1
85.2
86.8
99.6
74.5
Multihead Hyena
44.8
14.4
99.0
89.4
98.6
93.0
73.2
DeltaNet
42.2
35.7
100
52.8
100
100
71.8
Mamba
52.7
6.7
90.4
89.5
90.1
86.3
69.3
Hyena
45.2
7.9
81.7
89.5
78.8
93.1
66.0
GLA
38.8
6.9
80.8
63.3
81.6
88.6
60.0
Results comparison models from Yang (2024c)
Table 4: Results on the MAD benchmark
arXiv CS
arXiv Phys.
Github Python
Github C++
A03
BBC
Wiki Eng
Model
Eng
news
average
Qwen2.5-1.5B
8.12
8.65
4.42
4.40
11.76
9.58
9.49
8.06
RWKV-7 1.5B
8.25
8.77
5.57
5.29
10.93
9.34
8.97
8.16
Llama-3.2-LB
8.37
8.76
5.18
5.16
11.69
9.34
9.07
8.23
M2-11.2
8.38
9.04
5.17
4.94
11.20
9.40
9.46
8.23
Table 5: Compression rate LLMs on post January 2025 data sources
efficiency, linear time complexity, and constant memory usage, offering a compelling alternative to transformers.
Limitations Despite strengths, the architecture and models have some limitations, including prompt sensitivity, kernel precision limits, and lack of full instruct and alignment tuning. We also believe that downstream performance has been restricted by our access less training compute than other top models. See Appendix Q for detailed discussion.
Future Work We plan to train larger models, investigate training speed improvements, and train reasoning models that take advantage of increased inference time compute. Please see Appendix R for details.
Acknowledgments
We extend our gratitude Shenzhen Yuanshi Intelligent Co. Ltd. and Shanghai Yuanwo Intelligent Co. Ltd. for providing computational resources and their dedication to promoting and com- mercializing RWKV. We thank Featherless Al for their extensive experimentation with the RWKV architecture and their contributions to this paper. We grateful to the members of the RWKV and EleutherAl Discord communities for their collaborative efforts in extending the applicability of RWKV to diverse domains. We extend special thank you to Stella Biderman, Songlin Yang and Yu Zhang. Eric Alcaide acknowledges support for this work from the Swiss State Secretariat for Education, Research and Innovation (SERT) under contract number 23.00421.
10

---

Published as a conference paper at COLM 2025
References
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarin, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. Smollm2: When smol goes big data-centre training of small language model, 2025. URL htRps::: :/apiii.////xv.../////// arxiv. org abs 252. 250.0/2777
Simran Arora, Sabri Byuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Re. Zoology: Measuring and improving recall in efficient language models, 2023.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q. liang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. In The Twelfth International Conference on Learning Representations, 2024. URI http///pppeeeeeeeeeeeeeee net forum? dd=4NnqRR9155
David A. Barrington. Bounded-widiee width polynomial branching recognize exactly those languages in ncl. Journal of Computer and System Sciences, 38(1):150- 1989. URL https: // //www. sciencedi rect. com/ wtt.eciiieecccccccc../e/ee./.ccs wew.c//aricleeee.eeeee.oee /ttp::cccceep../00////0//////00
Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time, 2024.
Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro Werra. Cosmopedia, February 2024a. https: https///ggggggccccccc/oooo/////e cotaatsetsssssssso/aaaaaaas.a Seaattttts/uuugigggggggg /Hgggingg aceTB/ ceeTT//comopedia.
Loubna Ben Allal, Anton Lozhkoy, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Amollm-corpus July 2024b. Utthttss: https///hhhgggnaeeeeeee co/ datasets www.sstssttgggggeeeee /HuggingFac eTB/smol lemmoolmmmcce 1m-lmmmoppp corpus.
Sumithra Bhakthavatsalam, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, and Peter Clark. Think you have solved direct-answer question answering? try arc-da, the direct- ai2 reasoning challenge. arXiv preprint arXiv:2102.03315, 2021.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense natural language. Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT- NeoX-20B: An open-source autoregressive language model. In Angela Fan, Suzana Ilic, Thomas Wolf, and Matthias Gallé (eds.), Proceedings of BigScience Episode #5 Workshop on Chal- lenges Perspectives in Creating Large Language Models, pp. 95-136, virtual+Dublin, May 2022. Association for Computational Linguistics. doi: 10.18653/v 20118653/v//022// htt.//2......iiinc....... 1.9. URL https: pttts:://aclanttollooo org/ 2022 1e122..igscccc bww.scieeeeee.......
Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg- Kirkkarrrkk,k,, and Shlomo Dubnow. Hts-at: hierarchical ooken-semantic audio transformer for sound classification and detection. In ICASSP 2022-2022 JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 646-650. IEEE, 2022.
Yingfa Chen, Xinrong Zhang, Shengding Hu, Xu Han, Zhiyuan Liu, and Maosong Sun. Stuffed mamba: State collapse and state capacity of rnn-based long-context modeling, 2024a. URL https:: //arxi///xxvvv/ org/ otg/abss////eooooogoogooo/so 2210 0 07145.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long--coneed large language models, 2024b. URL https: /tattts:://xxiiii /arxxiv. org//bb//300) /209. 12307.
11

---

Published as a conference paper at COLM 2025
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating coos-llngual sentence representations. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2475-2485, 2018.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2023.
Dao Al Lab. Causal convld. http:s/:/ggatbbb..c////aAAAAa/// Accessed: 2025-02-26.
2023.
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, L. Cai, Jian Liang, Jianzhong Guo. Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingehang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang. Liyue Zhang. Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang. Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang. Tao Yun, Tian Pei, Tianyu Sun, W. I. Xiao, Wangding Zeng. Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang. Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. L., Xiangyue Jin, Xianzu Wang. Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang. Y X. Wei, X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang. Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ving Tang. Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Viyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang. Yue Gong, Yuheng 7ou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. Wa, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2025. URL / arxiv. wrg. /atgg//a//// 22122242222222 19437.
Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus
Mattern, Jordi Hutter, and Joel Veness. Language modeling is compression, 2024. URL /ttp///xvvvv..////// g/ //abb/2230 2209 10668.
Yuchen Duan, Welyun Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, Yu Qiao, Hongsheng Li, Jifeng Dai, and Wenhai Wang. Vision-rwkv: Efficient and scalable visual perception with rikv-like architectures. arXiv preprint arXiv:2403.022: 02300, 2024.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodes Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https://transformer hatpiassooomeerrrcttttsssssss httpr//meerrrrruuuuoroo/ee/ee//e /ttp///wwww//nnee.eiiiinte
Qihang Fan, Huaibo Huang, and Ran He. Breaking the low-rank dilemma of linear attention, 2025. URL https:t///// aarivv. www.xiiwv....vv....ooo...o.ooore /bss//11ll.. 07635.
Logan Ford, Hao Tang, François Grondin, and James R Glass. A deep residual network for large- scale acoustic scene analysis. In InterSpeech, pp. 2568-2572 2019.
12

---

Published as a conference paper at COLM 2025
Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Re. Hungry hungry hippos: Towards language modeling with state space models, 2023.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URthttps: zeeodo hwwpd//eord///// / 11225668 6836.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776-780. IEEE, 2017.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Ko- renev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Allonsius,
Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab Al- Badawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang. Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kar- tikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Sil- veira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vin- cent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiao- qing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit
13

---

Published as a conference paper at COLM 2025
Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Fisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching- Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filip- pos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean- -antistt Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Her- moso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Fgebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang. Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mi- hailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda I., Vilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. UtLhttps: htt//axxvv..o www.rxivv...rr.eeooo o abs / 247.2217777 21783.
Riccardo Grazzi, Julien Siems, Jörg H. Franke, Arber Zela, Frank Hutter, and Massimiliano Pontil. Unlocking state-tracking in linear rnns through negative eigenvalues, 2024. htts:::::
//aaxiiv.org org/ abs/ / /21121111111 12537.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2023.
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces, 2022.
14

---

Published as a conference paper at COLM 2025
Frank R Hampel. The influence curve and its role in robust estimation. Journal of the american statistical association, 69(34))3883 393, 1974.
Dongchen Han, Yifan Pu, Zhuofan Xia, Yizeng Han, Xuran Pan, Xiu Li, Jiwen Lu, Shiji Song, and Gao Huang. Bridging the divide: Reconsidering softmax and linear attention, 2024. URL https: / arxiv. org/ abs////////////// 2412. 06590.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https: //ar xiv. www.org///aa absg///b//// abaaabas/20 2009 03300.
Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
Haowen Hou, Peigen Zeng, Fei Ma, and Fei Richard Yu. Visualrwkv: Exploring recurrent neural networks for visual language models. ArXiv, abs/2406.13362 2024. URL https: : aapi.eiiee wttpsschhoaar.oo/oooououuuu wwtpooog/o/oooooo : 270620880.
marticscholi
Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recogmition (CVPR), pp. 6693-6702, 2019. ht.t::: /ttteps://ppppp /api.......// sewant iemanticscholar.
arg/CorpuSID: 152282269.
Jean Kaddour. The minipile challenge for lata-efficient language models, 2023.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns:: Fast autoregressive transformers with linear attention. In International conference on machine learning, pp. 5156-5165, PMLR, 2020a.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran,cois Fleuret. Transformers are mnss: Fast autoregressive transformers with linear attention. Proceedings of the 37 th International Conference Machine Learning, 2020b.
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng- Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El- NouN, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-ha In search of the next generation of training sets for language models, 2024a.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj. Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Polan-Gavitt Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you!, 2023a.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji rong Wen. Evaluating object hallucination in large vision-language models. In Conference Empirical Methods in Natural Language Processing, 2023b. URL http://ppaaaa........///// wemanticscholll. orq/CorpusID: 258740697
15

---

Published as a conference paper at COLM 2025
Yucheng Li, Yunhao Guo, Frank Guerin, and Chenghua Lin. Evaluating large language models for generalization and robustness via data compression, 2024b. Utthtpp:: / aariv.iiiv...//.i.///aa/////aava org/ ab 01240200088
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. Few-shot learning with multilingual generative language models. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 9019- 2022.
Bo Liu, Rui Wang, Lemeng Wu, Yihao Feng, Peter Stone, and Qiang Liu. Longhorn: State space models are amortized online learners, 2024. URL htwps:ttps/.oom httpps://aaiiy/ org/ abss/40000 142 07.
Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024. URL http:::: U httpg//tgggggcccece////// co/dataset.ss "gingFaceFF/ fineweb-edu.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. ArXiv, abs/2209.09513 2022. URL https: :ttpss///appa /apii.... semantic scholar. wttp:oopooooooooooo 252383606.
Shahar Lutati, Itamar Zimerman, and Lior Wolf. Focus your attention (with adaptive iir filters), 2023.
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large- batch training. ArXio, /abs//811.061611 2018. URI. Uttphtps:: /api. semanticsche r.org/ Ctrpus ID: 52262883
William Merrill and Sabharaal. The expressive power of transformers with chain of thought, 2024. URL hiLpst::: http://wwwww....aa///aaa//////// 079233
William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state in state-space models. ArXiv, abs/2404.08819, 2024. URL https:: /tpp//app..aa///aa// www.eeeeeeccaieeeeeeeeeeeeeeeeee www.aarorg/ooooooor...
269149088
Igor Molybog, Peter Albert, Moya Chen, Zachary DeVito, David Esiobu, Naman Goyal, Punit Singh Koura, Sharan Narang, Andrew Poulton, Ruan Silva, Binh Tang, Diana Liskovich, Puxin Xu, Yuchen Zhang, Melanie Kambadur, Stephen Roller, and Susan Zhang. A theory on adam instability in large-scale machine learning, 2023. URL httthtps/// /ttp://///xxvvoo// org wttggb//gga///a/e//e///////////e 9871.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah, In-context learning and induction heads, 2022.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv: 160v:1606.00 160,,06011, 2016.
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset sigh-tuulity mathematical web text, 2023.
Bo Peng. Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartiomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang. Johan Wind, Stanislaw Wožniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: Reinventing RNNs for the transformer era. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14048-14077 Singapore, December 2023. Association for Computational Linguistics. doi:
16

---

Published as a conference paper at COLM 2025
10.18653/v112223..mmm findingg wwwniiing--ennll....e. emnlp.936. URL URL /tt.itpp////cccatttooooooo////// find ings-emnip. 936.
Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, Przemyslaw Kazienko, Kranthi Kiran GV, Jan Kocoń, Bartlomiej Koptyra, Satyapriya Krishna, Ronald McClelland Jr., Jiaju Lin, Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song, Haoqin Tu, Cahya Wirawan, Stanislaw Wožniak, Ruichong Zhang, Bingchen Zhao, Qihang Zhao, Peng Zhou, Jian Zhu, and Rui Jie Zhu. Eagle and finch: Rwkv with matrix-valle valued states and dynamic recurrence, 2024a. URL https: arxiv. org/ abs / 4404. 05892.
Bo Peng, Daniel Goldstein, Quentin Gregory Anthony, Alon Albalak, Eric Alcaide, Stella Bider- man, Eugene Cheah, Teddy Ferdinan, Kranthi Kiran GV, Haowen Hou, Satyapriya Krishna, Ronald McClelland Jr., Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song, Haoqin Tu, Ruichong Zhang, Bingchen Zhao, Qihang Zhao, Jian Zhu, and Rui-Jie Zhu. Eagle and finch: RWKV with matrix-valued states and dynamic recurrence. In First Conference on Language Modeling, 2024b. URL https: openreview net Eorum weweeuumiddssooo Eorun??ddsoolllieee.s
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Yu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. International Conference on Machine Learning, pp. 00004-28078... PMLR, 2023.
Michael Poli, Armin W Thomas, Eric Nguyen, Pragaash Ponnusamy, Björn Deiseroth, Kris- tian Kersting, Taiji Suzuki, Brian Hie, Stefano Ermon, Christopher Ré, Ce Zhang, and Ste- fano Massaroli. Mechanistic design and scaling of hybrid architectures, 2024. URL https: //arxiv. wttp//rg//bbbaaaaeeee 17844.
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. XCOPA: A multilingual dataset for causal commonsense reasoning. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2362-23766 Online, November 2020. Association for Computational Linguistics. doi: 10. (0.18653/vel 11186633v1/2000 wtt./.2e22mm.mm.. main. //aclanthie ogy.og.oooooooooo wrgg 200g0/2220..m emn p- ma i 1 185.
URL https:
Qwen, ;, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei I., Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhoo Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https: aaaiiiiii.vv//...//aaaaaaaaaaavv org/ wtgg/bs/////oooooooo/ogo///aaera 2412.15115.
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling, 2019. URL https: //te:///axiiiiii atwaaaxiivvvoo.... orgi /bs/ 1911.05501
Mark Rudelson and Roman Vershynin. Sampling from large matrices: An approach through geometric functional analysis. J. ACM, 54(4):21 July 2007. ISSN 0004-5411. doi: 10.1145/12 55443.1255449, URLhttps: /oi wrw/ /10 1 1011111251114333 1255449.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9);99-106, 2021.
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers secretly fast weight programmers, 2021. URL htt.tttss///aii....t.t www.xxiw..gg/..x.oooo// http:///.....................111 11174.
Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139, 1992. doi: (.1.2/neco.1992.411
John Schultz, Jakub Adamek, Matej Jusup, Mare Lanctot, Michael Kaisers, Sarah Perrin, Daniel Hennes, Jeremy Shar, Cannada Lewis, Anian Ruoss, Tom Zahavy, Petar Veličković, Laurel Prince,
17

---

Published as a conference paper at COLM 2025
Satinder Singh, Eric Malmi, and Nenad Tomašev. Mastering board games by external and internal planning with language models, 2024. URLhttps: httptt:p:://axi..oottt wwwpxivvorrgaaaeo abs 2412.1 2119.
Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flash- attention-3: Fast and accurate attention with asynchrony and low- now-precision. arXiv preprint arXiv:2407.08008 10, 2024.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read, 2019.
Samuel L. Smith, Pieter-Jan Kindermans,, and Quoc Le. Don't decay the learning rate, increase the batch size. In International Conference Learning Representations, 2018. URL https: ///p/poaatt //pppnrevieww.. net. i
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. https: jama a 627b token meaxen-cleanee cleaned and ded opplicttee-vrrrioo--f--eeopppppp fama, June 2023. URL htth::::: htattts://hggggggffett http://ggggggn.ccccoo 627B.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. arXiv preprint, 2024.
Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin. Learning to (learn at test time): Rnns with expressive hidden states, 2024.
Yutao Sun, Li Dong. Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: successor to transformer for large language models, 2023.
Alexey Tikhonov and Max Ryabinin. It's all in the heads: Using attention heads a baseline for cross-lingual transfer in commonsense reasoning, In Findings of the Association for Computa- tional Linguistics: ACI-IJCNNPP 2021, pp. 3534-3546, 2021.
Oguzhan Topsakal, Colby Jacob Edell, and Jackson Bailey Harper. Evaluating large language models with grid-based game competitions: An extensible llm benchmark and leaderboard, 2024. URI /tapttss://rxivvi arxi .ow./ii.oorrrreo.e htt.ttpss///a....ooog//aa/b////
Various. Archive of our own, 2025. URL ht htt.tttpee//aaaaieeeooooooooooee org.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL neurips. wwtp:per_pppppppeeeeeeeeeeeee 1ttpee/aeee./////11/1111 http://iiieeeeeee23333334445 fidd0555 4a845aa-Paper. pdf.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNIPP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, 2018.
Jason Wei, Maarten Bosma, Vincent Y. Zhao abd Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models zero-shot learners, 2021.
18

---

Published as a conference paper at COLM 2025
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: ooppnreveew.... net/ forum lel___iolleess
Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User- gerergeeerateddd Text, pp. 94-106, 2017.
Bernard Widrow, Marcian E Hoff, et al. Adaptive switching circuits. In IRE WESCON convention record, volume 4, pp. 96-104. New York, 1960.
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing, 2024. pol.https://arxtte http://aaxxiv...a/aaaaaaa// wwtp//o/////////e/oo 08464.
Takuto Yamana. Egaroucid, January 2025. http:: / www. www.wegrouci.ddcommo d. htwp://www...i...na......m wwwnyan.aann.... dev/
Songlin Yang and Yu Zhang. Fla: A trtton-based library for hardware-efficient implementations of linear attention mechanism, January 2024. BRLhtttss//giihhbb wtt.tttw..c.ccaaaaggttittlt
sh-lihiee www.lliaaa--aa atteetion..
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware -effcrreeffficient training, 2023a.
Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule, 2024a.
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware efficient training, 2024b. URL battps://arxiv.ortt wttp://..rrr/////////oooooosss// absg/////////// / 12. 06635.
hardware-efficient
Songlin Yang, Bailin Wang. Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length, 2024c. httptttppps/aaa...tttaeeee 2406.0 6484.
Xiaocong Yang. James Y. Huang, Wenxuan Zhou, and Muhao Chen. Parameter-efficient tuning with special token adaptation, 2023b. Utthttps: watpiviiiivv//ii///aaaaaaaaa org/ /2100 04382.
Weihao Yu and Xinchao Wang. Mambaout: Do we really need mamba for vision? arXiv preprint arXiv:2405.072 07992, 2024.
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng. Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. Advancing IIm reasoning generalists with preference trees, 2024.
Xiang Yue, Tuney Zheng, Ge Zhang. and Wenhu Chen. Mammoth2: Scaling instructions from the web. Advances in Neural Information Processing Systems, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence?, 2019. URI.https://arxt http//xxivvvvro... org/ abs 000.909588800 07830.
Yifan Zhang, Yifan Luo, Yang Yuan, and Andrew Chi-Chih Yao. Training and evaluating language models with template- based data generation. arXiv preprint arXiv:2411.18104 2024.
Zhanchao Zhou, Tianyi Wu, Zhiyun Jiang, and Zhenzhong Lan. Value residual learning for alleviat- ing attention concentration in transformers, 2024. URL.tttss:/aaxxiv
.17897.
Alex Zhuang, Ge Zhang, Tianyu Zheng, Xinrun Du, Junjie Wang, Weiming Ren, Stephen W. Huang, Jie Fu, Xiang Yue, and Wenhu Chen. Structim: Towards building generalist models for structured knowledge grounding, 2024.
Nicolas Zucchet and Antonio Orvieto. Recurrent neural networks: vanishing and exploding gradients not the end of the story, 2024. URI.https:
/240////b/./......///// 21064.
19

---

Published as a conference paper at COLM 2025
A Author Contributions
Bo Peng Original RWKV-7 ideas, original code, performance optimizations, original experiments, dataset dataset composition, and trained models from 0.1B to 2.9B.
Ruichong Zhang Personnel organization, wrote Sections 3, 4, J.2, 8 8nd Appendices C, E, H, N, M.I.P, Figures 1, 2, 5, 6, 15, and Tables 1, 3,,,3, 14,9, 9, 11, 12, 13, 19, 21. Additional contributions on implementing RWKV-7 for flah-linnar-attenntoo and converting RWKV-7 models on HuggingFace.
Daniel Goldstein Manuscript organization, initial draft sections 1, 2, 4, 5, FLOPS portion of figures 4 and 7, and appendices B, F, G. Proofreading and revisions of full manuscript. Oversaw and chose experiments for Appendix M.2 and for pass-key subsection J.4. Assistance with Appendix D revisions and ideas. Developed and tested initial RWKV-7 Hugging Face model code.
Eric Alcaide Section validation of CUDA kernels for scalable training, and manuscript proof- reading.
Xingjian Du
Experiments and writing of audio modeling in Appendix K.2
Wrote Section K.1, covering architectural design, coding, model training, experi- mental evaluation, evaluutiin well figure (Figure 12), table (Table 17), and text writing.
Haowen Hou
Jiaju Lin
Lin Experiments and writing audio modeling in Appendix K.2
Jiaxing Liu
Experiments and writing audio modeling in Appendix K.2
Janna Lu Needle-in- haystack evaluations for Section J.4, experiments for figure 4, figure 7, figure 10, table 2, and table 19. Edits to Section 1, G and abstract.
William Merrill
Developmental discussion, proofreading and revisions for Appendix D.
Guangyu Song
Section 7. Experiments for
Kaifeng Tan Section 7, Figures 8, 9. Appendix on board game modeling with Othello/ /Rveeri,, including training data design, model implementation, experiments, and result analysis.
Saiteja Utpala
Section J.5 and state tracking experiments for Figure 11.
Nathan Wilce Extended context-length dataset development and extended-length model train- ing. Description of extended context-length dataset in J.4
Johan S. Wind Main author RWKV-7 CUDA kernel implementations. Experiments for Figure 3. Section 7 and Appendices I, D. Contributions Appendix C.
Tianyi Wu Appendix D and Appendix O. Contributions to section and Appendix C (proofreading and revisions).
Daniel Wuttke Constructed an itemized table of v2-v3 world datasets. Proofreading and revision the manuscript. Contributed to Abstract, Sections and B. Contributed to Tables 6, 7, 8, and 12 Performed evaluations of RWKV-7 world models and reference base models (Table and
Christian Zhou Zheng Experiments and writing for Appendix M.2 and Table 20. Contributions to Sections and 2. Proofreading and revisions of full manuscript.
20

---

Published as a conference paper at COLM 2025
B
Training Dataset Details
The RWKV World v3 corpus builds upon the RWKV World v2 corpus (Peng et al., 2024b) in two steps that we describe separately here for the purposes of reproducibility for the Goose training runs: World v2.1 adds the entries listed in Table to sum to a total of approximately 1.4 trillion RWKV World Tokenizer tokens. World v3 adds more entries, listed in Table 7, to sum total of approximately 3.1 trillion tokens. In the combined corpora, all tokens are given equal weighting unless otherwise noted.
Dataset
Domain
Dataset
Domain
slimpajama C4 dolma v1.6 (reddit only) saive-code-assistan--
Web
Maama---Maappi-Prre Pro-1M-v0..
Align
Forums
Nagpic-Pro-MT-3300--
Align
Code
Magpic-Mir-MTT-000-000
Align
m-a-p_Codee CoddCCCoeebbbae cosmopedia-v0.1 SystemChat-2., 2,0 Tess-v1.5 UltraInteract
Code
Magpic-Qwen2-Pro- - M--10.1
Align
Synthetic
Magpic-Phi--PPo--00---II
Align
Instruct
vl
Instruct
Magpie-Gennen Gemma2- P-oo-Pro-2000000
Align
Instruct
Filtered-vO.1
Table 6: Components added into the RWKV World v2.1 dataset, their source links, and their
domains.
"We added only the reddit datasets from dolma v1.6
b DM math as part of The Pile was in World v2 but missed being mentioned explicitly in (Peng al., 2024b)
Dataset
Domain
Dataset
Domain
REMOVED slimpajama parts" dclm Sain--bseline--e 10-of-lu o
Web Web
StarCoder
Code
python-edu
Code
ccnews
Web
cosmopedia-v0.2
Synthetic
fineweh-edu
Web Edu
WebinstructSub
Forums
TemplateGSM
Math
Buzz-vl .22
Instruct
open-web-mant
Math
SKGInstruct
Instruct
algebraic-stack
Math
FLAN
Instruct
Table 7: Components added into the RWKV World v3 dataset, their source links, and their domains.
"We removed the CC and C4 components SlimPajama from corpus World v3 'For For StarCoder, we now include datasets, instead just those datasets with at least stars
include only Shaooll-sarr_LLL 10 of_10
SlimPajama
Soboleva et al. (2023)
StarCoder
Li al. (2023a)
Cosmopedia
Ben Allal et al. (2024a)
Dolma
Soldaini al. (2024)
UltraInteract
Yuan et al. (2024)
Magpie
Xu et al. (2024)
FineWeb
Lozhkov et al. (2024)
DataComp LM (DCLM)
Li al. (2024a)
WebInstructSub
Yue et al. (2024)
StructI.
Zhuang et (2024)
TemplateGSM
Zhang et al. (2024)
SmolLM Corpus
Ben Allal et al. (2024b)
FLAN
Wei et al. (2021)
OpenWebMath
Paster et al. (2023)
Eigebraic-Stack
Azerbayev et al. (2024)
Table 8: RWKV World v3 dataset component citations
Most of the component data sources for the RWKV World v3 dataset are used intact, with no up- or down-sampling done that all tokens given equal weighting. Some sub-sampling done for
21

---

Published as a conference paper at COLM 2025
Category
Tokens (B)
Web
1945.2
Books
337.2
Code
258.4
Science & Wiki
222.7
Fiction
192.6
Chat & QA Instruction
110.0
Math
32.3
Law & Government
19.0
Poetry & Lyrics
1.7
Total
3119.2
Table 9: RWKV World v3 dataset categories
over-represented languages within a few data sources in the original World v2 corpus. All newly added tokens in v2.1 and v3 are given equal weighting.
C Transition Matrix Eigenvalues and Stability
We interested in all the eigenvalues of the transition matrix diag(w:) - (a, @ & and when it can be stable. We drop the subscript for statements which hold at all timesteps, to avoid clutter.
Theorem 3. Let = diag(w) ck" (a k) Mn (R) be a matrix, where all entries of w belong to (u, 1), where u exp(-e 1/2) 0.5452... is the clamping lower bound. Also, all entries of are located in (0, 1), and a unit row vector. When (0, + u), the following holds:
The matrix similar symmetric matrix, hence similar a diagonal matrix. 2. All eigenvalues of A lie in the interval (-1,1).
The matrix admits most one negative eigenvalue.
4. If further assumed that is rine-independent, then the update formula is guaranteed to be stable, i.e. there exists iime-independent constant K such that
π A
SK,
where II-llz denotes the spectral norm.
This hyperparameter in the formula be regarded "Global ICLR Multiplier". This hyperpa- rameter is set to in the current implementations of RWKV-7 language modeling.
Proof.
1. We notice that
A= diag(w) -ck (a = diag(w) ck -cck -ddiagg(a))
The matrix itself is not necessarily a symmetric matrix. However, we can use the fact that diag(a) is positive definite, rewrite
compute its square root. This allows us to
diag(a) 1/2 diag(a) diag(a) ( o 122 diag(a)diag(a) -1/2 diag(w) c ((Kdiag(a))13 (Rdiag(a)"
which is symmetric matrix. We denote this matrix by B. has exactly the same eigen- values A, since it formed by similarity transformation of A.
22

---

Published as a conference paper at COLM 2025
2. Since B diag(a)
(a(aa)))AAAiggg Adiag(a) is symmetric, all eigenvalues of B (and hence A) are real and located on the interval of
min max
[8|1
It suffices to show that $B$" E ()) This can be proved via direct expansion of B by
sB = 8diag(w)s' cS(Rdiag(a))2) 12y (Rdiag(a) (3)
2 uss" [
c-cc -l
Similarly,
$B$ sdiag(w) S cá(idiag(a)) (kdigaa ( 35 c "2
88
1,
which completes this part.
3. Recall B (iag((w)= 1 7 122) Then is congruent with ò diag(w) 112 diag(w) w) 1/2 u² u,
where u vox "iag(a)"
Clearly, has at most one negative eigenvalue with value 1 - || and all other eigenval- ues equal By Sylvester's law of inertia (Horn & Johnson, 2012), congruency preserves the number of negative cigenvalues. Hence, B also has at most one negative eigenvalue.
4. We drop the subscript in the time-invariant a;,
Since B, diag(a) 1/2 A,diag(a) 1/2 is symmetric, the spectral norm of B, is equal to the largest absolute value for eigenvalues of B,. We proved previously that the eigenvalues lie in (-1, 1), so B s 1. That is, B, is contraction matrix.
Furthermore, we have
T I [=1
A, = 11
diag(a)-112 112 B, B.diag(a) 1/2
i=1
= diaa(()) 1122 1/2
π B /=1
diag(a) 1/2
Then
T
[ A
s [|diag(a) 1/2
[ B, [=1
||dia((a
1/2
2
min(a) 1/2 1
We can set K = min(a) 00,, which is time- independent.
While stability proof only holds for time-indde -ntneeidddeedeent a,, do not observe any problems with time-varying a, in practice. We therefore put greater emphasis on the expressivity of RWKV-7, and include time-varying a,,
D Expressivity of RWKV-7
Transformers aggregate history through commutative sums, which limits their ability solve quential state tracking problems where the precise order of actions matters for the result (formally, these problems outside TC" under standard conjectures). A key example of state tracking
23

---

Published as a conference paper at COLM 2025
is computing the state of a chess board given sequence of moves, each specified in notation like 'e2e4', indicating the source and target square. Transformers, S4 and Mamba cannot solve this chess state tracking problem beyond short input lengths if no extra 'thinking' tokens are et al., 2024; Merrill Sabharwal, 2024). In contrast, our result shows that RWKV-7 can solve the chess state tracking task for any sequence length long as the model has at least four layers with sufficient width, and a head size that is linear with respect to the number of board positions.
We show that the RWKV-7 architecture can express NC² -complete state tracking problems that cannot be expressed by transformers other recurrent architectures such as S4 and Mamba, under standard complexity conjectures. We first show a particular NC -completee problem that can be expressed by RWKV-7 in Section and then generalize the argument to show that any regular language can be recognized by RWKV-7 in Section D.2. As regular language recognition can be understood to formalize finite state tracking problems, this suggests an expressivity advantage of RWKV-7 on state-tracking problems.
D.1 Warmup: Expressivity Beyond TC
Recall that the RWKV-7 wkv state is updated, at each token, by multiplication with A, diag(w,) c (a, @ x), where 1. In the following, will consider 2, and show that this yields expressivity beyond TC (unless TC NC²). TC is the complexity class which in- cludes transformers as well as all non-gated or diagonal SSMs (Merrill et al., 2024; Barrington, 1989).
Proof of Theorem
Proof. RWKV-7 can, by Lemma 2, solve the problem tracking swaps on five elements. This problem is NC² -complete under AC reductions (Merrill al., 2024).
Lemma 1.
The RWKV-7 transition matrix can represent an arbitrary swap matrix, where a swap matrix an identity matrix with two of its rows swapped.
Proof. Given indices and y, let
Wr = c= Ê, (ex (ey and a; =
Here e, denotes the vector with at position and 0 elsewhere
Then the transition matrix becomes A, - ex ex e ey ef + t efex. which is the permutation matrix that swaps indices x and y.
Lemma 2 (RWKV-7 can track swaps on 5 elements). Let a sequence of swaps on 5 elements be encoded in the format
# #[x1 - yillx2 - ynl...,
where # is a special beginning-of-sequeee between elements x; and yı. Then there exists one-layer RWKV-7 model which outputs if the sequence of swaps encode the identity permutation, and outputs 0 otherwise.
token, and [xi - yil is a token that denotes a swap
Proof. Let the RWKV-7 model have wky heads head dimension The embedding weights and "weight preparation" part of the time mixing layer are set such that the following two properties hold:
Firstly, when the model sees the special beginning-of of sequence token, the ith wkv head receives
w = 1, 2= = e; a 0, and k= ej.
Here e; denotes the vector with 1 at position i and 0 elsewhere. This sets the state of the ith wky head to wkv = e:" e,, which represents state i.
Secondly, when token represents swap between tokens 1 xxxxx x 5. In this case, using Lemma 1, all wkv heads receive
1, c= x= (ex e,)/v2, a= and k=
24

---

Published as a conference paper at COLM 2025
This changes the state to y it was or x it was y, or keeps it unchanged otherwise.
To calculate the output, the ith wkv head checks whether it represents state by applying recep- tance ej. Finally, the MLP layer combines these outputs check if all 5 heads agree with the identity permutation, and the output head outputs if this is true, and otherwise.
D.2 Main Result: RWKV-7 Can Recognize Any Regular Language
Moreover, we are able demonstrate that RWKV-7 has the capability to recognize any regular language. The regular languages are precisely those which can be recognized by deterministic finite automaton (DFA). Therefore, it sufficient to show that RWKV-7 can simulate any DFA. We define DEAs in the usual way:
Definition 1. Classically, DFA is tuple = (Q,2,6, qu. F) where Q is a finite set of states, Σ is finite vocabulary of tokens, 8α Q- Q is transition function for each token 2, qo E the initial state, and F F Q is a set of accepting states.
Equivalently, the DFA's computation on w can be represented by matrix computations. Each da can be represented by boolean matrix Ma (0, 1) QIxQQ where Mul(i j) iffō q: The initial state qo can be represented as one-hot vector (0, 1) and the set of accepting states F can be represented as a multi-hu vector u E (0, 10
For given string "U..TWT the DFA computes
a (-Mey- ... Mwy T
(25)
We say that I. if and only if expression evaluates to 1.
Having defined regular languages, are in a position show main result: Proof of Theorem 2
Proof. We demonstrate that the RWKV-7 architecture can recognize strings in an arbitrary regular language L. Consider string w E E and its membership in L. There exists DFA (Definition 1) which recognizes L by evaluating (25). To prove that RWKV-7 can recognize any regular language, it is therefore sufficient to construct RWKV-7 that evaluates (25).
overview. standard way to recognize a regular language would be to construct the transition matrices for each token and then multiply them to compute the final state. However, this does not work for RWKV-7 because because an arbitrary DFA transition can have rank n IQI (i.e., the number of states), whereas wkv head only implement simple elementary transition matrix per input token. A natural idea is to factor each DFA transition into n elementary transition matrices (Lemmaa 3), each of which can be directly implemented by wkv heads (Lemma 4). However, expanding DFA transitions in this way gives more elementary transition matrices than tokens, which means it cannot be directly implemented by RWKV-7.
Construction
Fortunately, a simple modification of this idea will allow us to implement regular language recog- nition in RWKV-7. Rather than expanding each transition matrix elementary matrices, we use the first three layers to convert blocks of DFA transitions to products of n elementary matrices with the same product. The final layer then multiplies these elementary matrices to obtain the final state.
Details: information routing via residual stream and Layernorm. The output of each layer is stored in the residual stream of the architecture, in independent subspaces, which makes them all available to deeper layers. Since the outputs come from a finite set, they may be represented by one-hot encoding.
The input to each time mixing block therefore contains the outputs of all previous layers. First, a Layernorm is applied. Note that one-hot encodings have constant norm, which ensures that the Layernorm preserves the encoded information.
Next is the "weight preparation" part of the time mixing block, which takes input x, encoding the , and a, for the wkv heads. The weight preparation sufficiently expressive to allow each of the 6 output variables to be an arbitrary
output from previous layers, and constructs r, W,, k, V R,
25

---

Published as a conference paper at COLM 2025
linear transform of X,. This allows selecting Tr, k,, & and a; based arbitrary outputs from previous layers. Therefore, the wkv heads are the main part of the construction.
WKV head construction. For any 1 = = T, the current position can be split into r = In i, for integers 0 S and 1 i n. We view the input sequence as blocks of length n, indexed by
First, describe 1 Consider the product of DFA transitions Mj M Mwn-nn-- ... M -aa- This product is also a DFA transition matrix (i.e., it has a single per column). Hence, Lemma allows us to factor M2 G1 GI,... GI,,, where Gi......, Gt.n are elementary transition matrices. Fix one such factorization for each possible DFA transition matrix. At position r == In + i, the wkv state in by the elementary transition matrix Gr,-- Since G, is
the fourth (final) layer is fight-multiple uniquely defined by the last 2n tokens and position modulo 2n, it can be computed in the third layer by Lemma and fed the fourth layer.
The first block = is handled as special case. Again, we consider the fourth layer. At the first token, set V1 e1 and a. All subsequent transitions identity transition matrices from Lemma 4. Then wkv, ej a for 1 = f = n.
set V2 k = O, and implement
With this construction, for all 1 t T, the fourth layer's wkv state becomes
wkv, = R, where å = a am, Mw na Gr.1 G12... GI,i+ where as usual, empty products (such as for / = 1) evaluate to identity matrices.
Duplicate this construction into wkv heads, where the ith applies receptance e;. In combination, the wkv heads read out the whole vector å
Final tokens. Consider plied by
T. To match the DFA evaluation formula from (25), äy can be multi-
o = G..i+1...Gin Man M Mwims
M Mwinr
Fortunately, this expression is fixed function of the current position modulo 2n and the last 2n tokens, and can hence be found by a lookup in the third layer by Lemma 7. The final MLP on the final token can thus output the scalar product à T which is equal to (25), thus completing the construction.
Our construction uses MI.Ps to implement lookup tables with sizes on the order of | which may require MIPP layers that are exponentially wide in the number of states the original DFA.
D.3 Lemmas for Theorem 2
The proof for Theorem requires many different lemmas, which are stated and proved in the following.
single RWKV-7 wkv state transition cannot directly implement an arbitrary DFA transition. However, DFA transition matrices can be decomposed into product of elementary transition matrices that can be directly simulated by wkv state transitions (cf. Lemma: 1):
Lemma 3. Let M be DFA transition matrix. Le., has shape n, and contains a single in each column. Then M can be factored into a product of n elementary transition matrices GT,..., Gp- Specifically,
M G2
where each of Gr has one of the following forms:
Identity matrix.
2. Swap matrix x y; an identity matrix with rows x and swapped.
3. Copy matrix x y; an identity matrix with column y replaced by copy of column x.
Proof. We will greedily build M from the identity matrix by hight-mmltiplying elementary transition matrices. Right-multiplying by the identity matrix does nothing, right- multiplying with swap matrix x - swaps columns x and y, and right- higgh-tultpplyin with a copy matrix x replaces column by copy of column
26

---

Published as a conference paper at COLM 2025
We use X to denote the current partial product of elementary transition matrices. Initially, X is the identity matrix, and the goal is to apply n transitions to make X M. We proceed greedily in three stages.
1. Find a column с of X that differs from column of M, but which matches a different column c' of M. If no such position exists, proceed to the next stage. Otherwise, right- multiply X by a swap matrix that swaps columns and c'. Note that M and X differed in columns and c' before the swap, but match in column after the swap.
2. Find column where M and X differ. no such column exists, proceed the next stage. Otherwise, the previous stage has ensured that there exists a column of X, necessarily different from which contains column identical column of M.
Right-multiply by copy matrix that replaces column of X with column c' of X. Note that .M and X differed in column c before the move, while they agree afterwards.
3. Right-multiply by identity matrices until X is the product of elementary transition matrices.
Initially, M and X differed in at most columns. Each subsequent mothttuultppiiationn by an elementary transition matrix reduced the number of differing columns by at least one. Thus, stages and required at most multiplications.
Recall that the RWKV-7 wkv state is at token index updated by multiplication with
A = diag( w;) - c (a, o R2),
with c= 1. To simplify the presentation of the core idea, we will instead present construction with c= and then show how to remove this assumption in Section D.3.
Lemma For any elementary transition matrix (in the sense of Lemma 3), there exist n- dimensional vectors & and a, where [| = 1, ā has elements in (0, 1}, and
G diag(w) - ck (ā R),
where c 2 and w = 1.
Proof. We use e; to denote the n-dimensional vector with at position and elsewhere.
1. Identity matrix: Select any length one vector k, for example x e1, and à =0
2. Swap matrix; an identity matrix with rows x and swapped: Select & (e, ey) /2 and
ä
3. Copy matrix; an identity matrix with column y replaced by copy of column x: Select ê (ex ey)/ 2yl/v2 and ă ey.
We now move on to the explicit construction of the first three layers. Lemma 5. There is a 1-layer RWKV-7 which outputs whether the current position is first, and whether the current position is even or odd.
Proof. RWKV-7 performs a token-shift operation before the wkv heads are reached. This token shift takes as input the last token, and can therefore detect whether previous token exists.
The first layer's wkv state can track position parity by selecting kι = V1 = e for the first position == and subsequently letting 2, W, a, k, e₁ and k V; 0 for = 2. This leads to wkv = ex é1 and subsequently wkv, = wkv, (I 2e e1) for t 2. Then wkv, e/ e1 for odd r and wkv, = -ej e, for even f.
Receptance e then be used read out the sign of the wkv state, which encodes the current position's parity.
The subsequent ML.P can transform these boolean outputs arbitrary format.
27

---

Published as a conference paper at COLM 2025
Lemma 6. For any positive integer there is 2-layer RWKV-7 that outputs the position modulo 2n.
Proof. We use Lemma 5 for the first layer, which tells the second layer whether the current position is first, and the parity of the current position.
At the first position, set k P1 e1, such that wkv =ffer e1. For all subsequent positions set c 2, Wr a, = and k, V, = 0. Furthermore, set R e1 for even and k = cos(a/ n)e sin(t/n)ez at odd r. Then wkvr ex e1 (/ 22 (2)...) (1-211/pp)) 2 Note that for even 2 2, the matrix 2k 2
:) 2R r+1) rotates the first two coordinates by an angle 2 n. Thus,
_ (cos( (innnestss(((((())))))e ef
if odd -in if even
wkv,
The wkv heads are immediately followed by group normalization, which discards information about magnitudes. We therefore use 2n wkv heads the type above, where the ktth head applies receptance cos(π k/ n)e1 + sin k/ e2. The signs of these readouts, along with the parity from the first layer, can then be combined in the subsequent MLP layer to deduce the current position modulo 2n.
Lemma 7. Let == t modulo 2n be the current position modulo 2n, and let Wt, Wi-1...,, (2-1) be the last tokens. Define w. = |21 for rnfore-sequence tokens s 0. Let Eit, wi, W- W(-(2n-1) be lookup table that takes as key the current position modulo 2n and the2n most recent tokens.
Given inputs i and Wr, Wi-1.-.., 10--(211)) there is a layer of RWKV-7 that simulates Moreover, there a 3-layer RWKV-7 that simulates
Proof. Recall that the wkv state is initialized to all zeros. We apply the wkv state update wkv, = wkv, (I - eº e;) e ep.
This can be achieved by selecting c I,, w, = a, = _ k =k, = e, and v= ew,- In words, the state update replaces the žth column of the wkv state with ew, Hence, the wkv state stores the last 2n tokens.
We make n such wkv heads, where the ith wkv head applies receptance e;. This reads out the full state, which contains the last 2n tokens. The state and are fed into the subsequent MLP layer, which performs the lookup into E.
Note that by Lemma 6, the first two layers can compute current position modulo 2n and the 2n most recent tokens. Hence, a 3-layyer RWKV can compute E.
Removing the assumption c 2 Some of our constructions use c 2, while the actual model However, since the transition matrix is A, diag(w;)- cR (a, k2), halving both and W, simply causes A, to be halved. This causes the wkv state to halve in magnitude at each token. However, since the wkv heads are immediately followed by group normalizations, the magnitude of the wkv state does not affect subsequent calculations. Additionally, since floating point numbers store separate exponent, this rescaling only requires log pegg-prccsion. The shrinking state could in principle be mismatched with the scales of D, and k,, but our constructions always satisfy U, = k, 0 whenever C # is required.
Additional Architectural and Training Details
Training Method and Model Data Amounts The RWKV-7 Pile models all use the GPT-NeoX-200 tokenizer (Black et al., 2022), and were all trained from scratch on the Pile dataset, which has 332 billion tokens.
All RWKV World dataset models use the RWKV World Tokenizer. Due to compute budget con- straints, the Goose World 3 BB and 0.4B models were trained from pre-existing RWKV-5 World vl
28

---

Published as a conference paper at COLM 2025
and v2 checkpoints, and the Goose World 3 1.5B and 2.9B models were trained from pre- existing RWKV-6 World v2.1 checkpoints. These checkpoints' parameters were then converted to the RWKV-7 format via a process described below. Once in the new format, the models are trained on either the additional full 3.1 trillion tokens the World v3 corpus, or an equally weighted sub-sampling of it. Under this methodology, some documents
three times.
The World v1, v2, v2.1, and v3 corpora contain 0.6, 1.1, 1.4, and 3.1 trillion tokens, respectively.
The amounts of training in each stage with each successive model architecture and corpus
shown in Table 10.
Model
World v1
World
World v2.1
World
Total
RWKV7-World3-0.IBB
0.6 (RWKV-5)
1.0 (RWKV-7)
1.6
RWKV7-World3-D..tt
(RWKV-5)
2.0 (RWKV-7)
3.1
RWKV7-World3-1.58
1.1 (RWKV-6)
1.4 (RWKV-6)
3.1 (RWKV-7)
5.6
RWKV7-World3-2.9B
1.1 (RWKV-6)
1.4 (RWKV-6)
3.1 (RWKV-7)
5.6
Table 10: Total trillions of tokens trained for all RWKV-7 World models
Our model format conversion process involves removing the token-shift shift low-rank MLPs, rescal- ing by half the embeddings, wkv receptance, wkv output matrix weights, and Layernorm and Groupnorm bias values. Layernorm and Groupnorm weights are clamped above zero and square rooted. We widen the FFN MIP from 3.5x (in RWKV-6) to 4x and add new small 10 9) uniform initalizations the new regions, removing the RWKV-6 FFN receptance weights. We widen the time decay Low-rank MLP and add new small (1 10-*) uniform initializations in the new regions. We replace the gate weights with a LoRA obtained through singular value decomposition and rescaling by half.
Architecture Diagram We provide comprehensive architecture diagram (Figure 5) in order to help readers thoroughly understand our architecture design.
Parameters and Dimensions Throughout this section, we denote by D the model dimension, the number of layers, h D/,,, the number of heads, and V the vocabulary size. All models are trained with head size D = 64, i.e., each time-mixing has = D/64 heads with dimension 64 64.
Pile models are trained with V 50304 with the GPT-NeoX 20B tokenizer. World models are trained with 65536 with RWKV World tokenizer.
Model Name
D
State Size (WKV Shift)
Parameters
RWWV7-World3-N
12
768
589824 + 18432
191 034 624
TWKV7-World3-O0
24
1024
1572864 152
450 872
RWKY7-WorldS-1.5B
24
2048
3445 728 98304
1527 404 544
RWKVV--World3-2.98
32
2560
5242880 163 843840
2947 735 040
Table 11: Released Goose models with parameters and state size.
RWKV-7 uses four low-rank MLPs for decay w, value residual in-context learning rate and gate respectively. The intermediate dimensions are listed in Table 12. These values are based on our mere speculation of how much information can be passed through.
The number of parameters for all RWKV-7 models can be computed by the formula:
# (Params) = 2DV + 4D LD (12D + 2 (dw - + da + d, + dg) + 19) (2Dd, + D).
(26)
Where:
The weights of the embeddings and head, and the Layernorms beside them, yield 2DV 4D parameters;
29

---

Published as a conference paper at COLM 2025
Softmax
Anceo and rourdd rustangles. dinsrsenn Pi cutadd WeVV head bV:
Dimension
Output
Dasblo arono oqpanoo and metting (dimensdone
Embedding
Puppl trinut parmeeter
b
Layer Norm
Pud. interna statio
Channel Mix
Lerp
K
ReLU
V"
Squaire RaLU (4x modill dimension)
X
w
b
Layer Norm
Time Mix
Σ
WKV Heads cange (2/54)
Group Norm,
*
Lerp
Nor
Fanh
R
V
K
Lem
Lem
X211
X
b
Layer Norm
RWKV Block
w
b
Layer Norm
RMN0-7 langhage modeling,
(p-b))
Input Embedding
+ Openttr meone sinsent -100577
c305-)6 Sigmoill(i))
Figure 5: The architecture of RWKV-7, drawn in detail.
30

---

Published as a conference paper at COLM 2025
Dimension (D)
dw
dg
dp
dg
768
64
64
32
128
1024
64
64
32
128
2048
96
96
64
256
2560
96
96
64
320
4096
128
128
6144
128
128
96
640
Table Suggested intermediate dimensions for the low-rank MLPs for RWKV-7 models
The weights of each layer yield D (122++2(dd) da + d, dg) 19] parameters, except for the first layer; The low-rank MLP for the value residual is not present in the first layer, subtracting (2Dd, + D) parameters.
Parameter Initializations Proper parameter initialization is crucial for ensuring training stability and achieving optimal performance for language models. RWKV-7 employs carefully designed initialization strategy tailored to its architecture. The detailed initialization scheme is beyond the scope here but can found the official code repository. We emphasize that using the recommended initialization is essential for replicating the results in this paper. Deviations from the prescribed initialization may lead to performance degradation.
Dataset Loading The dataset used for pretraining consists of 194 079 123 tokens stored on disk, which are memory- mapped using the mmap mechanism. To ensure a diverse and pseudo- random sampling of training sequences, we employ a custom data loading strategy based on mathematical function with desirable properties. Specifically, we utilize ageudo-randon number generator defined by the function f(x) = ax3 over the finite field Z/ Z/pZ, where p is a prime number of the form 3n+ + 2. This function is chosen because it is bijection (full map) in Z/ pZ, ensuring that all possible indices are eventually accessed exactly once within one epoch.
For pretraining with sequence length of 4096, the relative address of the k-th sample is deter- mined as:
start_ start_adrress = 4096- (ak- mod p), end_addrest = start_addres + 4097.
Here, is chosen as the largest prime of the form 3n + 2 smaller than [dataset_sizee 4096], yielding p = 761 521 949. The parameter a is an integer close 0.618p that ensures good mixing properties of the generated addresses.
This approach guarantees both simple calculation and uniform access the dataset while main- taining
By leveraging the properties of modular arithmetic and cubic mappings, we achieve balance between computational efficiency and data diversity during pretraining.
Training Details All RWKV-7 models were trained under b ofloat16 format on nodes of 8x Nvidia H800. The AdamW optimizer was configured with P1 0.9, ß2 0.99, e 10-18 18, and weight decay of 0.1 applied exclusively to linear layers and embedding weights. The choice of such a small e value motivated by the theory proposed by Molybog et al. (2023), which suggests that reducing can help stabilize training in large-scale models by ensuring that intermediate layers remain in a regime of active updates, thus mitigating sudden loss spikes and promoting smoother convergence.
The context length for pretraining was 4096 tokens. The base decay rate wo parameters are placed into special 2x learning rate multiplier grouping.
Besides the traditional cosine learning rate decay schedule, we used a phased dynamic batch size scaling strategy inspired by the concept of critical batch size proposed by McCandlish et al. (2018) and similar to the approaches Smith et al. (2018). Our strategy involves progressively increasing the batch size during training, accompanied by corresponding adjustments to the learning rate.
31

---

Published as a conference paper at COLM 2025
Model
Phase
Nodes
Batch size
Proposed Initial LR
Final Loss
RWKV7-World--..1B
240 x 4096
6 x 10
2.5290
RNKK7-World3-0045
240 4096
5 10 6 10
2
2
480 4096
2.2580
RWKV7-World3-1.5B
480 4096
4 x 10-4
2
672 4096
4.5 10
3
6
1152 4096
6.1 x 10
1.9970
RWKV7-World3-22
640 4096
10
2
6
1008 4096
5 10-4
1120 4096
5.4 x 10
2016 4096
8 x 80-4
1.8745
Table 13: Training schedules and batch sizes
The detailed training schedules for different model sizes are listed in Table 13.
The learning rate undergoes cosine decay schedule from the proposed initial learning rate at the beginning of the entire training run to the expected final learning rate of x 10 at the end of the entire run, but the implied initial rate varies across phases.
This approach not only enhances training efficiency but also utilizes GPU resources economically. After smaller models complete their training, additional GPU resources become available for the later stages of training larger models. This cascading resource allocation ensures that compu- tational power is dynamically reallocated, maximizing hardware utilization and reducing idle time.
We observe extremely stable training without any loss spikes in all four runs, indicating that the likelihood of encountering such spikes during the training of very large RWKV-7 model is minimal.
See Figure 6 for the resulting learning rates and observed loss curves.
Despite the general stability of loss curves, we did sometimes observe NaN loss across single training step, which we theorize may be due to our use of such an extremely low AdamW e. When this occurs, rewind the training to the prior checkpoint, clear optimizer states, and continue from that point.
F Additional Architecture Discussion
The following is general overview of the RWKV-7 architecture and selected rationale for its design choices:
The RWKV-7 Time Mixer begins with a feature that has been in RWKV since its inception: token shift. Token shift is variety of 1D short convolution that is intended to allow the model create induction heads within a single layer. At its core, token shift is just a linear interpolation between the current and prior tokens on per channel basis, where the amount per channel is learned parameter. Many other modern models (e.g. Mamba) have begun including short convolutions before attention replacements (See also Dao AI Lab (2023)). RWKV-6 introduced an advanced new form of token shift that was data dependent. While we found that this was beneficial in terms of loss decrease per step, we made the judgement call that the improvement training and inference efficiency was not worthwhile. Therefore, RWKV-7 includes only the simple token shift found in RWKV-4 and RWKV-5.
RWKV-7 time mixing follows the overall form of delta rule, applying SGD-like mechanism to train the state at test time, such that when presented with a key it will respond with an appropriately matching value, like those it has been shown previously. This can be conceptualized as simple order of operations: 1) decay the state 2) remove part of the old value located at the current key 3) add part of the new value into the current key. Then, a query (we call it receptance) is applied to
32

---

Published as a conference paper at COLM 2025
LC40
1100
2.79
2.74
2.2
3.44
3.4
2.00
2.52
2.34
2.59
2.44
2.9
2.5
(a) O.1B
(b) 0.4b
2.54
2.54
232
2.38
2.14
1.541
15.04
3.53
2.00
3.40
2.08
1.244
1.50
18.00
2.02
3.4
1.544
34:00
501
1061
1941
2000
2500
681
1301
390
2001
2821
2041
(c) 1.5B
(d) 2.9B
Figure 6: Training loss curve for RWKV-7 World models. Blue line: loss; Red line: smoothed loss; Green line: actual learning rate; Dotted red line: learning rate schedule.
the state in order to request the value located at specific key. After that, the remaining operations normalize the returned values to keep numerical sizing consistent, apply gating, and recombine the heads to obtain the output.
In SGD terms, the decay found in models like RWKV-7 and Gated DeltaNet can be thought of as similar to weight decay, RWKV models since RWKV-4 have employed some form of valued decay on the state in place of positional encoding. We continue this tradition with RWKV-7. While vector valued decay is harder to compute efficiently than its scalar valued equivalent, it brings significant improvement in loss per step. Many other models, like Mamba-2, make the choice to maximize training efficiency and simplify their kernels by using scalar decay, purposely trading quality for speed.
We are forced limit the maximum decay that can occur in a given time-step both in order to maintain training stability and to assist the creation of fast but numerically stable kernel implementations. We have many varieties of kernel available today, and are still working on new designs with enhanced efficiency and accuracy. Please see Parallelizing Linear Transformers with the Delta Rule over Sequence Length (Yang et al., 2024c) and its accompanying blog post series for insightful details on many components of such algorithms. We hope to be able to reduce the decay limit further in future revisions.
This lower bound on the decay multiplier is expressed by the exp(-e 0.5 (d,)) formula for decay. a rearrangement of an original source formula exp(-exp(- 0.5 softplus(d:)). The outer exp(-exp(x)) in the original is nearly flipped version of sigmoid, but with better gradient. The -0.5 - softllssd is a way of limiting the inputs to this sigmoid- like function to be less than -0.5, which results in the final decay being greater than 0.545. This means that decay can remove at most 45.5% of the pre-exiie pre-existing values in the wkv state from one timestep to the next. More can be then removed by the delta rule mechanism.
33

---

Published as a conference paper at COLM 2025
The in-context learning rate is usually equivalent to the learning rate in SGD, Ours is bit less restrictive than the traditional delta rule or SGD would allow. We also make this data dependent and extend it to be vector-valued instead of being merely scalar. This is allows each key channel to have its own learning rate, dependent on the current token. Note that in TTT, Gated DeltaNet, and Titans although the in-context learning rates are data dependent, they are only scalar valued.
Some of the design of RWKV-7 with regard to the in-context learning rate is theoretically motivated but may not be apparent from the equations. RWKV-6c, a later v6 sub-version with no trained models but which worked well experimentally, kept its state somewhat normalized using new design. As mentioned early in this paper and consistent with the observations in Yang et al. (2024b); Chen et al. (2024a), there is a fundamental issue with linear attention numerically growing in an unbounded fashion, and the base RWKV-6 revision has this problem. RWKV-6c, however, defeats this issue by ensuring that the amount of key added to the state is never more than the amount removed by decay. It accomplishes this by pre-multiplying the key by one minus the decay before adding it into the wkv state.
Early versions of RWKV-7 attempted to use similar mathematical formulations to keep everything normalized. But experimentally we found that the model both did best and was most efficient when we allowed it enough mathematical leeway to make these decisions on its own, rather than enforcing them. So instead of pre- multiplying the key, we give the replacement key latitude in its learning rate via the replacement rate booster.
Similarly, the removal key is decoupled from the replacement key. Note that the removal key is normalized. This is important in delta rule variations, because otherwise the outer product of removal key with itself would cause unwanted changes in the amount removed due to the implicit squaring of its length during that process.
Another point that may not be clear upon first examination is the importance of RWKV-7 being outside of the TC complexity class in which transformers, linear attention, and many varieties of SSMs lie. Consider single layer of the Key Value Cache of transformer. It is appended to upon each new token processed, and can never be changed. RWKV-7, however, can and does edit its state at each subsequent token. This can include simple operations like replacement, which can be viewed as functioning similarly to a transformer losing KV Cache entry via some sparsity mechanism e.g. a sliding window. But it can also include complex operations like swapping two entries in the state; operations transformer cannot do within its immutable KV Cache. These kinds of operations can be used to enact computations on the state itself, which lends greater computational abilities to RWKV-7 when processing a fixed set of inputs. You might think of the RWKV-7 state as being like an internal scratchpad.
There are easy problems that simply cannot be solved by transformers on fixed inputs because they lack this ability. One example is that if you give a transformer an ordered set of items and a list of which ones swap places, it will not be able to tell you which item ends up in which position at the end. Both architectures gain even more power when they are allowed extra outputs (e.g. chain of thought), becoming Turing complete. But this requires costly additional inference-time computation, whereas the RWKV-7 state does not.
A possible way to interpret the designation of RWKV-7 is training a model to learn (how to train an internal model). A WKV head can be viewed as a linear transformation that takes the input (receptance) and outputs o. Every WKV head can be regarded as a linear model that can update its weights as the context progresses. The entries of WKV states are exactly the parameters of these models.
After receptance is applied to the WKV state, we normalize the result on per head basis. This has become common across many linear attention and post-linear-an attention architectures. is way of ensuring that change in the numerical size of the state over time does not impact the model's ability to use the state. The original formulations of RWKV-4 used denominator in place of normalization to achieve a similar effect, but it is costly, harder to code, and uses more memory.
The readout part of RWKV-7 differ from RWKV-6 by the addition of the per-head (r, o k,))) scalar gating of v;. The trainable parameter P resembles the design of time--irst' term existing from RWKV-4 to RWKV-6, under the belief that the information of the current token deserves special treatment. The fime-first" term was fused inside the WKV kernel in previous architectures
34

---

Published as a conference paper at COLM 2025
of RWKV, but we decide to extract this term out of the kernel to simplify the implementation of RWKV-7.
G Pseudocode For RWKV-7
import math import torch th
import terch. functional as
def rwkv timemix (params, vprime_ 0, shift_ ikv_state, layer id)
B, T, shape
batch pequeecce_eengtien d model head size
wkv_state. shape [-1]
H ///N
head count
weight preparation
i.shifted th. [[shiftssaaee x(:, 1-1, :]], dim=1) shift_state x [ -1;, :]
x_receptance = th. lerp (x, x_shifted, params.mu.
Gecay
th. (x, x_shifted, params
x_key
th. th. x_shifted, params. mm_k)
x_value
1eh, x_shifted, params params.mu_v) mu_v)
x_iclr
1th. x_shiffed, params.mu.e = -th. (x, shifted, params
gate
x receptance params. W receptance
$ BTC
d
params. doray_lora (x_decay))
| BTC
k
x_key 0 params.M_key
$ BTC
vprime x_value 8 params.N_valur
$ BTC
gate params.gate_lon
aa_oooaaaagaaaeee (x_gate)
BTC
iclr
(x_ (x_iclr) sigmoid
# BTC
1st layer: return value to use in later layers, no interpolation if layer_id == 0:
v vprime_0 0 vprime
else:
value_residual - th. (params. th. (vprime, vprime_0, value_residdaa_gate))
decay tx.exp (-math.exp removal_k k removal. mey_muulippier mew.eeeeeeea...e._tputttieeeeer.
(th. sigmoid())
removal_k F t ", dim=-1) (B, C)
replacement_k th. k iclr, params. iclmm_aall
recurrence relation out th.empty_like
for t in range (T):
single step aecay_t
state transition
decayl:, tl.view H, N, 1)
iclr_t
icirli, tl.view H, N, 1)
removal__tt
= removal_k[;, t].view (B, H, N, 1)
r_ replacement_kli, t].view Н, N, 1)
v_t rat
ví:, t).view(B, H, N, 1) = r(;, (Be H, N, 1)
wkv_state wkv_state decay_t.mT wkv_state state @ somoval_k_t iclr_t removal_k_t) .m mm
wkv_state wkv state v_t ceplacement_k_t.rm y = wkv_state r_t # BHVK 8 ВИKЛ BHVI nutli = v.view CB $ recombine heads
normal ization
35

---

Published as a conference paper at COLM 2025
y gwwppp
-1), groups-H, parans .nn x.xeight, params In_x.bias, eps 1e- 5) vvew (B, I, -1)
bonus and output
bonus (r*k*params www.sbonuuuuutoeeeeeeeeeeeeettet mww.mmppieeirrr..mmmmmmmm. sum (dim=-1, keepdim-True)
bonus bonus virus.vieelee (B, T,C) # recombine heads out out bonus out (out gate) 8 params R cutput $ BTC
out, v0, shift_state, wkv_state
def rwky chwnnelminxi (x, state):
x_shifted th.cat (aaa(((sshii [[siift___ ([shii[tstaaaeeeeeee staiet_sttte, state, xi:, :-1, =]], dim=1) shift_state state x x(:, -1;, :] xk th. (x, x_shifted, params.mu_x)
k = params.. w_k 8 xk
params W_v th. square
return v, shift_state
def wwkv_modes (params, input_ ids, state)
params embedding (input ids)
x params Layer _areer_oorm_______________ norm pre (x)
v0 None layer_id -1
for layer in params layers:
layer_id id layer_io id + cx, v0,
state.timemix. w_wss_wwwwteeeeeeeeeeeeeee rwky_timemix
Layer.time_mix,
layer Layer.layer_a www._eyyyoooommmimmiii...o v0, state. titemitmmmiii seamiiissfttttteee weteeettmmmiixxwsseeeee layer_id
)
x = x dx
cx, state. caagerechhnneemii..
shiftstate rwkv wwkv_chanmix
layer. layer_nomm_chammix state.
)
dx
x params, Layer_norm_oon por__ooooooutttteeeeooo (x)
logits parans. head (x)
return logits, state
H PyTorch code For Naive WKV7 Kernel (Forward and Backward)
class WKv7_Kerne (nn. Module)
def [aat__(seff)__ super () init_ ()
def (sewf, r, w, k, v. a, b, state)
r r.view I H, N) k k.view (B, T, H, N)
v.view (B, T,
a.view (B, T, H,
36

---

Published as a conference paper at COLM 2025
b b.view I, self.state_cache torch. zerost (B, T+1, H, N, N
self.state_cache [;, 0, :1 state v..wie T, H, N)
w = worch.oxp(-toreee
out torch.zeros (B, T, H, N)) for in range
kk k[;, t, :]
rr Ilir t, :1 vy vli, t, :]
alir E,
bb bli, ty
state (
state w[: t, None, :]
+ torch.einsum (rrssumm'bhiie bhj-bhii',"'' state, aä, bb) torch.einsum b'bajibbi-bbhij', kk, vv)
t+1, :] = state out t, :] = torch. cinsum (bbhj, bhij->bhi' rr, state) return out, state
stlffstate_cachte
def backward r, w0, k, v, a, b, gout, gstate)
gout vout.view(0 (8, T, H, N)
teroh (BB T, a, N) gw torch.zeros (B, T,
N)
gk torch. (B, T, H, N) gv zorch.zeros (B, T, H, N))
ga terch. (B, T, H N))
gb torch. T, N)) torch.exp for t - grl:, t, :] wtw.:ttorccmmmm..... torttoo.mmmm.
T, N))
-1):
stcoaacccch tt1, :l) squeeze
sqquueee (-2)
getatt..ddd.ttoorc....m..
:, None, 1
gkl:, t, :] torch.
(statte..queee ( (-2)
gol:, t, torch torchh.atmul
gstate, Non. gal:, =to.hoocc.eeiiuurr bbiik,
(-1)
self.ssate_cache// t, :l, bli, t, :1, gstate)
gbl:, t, = torch.einsum..
bhk,
sclf.statt_cachell: t, ;l, ali, t, :l, gstate)
gwl:, t, :] torch. tbhij,bhij->ohj,,
self.state_cachel: :l, gstate)
gstate torch.einsum(
'bbj,bnij-sohij', wli, t, :l, gstate) \
+ torch.
al:, :,, bi:, :1, gstate)
gw
-torch.exp 1x0-torrh.eexx errch.exp(ww))) (w0)) gw
return gr, gu, gk, gv, ga, gb, gstate
I Speed and Memory Usage Details
We compare the training speed and memory usage the RWKV-7 attention-like kernel with the RWKV-6 kernel and Flash Attention v3 (Shah et al., 2024). The "RWKV-7" kernel accelerates bfloat16 matrix multiplications with modern CUDA instructions. We also include the "RWKV-7 fp32" kernel, which is simpler and performs all its internal calculations using float32. Although the bfloat16 kernel is faster, to maximize precision, the RWKV-7 fp32 kernel was used to train the RWKV-7 World models.
37

---

Published as a conference paper at COLM 2025
Our CUDA kernels are tuned for head dimension 64, as used in the RWKV-7-Worde models. The kernels still perform well for head dimension 128, but their efficiency drops off at larger head dimensions. There exist other RWKV-7 implementations which focus head dimensions greater than 128. A key example is the Flash Linear Attention library (Yang & 7hang, 2024), which offers a Triton-based implementation designed for these larger configurations.
1.I Speed
In Figure 3, we time the forward + backward pass of each kernel for batch size 8, head dimension 64 and model dimension 4096 (64 wkv heads) on an H100 SXM GPU, for varying sequence lengths. Although Flash Attention v3 is heavily optimized for the H100 GPU, it scales quadratically with sequence length, while the RWKV models scale linearly. This makes the RWKV models faster than attention for large sequence lengths. Furthermore, the optimized RWKV-7 kernel is about three times faster than the official RWKV-6 kernel.
The forward pass of RWKV-7 is about twice as fast as the backward pass. For inference, the forward pass does not need to store the wkv state, making it faster. For example, for sequence length 16k, the forward pass without storing state takes 7.9 ms, while the forward pass with storing state takes 11.2 ms, the backward pass takes 22.5 ms, and the Flash Attention v3 forward pass takes 33.9 ms.
1.2 Memory
The peak training memory usages of the tested models are well described by the formulas derived below. For example, the runs in Figure 3 required peak memory within 2% of the estimates.
In the tested kernels, the memory required per stored variable (e.g. q. k, or in attention)
batch size model dimension x sequence length 2 bytes for bfloat16.
For sequence length 1024, this is 64MB per variable. To calculate memory usage, we may use Flash Attention v3: 10 variables, RWKV-6: 10 variables, RWKV-7: variables, RWKV-7 fp32: 24 variables.
Flash Attention v3 requires 4 variables q, k, v and output for the forward pass, and the correspond- ing 4 gradients. Finally, the backward pass uses a temporary variable to accumulate the gradient of in float32, yielding 2 variables worth of addition memory, for total of 4+4+2 10.
RWKV-6 requires variables r, w, k, v, and output for the forward pass and also the corresponding 5 gradients for the backward pass, for total of 10.
RWKV-7 uses 7 variables the forward pass (r, w, k, v, -k, k and output), and the corresponding 7 gradients. Additionally, it stores the wkv state every 16 timesteps. At head size 64, the state contributes the equivalent of 4 variables, for a total of 18. "RWKV-7 fp32" has the same 14 forward variables and gradients, but uses more memory store the states in float32, for a total of 24 variable equivalents.
Memory usage is constant for single token inference and follows the formulas above, minus the gradients and state storage. Pre-fill can easily be accomplished a chunked manner, with memory usage growing linearly with regard chunk size. This allows easy trade-off for fast parallelized pre-fill, with user selectable maximum memory usage.
J Extended Evaluations
J.I Active Parameters versus Average Benchmark Accuracy
In Figure 7 we plot active parameters versus accuracy averaged across series of multilingual and English benchmarks.
J.2 Associative Recall
Associative recall (AR) (Arora et al., 2023) evaluates the ability of the model to recall previously encountered information within a given context. Research indicates that a model's capacity for AR can reflect its effectiveness in learning from context (Elhage et al., 2021; Olsson et al., 2022).
38

---

Published as a conference paper at COLM 2025
65
Qwen2.5 7B
RWKV7-Warld3 2.9B
60
7B
RWKY7-World3
2.93
3B
1.58
70
1.5B
Average Accuracy (%)
.58
55
5wen2.5
3B
1.5B
SmolLM2
0.4B
SmolLM2 1.7B
360M
0.58
.47B
2.7B
50
0.5B
0,48
- 1..3
421M
: 780M
0.19B
360M
135M
370M
135M
50
0.198
RWKY7-Pile
Mamba2-Pile 130M
40
1E+8
5E+8
1E+9
5E+9
1E+8
5E+8 1E+9
5E+9
Inference Active Parameters (log scale)
inference Active Parameters (log scala)
Figure 7: Active params average benchmark accuracy for Multilingual (left) and English (right). Multilingual benchmarks omit Pile models due to a lack of multilingual training data in the Pile.
Consequently, AR has become standard benchmark for developing new architectural designs in language models (Fu et al., 2023; Poli et al., 2023; Lutati et al., 2023).
In Table we train two-layer RWKV-7 with MQAR, scaling the sequence length up to 2048, with RWKV-7 specific initialization, weight decay 0.1 on weight matrices, and AdamW e of 1 18 18 for stabilization. Interestingly, with only WKV size of 8192, RWKV-7 able to re-
Dim
WKV state dim
(64, 4)
(128,8) 8)
(256, 16)
(512, 64)
(1024, 128)
(2048,
8192
98.43
95.01
72.93
128
16384
v
V
94.97
256
32768
v
98.97
512
65536
Table RWKV-7 MQAR test results. (a, b) denotes sequence length and KV pair count, respec- tively. v means over 99% accuracy, Results maxed over different learning rate settings.
call 72.93% at the setting of 256 Key-value pairs. This suggests that a total of roughly 256 x 0.7293 x log, (number of key tokens number of value tokens) 186.2 log, (4096) 4480.8 bits of in- formation, is stored in 8192 dimensional state, yielding an information density of 0.547 bits per dimension.
J.3 Fresh Internet Text Evaluation (extended)
We collected new data created after January 2025, including: newly submitted computer science and physics papers on arXiv, newly created Python/C++ open-source repositories on GitHub, recently published Wikipedia entries, new fiction Archive of Our Own (Various, 2025), and recent news articles. Inspired by Delétang et al. (2024); Li et al. (2024b), we used compression rate as evaluation metric. See Table 15 for details.
Remarkably, despite being trained on significantly less data than other top models, RWKV-7 Goose showed competitive performance on this temporally novel data.
J.4 Long Context Experiments
To evaluate the ability of RWKV models retain information over long sequences, measured loss versus sequence position (we select tokens in range [L/2 - 16384, L/2 16384) for document length I.) the PG19 test set (Rae al., 2019) for
types of RWKV7 models and their predeces-
39

---

Published as a conference paper at COLM 2025
arXiv CS
arXiv Phys.
Github Python
Github
A03
BBC news
Wiki
Model
C++
Eng
Eng
average
Qwen2.5-1.35
8.12
8.65
4.42
4.40
11.76
9.58
9.49
8.06
RWKV-7 1.5B
8.25
8.77
5.57
5.29
10.93
9.34
8.97
8.16
Llama-3.2-1B
8.37
8.76
5.18
5.16
11.69
9.34
9.07
8.23
SmolLM2-1.775
8.38
9.04
5.17
4.94
11.20
9.40
9.46
8.23
Index-1.98
8.34
8.59
5.65
5.29
11.49
9.51
9.23
8.30
stablelm-2-1.6b
8.58
9.08
5.54
5.45
11.42
9.24
9.06
8.34
RWKV-6 1,5B
8.62
9.00
6,06
5.80
11.09
9.57
9.30
8.49
RWKV-5 1.5B
8.77
9.11
6.20
5.92
11.25
9.75
9.50
8.64
mammba2-1.38
8.74
8.74
6.32
5.71
11.63
9.74
9.86
8.68
MobileL.LM-1.55
8.82
9.29
6.79
6.29
11.59
9.15
9.22
8.73
mamba- 1.4b-h
8.88
8.86
6.43
5.81
11.70
9.83
9.97
8.78
7amba2-1.28
8.57
6.91
7.08
11.39
9.38
9.26
8.83
SmolLM- L.7B
8.38
9.02
5.76
6,55
12.68
9.85
9.89
MobileLLM-1B
9.03
9.57
7.03
6,53
11.86
9.35
9.43
8.97
RWKV-4 1.5B
9.34
9.80
6.54
6.16
11.33
10.00
9.82
9.00
pythia-I.4BB 1.4b-v0
9.12
9.20
6.79
6.15
12.19
10.20
10.43
9.15
Falcon3 IB-Base
8.60
9.20
6.92
7.16
13.04
10.45
10.75
9.45
3.2-3B
7.78
8.10
4.15
4.59
10.90
8.70
8.28
7.57
Qwen2.5 5-3B
7.79
8.25
4.15
4.12
11.23
9.15
8.96
7.66
RWKV-7 2.9H
7.90
8.34
5.16
4.88
10.48
8.92
8.47
7.74
hablelm-3b-4ed 4elt
8.15
5.28
4.85
10.89
8.82
8.51
7,86
Minitron-4B-Bant
8.09
8.70
5.13
4.74
11.05
9.08
8.90
7.96
recurrentgemma-2b
8.24
8.52
5.22
4.80
11.30
8.94
8.88
7.99
RWKV-63B
8.27
8.58
5.66
5.39
10.67
9.17
8.82
8.08
gemma-2-2b
8.39
8.81
5.36
5.01
11.35
8.90
9.03
8.12
manba2attn-2.7b
8.33
8.29
5.78
5.22
11.13
9.28
9.26
8.18
RWKV-5 3B
8.42
8.70
5.78
5.51
10.83
9.36
9.00
8.23
mamba2-2.7b
8.43
8.37
5.93
5.34
11.21
9.37
9.38
8.29
Zamba2-22 -27BB
8.17
8.70
6.30
6.39
10.97
8.95
8.74
8.32
mamba-2.8b-he
8.57
8.52
6.03
5.46
11.31
9.49
9.53
8.41
RWKV-4 3B
8.90
9.27
6.07
5.67
10.90
9.57
9.30
8.53
pythia-2.8b-vD
8.72
8.73
6.29
5.71
11.66
9.74
9.82
8.67
Table 15: Compression rate (unit: %) compared across different language models on various data sources, including arXiv papers, GitHub repositories, AO3 fiction, and news articles cre- ated after January 2025. We define compression rate as: (log2(e) average loss over document document tokens)/B document length bytes).
sors trained on either The Pile dataset or World dataset. Despite sharing the same architecture and being pretrained on 4k context windows, models trained different datasets exhibited different behaviors. The Pile-trained RWKV7 showed more significant loss reduction on long contexts compared its predecessors, demonstrating effective long- contect extrapolation (see Figure 8). Surprisingly, for RWKV7 trained on the World dataset, when processing contexts longer than 10k, the loss began show increasing trend (see Figure 9). We speculate this is because the larger dataset and model size created inductive biases that caused overfitting to specific context lengths. Further experiments showed that fine- fine-tuning on long contexts can restore its long context capabilities.
To further RWKV-7 long-context retrieval abilities, we conduct pass-key retrieval evaluation following the approach of (Chen et al., 2024b) and plot the results in Figure 10. In this evaluation, single sentence is repeated multiple times within long context window, with a key phrase embedded at different positions. RWKV7-World3-11. 1.5. achieves perfect accuracy up to a context length of 19600 tokens but exhibits degradation beyond 20600 tokens. The larger RWKY7-World3- 2.9B extends perfect retrieval up to 32000 tokens, highlighting the benefits of scaling. However, performance begins to degrade beyond this point.
To explore potential improvements, we fine-tuned RWKV7-World3-11 and RWKV7-World3-2 on packed training sequences length 128k tokens from a specially constructed dataset, which leads further improvements in retrieval accuracy. With this fine-tuning, RWKV-7 (1.5B) reliably retrieves key phrases up to 29k tokens, and degradation is observed only around 40k tokens. RWKV-7 (2.9B) reliably retrieves the pass key up to 30k tokens, and degrades around 50k tokens.
40

---

Published as a conference paper at COLM 2025
3.4
RWKVV Pile
RWKV6 173M Pile
3.3
RWKKV 169M Pile
Mamba2 130M
3.0
2.7
10k
11k
13k
17k
18k
19k
20k
21k
22h
23k
24k
25k
28k
27k
28k
20k
30k
314
32k
Token Positions
Figure 8: PG19 loss versus sequence position for RWKV and Mamba models trained on The Pile datasets.
2.6
RWKV6-3B-World
RWKV5-3B-World
RWEV4-38-World
2.5
RWKV7-2. 98--9
2.4
Average
2.3
2.1
10k
11k
13k
15k
17k
19k
20k
21k
22k
23k
24k
25k
20k
27k
28k
29k
30k
31k
32k
laken Positions
Figure 9: PG19 loss versus sequence position for RWKV7 models and predecessors trained the World dataset.
Our context length extension dataset is comprised of both public and custom sources listed in Table 16. We employed document length- based weighting scheme to prioritize longer contexts during training. To approximate document lengths of 128,000 tokens, we used character counts of 512,000. Documents less than 32,768 characters were assigned a weight of 1.0, while longer documents were assigned linearly increasing weights between 2.0 and 3.0, with cap of beyond 512,000. This method increases the inclusion longer documents to bolster the model's handling of extended contexts while retaining shorter documents for diversity.
41

---

Published as a conference paper at COLM 2025
0
Depth (%6)
Ans. Depth (%)
50
50
100
100
Ok
21k
42k
62k
42k
62k
Context Len.
Context Len.
(a) 1.58
(b)
0
Ans. Depth (%6
Ans. Depth (%)
50
50
100
100
0k
21k
42k
62k
0k
21k
42k
62k
Context Len.
Context Len.
(c) 1.5B extended
(d) 3B extended
Figure 10: RWKV7-World3 pass-key retrieval evaluation
Dataset
Type
Amount
Iclm-baseliner 1.0
Public
25%
fineweb-edu
Public
15%
fineweb
Public
5%
codeparrot/ github-cohhhooo
nodeparrot/github-cool arXiv-CCO-v0.5 SuperWikiNEXT-322
Public
10%
Custom
10%
Custom
10%
public domain books
Custom 15%
the-stack (filtered)
Custom 10%
Table Context length extension dataset components
J.5 Evaluating State Tracking Using Group Multiplication
We adopt the experimental setting from Merrill et al. (2024) to evaluate the state- tracken capa- bilities of RWKV7 in comparison to Transformer, Mamba, $4, and classical RNN models. Given
42

---

Published as a conference paper at COLM 2025
sequence gu.g1,
82....8w 82..2....... drawn from As, xz, or Zso, each step i labeled with the cumulative product of the first elements.
We plot the minimum number of layers required to achieve over 95% validation accuracy group multiplication tasks, as function of sequence length and group structure. The results are shown in Figure 11. Our findings indicate that RWKV-7 exhibits stronger state- tate-tracking capabilities than Transformers, Mamba, and S4, though slightly weaker than classical RNNs. Figure also aligns with our theory from Appendix D.2, which predicts that RWKV-7 can perform state tracking and recognize any regular language with a constant number of layers. RWKV-7 has no expressivity advantage for state tracking compared to classical RNNs, which can recognize any regular language in single layer. However, classical RNNs, while being theoretically expressive, typically suffer from gradient vanishing and memorization problems (Zucchet Orvieto, 2024) and cannot be parallelized efficiently, unlike RWKV-7.
(Classical) RNN
S4
Mamba
Transformer
RWKV7
Zo
As
Min. # Layeers
20
Sequeace Leagth
Sequeace Leugth
Sequeace Length
TC
NC1
Figure 11: Minimum number of layers (lower is better) required attain 95% validation accuracy on group multiplication problems by sequence length and group.
K
Multimodal Experiments
In this section, we explore the capabilities of Goose when extended handle multimodal tasks, where the model processes and integrates textual inputs with inputs from a different domain.
K.1
RWKV for Image Understanding
LM Head
Input Image
Vision Encoders
SigLIP
DINOv2
SAM
Image
Embedding
MLP with Context Gating
what the name this bird?
Concat
The architecture of VisualRwKV-7. The input image is processed by three vision encoders, and the obtained features concatenated. Afterward, they projected through an MLP with context gating to align with the dimensions of the RWKV-7 block. Finally, the image features are concatenated with the text embeddings and fed into the RWKV-7 LLM.
Figure 12:
43

---

Published as a conference paper at COLM 2025
To demonstrate the modeling capabiliitess of RWKV-7, we constructed VisualRWKV-7 (Figure12), visual language model based on the RWKV-7 block, to evaluate the image understanding capa- bilities of RWKV-7. VisualRWKV-6 (Hou al., 2024) used the CLIP encoder, which focused on processing low-resolution images and achieved good results. VisualRWKV-7 replaces the CLIP encoder with IPgLIP and DINO visual encoders, and introduced a new high- nigh-resolution SAM vision
encoder, which enhances the model's supported resolution to 1024 x 1024.
Method
Vision Encoder
LLM
VQA
SQA
TQA
GQA
VisualRWKV-6
SigLIP+DINOVV+SAM
RWKV6- 1.68
73.6
57.0
48.7
58.2
VisualRWKV-6
SigLIP+DINOV2+SAM
RWKV6-3.1B
79.1
62.9
52.7
61.0
VisualRWKV-7
SigLIP+DINOV2-SAM
RWKV7-0.18
75.2
50.6
37.9
59.9
VisualRWKV-7
SigLIP:DINOv2+SAM
RWKV7-0.4B
77.9
55.0
41.1
62.3
VisuaIRWKV-7 SigLIP+DINOV++SAM RWKV7-1.5B
79.8
59.7
49.5
63.2
VisuaIRWKV-7 SigLIP-DINOV++SAM
RWKV7-2.9B
80.5
63.4
58.0
63.7
Table 17: A comparison of VisualRWKV-7 to other Visual Language Models across 4 distinct benchmarks. We evaluate these models on benchmarks: GQA(Hudson Manninn, 2019), SQA(Lu et al., 2022), TQA(Singh et al., 2019) and VQA(Li et al., 2023b).
The experimental results of VisualRWKV-7 are shown in Table 17. The vision encoders used in both VisuaIRWKV-7 and VisualRWKV-6 are identical, and the training data remains consistent, aligned with the training data of LLaVA-1.55 The first stage consists of 558k alignment data, while the second stage includes 665k SFT data.
VisualRWKV-7 0.1 and 0.4B outperform VisualRWKV-6 1.6B on the in-domain benchmarks VQAv2 and GQA and rapidly approach VisuaIRWKV-6 1.6B results are highly compelling. With only 1/4 of the parameters (1.6B vs. 0.4B), VisualRWKV- 7 surpasses VisuaIRWKV-6 the VQAv2 and GQA benchmarks, demonstrating the powerful modeling capabilities of RWKV-7.
other benchmarks. The experimental
On the out-of-domain benchmark SQA, VisualRWKV-7 2.9B also outperforms VisualRWKV-6 3.1B, indicating that VisualRWKV-7 possesses strong generalization ability. In the TexIQA (TQA) benchmark, which assesses a model's associative recall, VisualRWKV-7 2.9B achieves 5.3 5.3-po improvement over VisualRWKV-6 3,1B, further proving its superior associative recall capabilities.
K.2 RWKV for Audio Modeling
To investigate the effectiveness of RWKV-7 for audio modeling, we introduce AudioRWKV-7, novel adaptation of RWKV-7 for audio embedding analysis. We use bi-directional modification to RWKV-7, similar to the modification used by Duan et al. (2024). We employ this approach to interpret and process complex, high- nigh-ddimnsiooal spectrogram features. To further capture acoustic and temporal characteristics, an spectspectrogram is divided into patch tokens using a Aatch-Embed CNN with kernel size of (Px P) and sequentially fed into the model. The width and height of an audio mel-spectrogram represent the time and frequency bins, respectively. Typically, the time dimension is significantly longer than the frequency dimension. To effectively capture relationships among frequency bins within the same time frame, the mel-spectrogram is first segmented into patch windows W1, W2..... Wn, followed by further division of patches within each window. The token sequence follows the order: time frequency- - window. This arrangement ensures that patches corresponding to different frequency bins within the same time frame are positioned adjacent to each other in the input sequence.
evaluated the performance of Audio across multiple model scales and architectures, using the AudioSet dataset (Gemmeke et al., 2017). Detailed experimental outcomes are presented in Table 18. From the results we find that our model achieves comparable performance with much smaller parameter count when compared with CNN, Transformer and Mamba based architectures, and exceeds the performance of AudioRWKV-6 These findings demonstrate the robustness and versatility of AudioRWKv. Note that to ensure a fair comparison, we retrained AudioRWKY-6 without ensembling models with different patch settings, which accounts for the difference in results from (Peng al., 2024a).
44

---

Published as a conference paper at COLM 2025
Model
#Parameters 26M
Architecture
MAP [
DeepRes (Ford al., 2019)
CNN
0.392
HST-AT
88.5M
Transformer
0,433
HST-AT pretrained(Chen al., 2022)
88.5M
Transformer
0.429*
MambaOut(YYu Wang, 2024)
101.3M
Mamba
0.397
AudioRWKY-6
8.9M
RWKV6
0.381
AudioRWKV-6
19.8M
RWKV6
0.426
AudioRWKV-7
8.9M
RWKV7
0.392
AudioRWKV-7
19.8M
RWKV7
0.431
Table 18: A comparison of mean Average Precision (mAP) among AudioRWKV7 and other baselines on AudioSet dataset. HST-AT pretrained a variation that uses vision transformer to initialize the weights. 'Results reproduced by ourselves.
Board Game Modeling
Previous research (Schultz et al. (2024); Topsakal et al. (2024)) has shown that board games can serve as potentially effective tools for evaluating model's capabilities. As an RNN with powerful state tracking abilities, RWKV-7 is highly suitable for board game modeling and conducting extended searches directly within context to find better strategies. As an early exploration, we tested RWKV-7's board game modeling capabilities and its ability perform in-context search the game of Othello (Reversi). We found an expanded formulation of RWKV-7 to be useful for this task, which we designate RWKV-7a: S, S, diag(w,) ck (a, k2)) + v k, This formula allows the full range of (-1,1) eigenvalues when c= 2.
Data We designed training data that guides the model predict legal moves, evaluate these moves, and perform Alpha-Beta pruning. Each training sample consists of three components:
Input section:
Captures the game state, including the current board position, active player, and search parameters (tree depth and width settings).
Reasoning section: Varies with search settings:
Wiihhut search (depth width 1): Lists legal moves and their evaluations
With search (depth and width 1): Performs Alpha- pruning find optimal
moves
All move evaluations in the reasoning section were generated using the Egaroucid engine
(Yamana, 2025).
Output section: This contains the final move decision and displays the resulting board position after implementing the move.
An example of our training data is shown in sample
Code Listing 1: Training Data Example
<input>
NEXT O
MAX WIDTH-2
MAX DEPTH-2
</input>
<reasoning>
Possible moves and score: gl -19 hl
-01
b2
-08
h2
-23
b7
-12
g7
-09
<stack>
45

---

Published as a conference paper at COLM 2025
Remaining Depth:2
Max Node Alpha: -in Beta: +in Best: -- Current: -01 Unexplored: b2 -08
</stack>
=> Search next node
[Depth limit not reached] <board>
</board>
NEXT
Possible moves and score: bl +02 b2 +05 h2
+10 h6 +03 h7 +08 c8
+06 d8 +01 g8 +09
[Current player has legal moves]
[Internal node expand)
<stack>
Remaining Depth:1
Min_Node Alpha: in Beta: +in Best:
Current: d8 Unexplored: bl +02
Max_ Node Alpha: in Beta: +in Best: Current: hl -01 Unexplored: b2 -08
</stack>
Search next node
(Depth limit reached evaluate all leaves]
(Updated stack]
<stack>
Remaining 1
Min_Node Alpha: -in Beta: +01 Best: d8 Current:
Unexplored: Node Alpha: +01 Beta: +in Best: hi Current: b2 -08 Unexplored:
</stack>
=> Search next node [Depth limit not reached] <board>
</board>
NEXT
Possible moves and score: al -07 bl +13 h6
-01 b7 -08 g7
+08
h7
-02
d8
-03
g8
[Current player has legal moves]
[Internal node expand]
<stack>
Remaining_Deptte
Min Node Alpha: +01 Beta: +in Best: Current: b7 -08 Unexplored: al -07
Max Node Alpha: +01 Beta: +in Best: hl Current: b2 -08 Unexplored:
</stack>
=> Search next node
[Depth limit reached evaluate all leaves
[Updated stack]
<stack>
Remaining 1
Min Node Alpha: +01 Beta: -08 Best: b7 Current:
Unexplored:
46

---

Published as a conference paper at COLM 2025
Max Node Alpha: +01 Beta: +in Best: h1 Current:
Unexplored:
</stack> [End of search] Playing h1
</aasoning>> <output> h1
</output>
Training We trained RWKV-7 models with 9M and 26M parameters respectively on 6 million samples. By tracking the loss across different token types during training (figure 13), noticed that the model first mastered output formatting, then developed board state tracking capability, and continuously improved its evaluation accuracy throughout training.
Loss Trencs During Training
50002 trocking LOGS
Dog Vilse Loce.
Avg Tormat Loss
Los Vélues
02
01
Trining Progress
Figure 13: Reversi training loss for different token types
We control the model's thinking budget by setting the width and depth of Alpha Alpha-beta beta pruning, and test the win rate against baseline model (depth=1, width=1) under different budgets. As shown in figure 14, by increasing the testing budget, RWKV-7 can effectively search for better strategies, demonstrating positive test-time scaling law this task.
Evaluation
47

---

Published as a conference paper at COLM 2025
0.85
d-4 b-4
0.80
d
d=4
bm2
bm3
d=2
d=2
dm3
b
d-3
bm4
Win Rate
0.70
d=2
bm2
0.65
0.60
b=2
Test-time tokens
Figure 14: Reversi token consumption and win rates under different search configurations
M Ablation Experiments
M.1 The Pile
To demonstrate the architectural advantages of RWKV-7, we conducted ablation experiments by training models three different sizes 168M, 421M, and 1.47B parameters on the full Pile dataset (Gao et al., 2020).
Model
Tokens (B)
Imb.o ppl
Imb.o acc
hella
piqa acc
arcE
arcC acc
glue acc
WG
sciq acc
avg
acc_n
acc
acc
acc
WWVKV-169M-PPM
332
29.2
33.2
32.2
64.8
47.1
19.9
47.6
51.2
77.6
46.7
P60hia-160M
300
37.3
35.4
30.3
62.3
43.6
19.5
46.5
51.3
75.1
45.5
Mamba-130M
300
16.0
44.3
35.3
64.5
48.0
19.7
48.5
52.1
78.2
48.8
Mamba2-130M
300
16.8
43.9
35.3
64.9
47.4
20.9
45.8
52.6
81.0
49.0
WWKV6-173M-PPM
332
16.0
44.5
34.9
64.4
48.3
19.7
48.9
51.9
80.6
RWKV7-168M-Pile
332
45.7
36.9
65.5
47.9
19.7
49.1
52.4
81.6
49.8
RWKV4-430M-PPM
332
13.1
45.1
40.8
67.7
52.8
24.1
49.4
52.0
80.7
51.6
Pythia-410M
300
40.6
66.7
51.9
21.4
44.1
53.3
81.5
Mamba-370M
300
8.1
55.6
46.5
69.5
55.0
25.0
55.5
84.5
54.8
Mamba2-370M
300
8.0
55.9
46.9
54.8
25.1
48.1
55.4
85.3
55.2
RWKV7-121M-Pile
332
7.2
57.9
48.0
69.3
56.3
23.5
50.3
56.4
85.9
56.0
RWKvie1.5B-Pile
332
7.1
56.4
52.8
60.7
24.9
50.5
54.3
85.8
57.2
Pythia-1.4B
300
6.1
61.7
52.0
70.8
60.5
26.1
47.7
57.5
86.6
57.9
Mamba-1.44 1.4B
300
5.0
65.0
59.1
74.2
65.5
29.8
46.2
61.4
87.3
61.1
Mamba2-1.3B
300
5.0
65.6
59.9
73.2
64.2
29.9
46.1
61.0
89.8
61.2
RWKV7-1.47B-PiG
332
4.8
67.0
61.8
73.6
64.9
30.2
48.0
64.4
91.1
62.6
Table 19: English focused benchmarks, including LAMBADA (lmb.o) (Paperno et al., 2016),
Hellaswag (hella) (Zellers et al., 2019), PIQA (Bisk et al., 2020), A12 ARC (arcE, arcC) (Bhakthavatsalam et al., 2021), GLUE (Wang al., 2018), Winogrande (WG) (Sakaguchi et al., 2021), SciQ (Welbl et al., 2017).
These results highlight the consistent improvements brought by the RWKV-7 architecture earlier RWKV models, even when trained on the same dataset. As all RWKV models shown were trained under identical configurations and dataset, this underscores the inherent architectural
48

---

Published as a conference paper at COLM 2025
advantages of RWKV-7 over its predecessors. Notably, the performance gap sustains the model size increases, suggesting that RWKV-7 may scale more effectively than its predecessors.
M.2 Architecture Choice Ablations
We series of ablation experiments validate the various improvements made in RWKV-7 versus some the more restrictive choices seen in other DeltaNet and West-DeltaNet related work.
These improvements were:
using a vector valued decay w instead of a scalar-valued decay;
using vector-vale vallud in-context learning rate instead of scalar- scalar-valued rate;
using different removal and replacement keys, instead of the same for both; and adding the bonus term Urj to the output step (Equation 20).
We trained a small 6-layer, dmodes 768 Goose model on the 1.6B token minipile (Kaddour, 2023)
dataset at context length 512 and obtained the loss results shown in Table 20.
Model
Training Loss Validation Loss
Goose
2.834
2.541
Goose, scalar decay
2.873
2.609
Goose, scalar in-context learning rate
2.843
2.591
Goose, same removal/ hetpra///eppacmeennnt keys
2.840
2.560
Goose, no bonus term
2.841
2.588
Table 20; Ablation results for layer 768 dimension Goose model
N State Inspections
We inspected the WKV matrix states of RWKV-7 and compared them with those of RWKV-5 and RWKV-6. We analyzed the following aspects:
1. Visualization example of WKV state matrices, better understand the structure and
behavior of the matrices.
2. Root Mean Square (RMS) of matrix elements, assess the numerical stability of those WKV matrices.
3. Stable Rank (SR) of the matrices (Rudelson & Vershynin, 2007), defined as the square of (the Frobenius norm divided by the spectral norm):
||Al|
SR(A)
||lll
This serves rough measure to the effective amount of information of the states.
For this analysis, we selected 10 samples from the validation set of the PG19 dataset (Rae et al., 2019), ensuring that each sample had sequence length of at least 8192 tokens. We tested the 1.5B parameter versions of RWKV-5, RWKV-6, and RWKV-7, plotting on the appearance, stable rank and RMS values of their WKV matrices.
We observed that the WKV states of RWKV-7 had significantly smaller RMS values compared to RWKV-5 and RWKV-6. The entries of the WKV matrix in RWKV-7 were consistently of order O(1) (i.e., no outliers, and does not grow over context length), whereas RWKV-5 and RWKV-6 constantly produced outliers on the order thousands (see Figures and 16). This indicates that RWKV-7 has better numerical stability during training and inference.
Interestingly, the stable rank the WKV matrix in RWKV-7 has shown to be lower than that of RWKV-5 and RWKV-6 for context longer than 32. A lower stable rank typically suggests that the matrix contains less information or has a more compressed representation. However, this observa- tion appears to contradict the experimental results showing that RWKV-7 performs better tasks requiring long- 7's enhanced state evolution mechanism, which enables RWKV-7 to achieve stronger information
long-term term memory. We hypothesize that this contradiction can be explained by RWKV-
49

---

Published as a conference paper at COLM 2025
Head SR: 2.00 RMS
12 Head SR:
Layer
23
10-
200
32-
22
-00
50-
-26
52-
-200
60
20
46
54
10
50
1.
56
(a) RWKV-5 1,5B
Layer MC0d4, 5R 175, KVS: 446.71
Layer 12 Hozd SR. 1.55, RMS 8.39
Layer 22 Hoad 22, 5R: 107, RMS. 5.79
200
10
-300
50
1022
10
93
20
10
(b) RWKV-6 1.6B
Layer Head 1.47, RMG
12 Head 12,
21 Head SR: RMS:
10
10-
10-
20-
-2.00
40-
-3.0
20
23
26
50
10
(c) RWKV-7 1.5B
Figure 15: Visualization example of RWKV's WKV matrices.
RMS RWKV State Entrics
Avarage Stable Rank of WKV Matrices
RWACVS 1.58
1.5
HWWVV--1. MM
RWWC/7-1.50
2.5
10a
1.4
R385
HAKVS-150 RAKVE-1.48
Averaye State
2.3
RAKV7 1.58
10P
111
10-
125
10242044 4094
12a 25s 10042249400
Figure 16: The global RMS and average stable rank of WKV matrices, plotted over sequence length.
compression and utilization capabilities and allowing it to maintain important information in more compact form. This dual capability likely contributes to both the reduced stable rank and the improved performance memory-intensive tasks.
50

---

Published as a conference paper at COLM 2025
0 Parameters Statistics
In Section 4, the actual ranges and statistical metrics the parameters within the trained model not specified. To facilitate better understanding of the role of these parameters in the practical models, this appendix provides empirical statistical metrics of selected parameters from the released RWKV-7 model.
Q
1.2
0.8
1.1
0.7
1.0
0.6
0.9
0.5
0.1
0.4
1.5
2.9
0.1
0.4
1.5
2.9
Parameters(B)
Parameters(B)
Figure 17: Box plots of and across models
0.8
0.6
0.1B
0.4B
10
20
30
1.5B
1.2
2.9B
a
1.0
0
10
20
30
Layer Index
Figure 18: Mean and a across layers for different models
P Initial Token Sensitivity
In our evaluation of LAMBADA (Paperno et al., 2016), we found that RWKV-7's performance varies significantly under different settings. After ruling out precision issues, investigated the
51

---

Published as a conference paper at COLM 2025
7.5
Max
5.0
0.1B
0.4B
1.5B
2.9B
-2
Min
10
20
30
Layer Index
Figure 19: Maximum and minimum of across layers in different models
15
Max
10
5
0.1B
0.4B
1.5B
0
2.9B
10
-20
0
10
20
30
Layer Index
Figure 20: Maximum and minimum of across layers in different models
consistency of the input and discovered that omitting the special token < endooftext > at the beginning of the input caused substantial and statistically significant differences in perplexity (PPL) and accuracy (ACC) for some RWKV-7 models.
Previous research highlights that Transformer models sensitive to special tokens, and fine- tuning these tokens can yield notable improvements (Yang et al., 2023b). However, to our knowl- edge, no quantitative study has examined this effect systematically.
We analyzed the cases with the largest performance discrepancies, and identified a key pattern:
The answer appears as the first word of the paragraph. This first word does not reappear elsewhere in the text.
This suggests that the model may struggle retain the first token in memory.
52

---

Published as a conference paper at COLM 2025
-25
3.0
3.5
4.0
4.5
5.00
5.5
0.1
0.4
1.5
2.9
Parameters (B)
Figure 21: Box plots of biases of d, across models
-2.5
-3.0
0.1B
3.5
0.4B
4.0
1.5B
2.9B
4.5
-5.0
-5.5
0
10
20
30
Layer Index
Figure 22: Mean biases of d, across layers for different models
In the LAMBADA test set, we identified 142 such examples out of a total of 5153 questions. One example is:
Beth smoothed her wiry half-black, half-gray hair from her makeup-free face. In New Mexico, the natural look common. Standing disaster. She hid her ragged nails under the sleeves of her sweatshirt. "Hi, I'm Cindy. It's nice meet you, Beth."
Cindy Fanucci, she felt like
The following table summarizes the performance of different models with and without the padding token endo text
53

---

Published as a conference paper at COLM 2025
7.5
Max
5.0
2.5
0.1B
0.4B
1.5B
2.9B
-10
-15
-20
10
20
30
Layer Index
Figure 23: Maximum and minimum of biases of d, across layers in different models
Model
EOS padding
PPL
ACC (%)
Significance
RWKV7 World 0.1B
0
357
9.2
16.4
36.6
RWKV7 World 0.4B
42.7
28.9
7.25
48.6
SmolLM2 360M
21.1
39.4
1
9.17
49.3
Qwen2.5 0.5B
0
12.2
47.9
NS
7.97
54.9
Table 21: Performance comparison of different models with and without the </endoftett |> token in the partial set 142 samples from LAMBADA. Significance levels: <.05, "'p 0.01, ..- 0.001, NS Not Significant.
The results indicate that the inclusion the </endo token improves the performance of RWKV-7 very significantly, especially for small models (e.g., RWKV-7 O. This finding high- lights the importance of proper state initialization and context context setting for RNN-based architectures like RWKV. Unexpectedly, we also found that two consecutive < lendoffext >> tokens at the beginning can further improve the performance of RWKV-7, despite that endoftext: >> never appears consecutively in the training corpus.
However, for Transformer baased models such as Qwen2.5-0.5B, we observe that the impact of the <| endo exx token is less pronounced, suggesting that these models may have better mechanisms for attending to the initial token.
As result, for optimal performance when prompting RWKV-7, recommend always including the < endo ftext token at the beginning of the prompt. For example, if you plan to use RWKV-7 chat assistant, consider the following structured prompt format:
54

---

Published as a conference paper at COLM 2025
</endoftext >Üser:
<Your Question>
Assistant:
<Assistant Answer>
User:
<Another Question
Assistant:
Limitations
Despite its strengths, the RWKV-7 architecture and models face certain limitations yet to be mitigated in future work.
Numerical Precision. We observed that some operators, particularly the WKV7 kernel, are sen- sitive to the numerical precision of the implementation. This highlights the need for careful handling of numerical precision during model deployment. We also observed differences in train- ing dynamics when using different kernels, which implies that the correct handling of precision while calculating and applying state updates is of utmost importance in this architecture.
Lack of Instruction Tuning and Alignment. All RWKV-7 models presented in this work are pretrained base models and have not undergone the phase of Supervised Fine-Tuning (SFT) for instruction following nor alignment with human preferences (RLHF). Future efforts should focus incorporating these capabilities to enhance the model's usability in real-world applications.
Prompt Sensitivity. We found that the absence of the special token < endo endoftext fextt | degraded performance of RWKV-7 models, e.g. inability remember the first token of the input. See Appendix P for details.
results in
Compute Resources. Due to computational budget constraints, our training was limited on at most 12 x 8 96 Nvidia H800 GPUs. This falls short of the resources required for recent large-scale training efforts, such as DeepSeek-V. V3 (DeepSeek-A) Al et al., 2025). Additionally, we are forced to continue training from pre- pre-existing checkpoints of earlier RWKV architectures and therefore re-use some parts of our dataset. This may limit the capabilities of our models versus pretraining from scratch. Scaling up RWKV-7 larger sizes and datasets will require additional computational resources.
R Future Work
In addition to training larger RWKV-7 models with more tokens the future, we also aim to explore several promising directions to further enhance the architecture and its capabilities.
Speedup Techniques. variety of speed optimization techniques were highlighted in the tech- nical report DeepSeek-V2 (DeepSeek-Al al., 2025), including Dual Pipelining Mechanism, cetture-of-Experre Multi-Token Prediction, and FP8 Training. We are aware that many of these techniques are orthogonal to RWKV-7's architectural optimizations, therefore could integrated to further accelerate training in later RWKV models. However, RWKV-7 has been trained com- pletely without pipeline parallelism. We also noticed that there is room for speed optimization of RWKV-7 kernels and operators. We will explore both kernel-level optimizations and distributed training strategies the future.
Incorporating Chain-of- Thooughht Reasoning. We believe that RWKV-7, as linear RNN, is well- suited for efficient Chain-od of- Thought reasoning (Wei et al., 2022). However, this capability has been barely explored due to the lack of suitable reinforcement learning pipelines. In future work, we plan to incorporate deep thinking abilities into RWKV-7, enabling it to excel in tasks requiring multi-step logical reasoning and complex problem-solving.
55


Reviews:
[{"review_id": "4qsfmPudim", "reviewer": "Reviewer_bdD1", "summary": "RWKV-7 Goose introduces a new sequence modeling architecture that achieves state-of-the-art performance at the 3B parameter scale for multilingual tasks while matching SoTA on English language tasks, despite being trained on fewer tokens than competing models. \n\nThe architecture features constant memory usage and inference time per token, making it more efficient than Transformer models for long sequences.\n\nKey contributions includes:\n\n- A generalized formulation of the delta rule with vector-valued gating and in-context learning rates\n\n- A relaxed value replacement rule that allows selective state updating\n\n- Improved state tracking capabilities that can recognize all regular languages\n\n- Open-sourced models and recipes to the community. \n\nThe paper demonstrates that RWKV-7 has expressivity that exceeds TC complexity class under standard complexity conjectures, which is a theoretical advantage over Transformers.  But I am not so sure or confident about this conclusion, which I would encourage authors to elaborate further.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Novel Architecture: The paper presents a significant evolution of linear RNN-based architectures through the generalized delta rule, offering a compelling alternative to Transformers with linear complexity.\n\n- Strong Empirical Results: RWKV-7 achieves state-of-the-art performance on multilingual tasks at 3B scale despite being trained on fewer tokens than competitors, demonstrating impressive parameter efficiency.\n\n- Computational Efficiency: The architecture offers constant memory usage and inference time per token, addressing key limitations of Transformers for long sequences.\n\n- Open Source Contribution: The research provides multiple pre-trained models and a large multilingual corpus, contributing valuable resources to the research community.\n\nI also want to highlight their comprehensive evaluation: The paper includes extensive benchmarking across English and multilingual tasks, long-context understanding, associative recall, and state tracking capabilities, setting good examples for future model architecture design papers for scientific studies.", "reasons_to_reject": "- Numerical Precision Issues: The paper notes sensitivity to numerical precision in certain operators, particularly the WKV7 kernel, which may complicate deployment. But I don't think this belies the strength of the paper.\n\n- Lack of Instruction Tuning: The presented models are base models without instruction tuning or alignment, limiting direct comparability to fully-tuned models in practical applications. But this would be helpful to further understand how this model can be actually used in the production environment.", "questions_to_authors": "In my view, the paper would benefit from:\n\nMoving some of the key theoretical results from the appendix to the main paper\n\nProviding more intuitive explanations of why these complexity results matter\n\nIncluding concrete examples that demonstrate how these theoretical capabilities translate to practical advantages\n\nBetter connecting the theoretical guarantees to the empirical results observed in benchmarks", "extra_scores": {}}, {"review_id": "GD00AP7Khj", "reviewer": "Reviewer_GjhW", "summary": "The paper introduces RWKV-7 \"Goose,\" a new RNN architecture for effective language modeling.  RWKV-7 builds on recent work for the delta rule, thus offering new modeling capabilities not featured in many recent SOTA recurrent models (e.g., Mamba/Mamba-2, RWKV-6, etc.).  Along with this new architecture, the paper also debuts a large-scale (3.1 trillion tokens) open-source corpus supporting English, math, and multilingual task modeling.  A large portion of the paper is dedicated to flushing out the design choices, situating these choices within the context of previous work, and exactly highlighting what contributions are novel.  Finally, the paper ends with a large number of experiments, demonstrating the:\n- speed and memory efficiency of the developed RWKV-7 for modern hardware\n- performance on a large number of standard benchmarks, as well as multilingual benchmarks, demonstrating that RWKV-7 is highly competitive with SOTA attention and RNN LLMs of similar parameter counts (many of which are pre-trained on significantly many more tokens)\n- RWKV-7 is SOTA/highly competitive with other models on the Mechanistic Architecture Design tasks", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written, and the pace and clarity of discussion are appreciated.  The architectural choices are intuitive, well explained, and improve the modeling expressiveness over other recent RNN innovations (which are reflected in the overall results).  The included benchmarks are extensive and comprehensive, and further demonstrate the efficacy of the design choices for RWKV-7.  In particular, though the model is pre-trained on far fewer total tokens than other SOTA attention/RNN models, it remains competitive across a large number of natural language english/multilingual tasks.  Thus, the RWKV-7 design, RWKV World v3 corpus, and actual pre-trained model checkpoints serve as valuable contributions to the community.  There are also significant additional experiments which show the RWKV-7 model's aptitude on recent synthetic tasks (stress testing several sequence modeling capabilities) and compression tasks.", "reasons_to_reject": "There are only minor issues with clarity/missing details (see questions below).", "questions_to_authors": "- \"we expand the in-context learning rate\" <- What is the ICL rate?  Are the authors referring to the context window conditioned on (i.e., previous token for Mamba vs previous n tokens for autoregressive Transformers?).  Addendum after reading: please add a forward reference to where the ICL rate is defined, i.e., \"we expand the in-context learning rate, described in Section...\"\n\n- \"For now, we choose to allow only part of the range of possible negative eigenvalues in our pre-trained large language models due to experimentally observed training instabilities.\" <- Can the authors speak to both the range of values and the types of instabilities encountered?  Might these instabilities be side-stepped and the range of negative values increased during fine-tuning?\n\n- What model is Flash Attention 3 being run on in Figure 3?\n\n- Why is English Lambada perplexity not reported in Table 2?\n\n- Unlikely that it would have placed highly in its weight\tclass, but it would have been interesting to see mamba-2.8b-slimpj in Table 2 and 3.\n\n- To verify, the non-instruct-tuned versions of relevant models are used for comparison in Table 2 and 3,\tcorrect?  If possible, could the authors include an appendix table of the competitor HF checkpoints?\n\n- What are the parameter counts for the models in Table 4?\n\n- Could you please add implementation details for the compression rate metric?  Also, to make the paper self-contained, could you add a description of the compression rate metric used in Table 15.\n\n## Additional comments\n- \"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023.\" <- Attention citation is incorrect\n- \"For short sequences, much of this cost can be covered by modern GPU parallelism techniques\" <- Ideal for Flash Attention citation\n- \"as a RNN\" <- \"as an RNN\"", "extra_scores": {}}, {"review_id": "ZGbnFpitS7", "reviewer": "Reviewer_QvXt", "summary": "This paper introduces RWKV-7 \"Goose,\" a novel sequence modeling architecture that builds on past works on SSMs and Linear Attention and has constant memory usage during inference. In addition to the architectural innovations, the paper also introduces a new 3.1 trillion token multilingual corpus (RWKV World v3). Despite being trained on fewer tokens than comparable models, the 2.9B parameter RWKV7 model trained on the new corpus achieves state-of-the-art performance on multilingual tasks at its size and matches current SOTA on English benchmarks.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a novel RNN-based architecture building on past works on SSMs and Linear Attention. The model has linear complexity in the input length and constant memory usage during inference.\n2. When compared to other models such as Mamba2 trained on Pile under comparable compute settings, the proposed RWKV-7 architecture reports improved performance.\n3. The authors introduce a public 3.1 trillion token multilingual corpus. The RWKV-7 models trained on this corpus demonstrate impressive performance on multilingual benchmarks, achieving SOTA for its parameter count compared to larger models trained on more data. The model outperforms established architectures like Llama-3.2, Qwen2.5, and SmolLM2 on multilingual tasks with fewer training tokens.\n4. The authors provide extensive benchmarking across standard NLP tasks, recent internet data evaluation, image understanding and specialized tests for associative recall and state tracking abilities.\n5. The authors publicly release the pre-trained models and the training corpus.", "reasons_to_reject": "1. The state update mechanism and overall architecture are quite complex, making the model potentially difficult to implement and understand for those not already familiar with linear attention variants.\n2. While some ablations are provided, a more comprehensive analysis separating the contributions of each architectural component would strengthen the paper.", "questions_to_authors": "1. When comparing architectures trained on identical datasets (as in the Pile experiments), what are the specific speed advantages of RWKV-7 over simpler architectures like Mamba2? Are there inference speed or memory usage tradeoffs that would help readers understand when to prefer RWKV-7?", "extra_scores": {}}, {"review_id": "n6LrVxuZz1", "reviewer": "Reviewer_UGzf", "summary": "This paper presents RWKV-7 \"Goose\", a sequence modeling architecture described as having constant memory usage and constant inference time per token. The architecture introduces a generalized formulation of the delta rule with vector-valued gating and in-context learning rates, as well as a relaxed value replacement rule. Regarding empirical results, the 2.9 billion parameter model, stated to be trained on fewer tokens than other top models, achieves a new 3B state-of-the-art (SoTA) on multilingual tasks and matches the current 3B SoTA on English language downstream performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe architecture design of RWKV-7 looks nice and solid.\n\n2.\tThe newly proposed RWKV World v3 dataset could be beneficial to the large language model pretraining community.\n\n3.\tResults shows that RWKV-7 shows comparable results with other transformer-based models while achieving better efficiency.", "reasons_to_reject": "1.\tOne possible concern of the contribution of this paper is the entangled effect of the new model structure design and the newly proposed dataset, that is, the improvement and the comparable results of RWKV 7 could be gone if other models use the same dataset for their pretraining process. There lack ablation study experiments. This is especially important for the claims related to the multilingual performance.\n\n2.\tThere lacks the evaluation on real-world generation tasks (e.g., story generation, stylist generation, role playing, etc.) which requires annotations from human evaluators.", "questions_to_authors": "1.\tCould the dataset proposed RWKV-world3 dataset be used on other architectures? What would be the results?\n\n2.\tFigure 3 could be improved with better resolution and alignment.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel architecture for language modeling called RWKV-7, based on RNNs and yet outperforming similarly-sized transformer and Mamba models on multilingual tasks and showing promise on English tasks. The authors will release their model and the 3.1-trillion token data used to pre-train the 2.9-billion parameter variant.

The reviewers were enthusiastic about the paper, apparently convinced by the strong experimental setup and the impressive results. Minor qualms were answered to their satisfaction; one reviewer who was less supportive asked a more fundamental question but did not engage with the reply which, to me, made sense. It seems like there is no serious concern about accepting this rigorous, useful piece of work.

[Automatically added comment] At least one review was discounted during the decision process due to quality]
Decision: Accept