RWKV: Reinventing RNNs for the Transformer Era
Authors: Bo Peng
Eric Alcaide
Quentin Gregory Anthony
Alon Albalak
Samuel Arcadinho
Stella Biderman
Huanqi Cao
Xin Cheng
Michael Nguyen Chung
Leon Derczynski
Xingjian Du
Matteo Grella
Kranthi Kiran GV
Xuzheng He
Haowen Hou
Przemyslaw Kazienko
Jan Kocon
Jiaming Kong
Bartłomiej Koptyra
Hayden Lau
Jiaju Lin
Krishna Sri Ipsit Mantri
Ferdinand Mom
Atsushi Saito
Guangyu Song
Xiangru Tang
Johan S. Wind
Stanisław Woźniak
Zhenyuan Zhang
Qinghua Zhou
Jian Zhu
Rui-Jie Zhu
Venue: EMNLP 2023 Findings (2023)
Keywords: large language model
scaling laws
open source
pretraining
TL;DR: We define and evaluate a scaling suite of RWKV models, an RNN-based language model with computational benefits compared to transformers

Abstract: Transformers have revolutionized almost all natural language processing (NLP) tasks but suffer from memory and computational complexity that scales quadratically with sequence length. In contrast, recurrent neural networks (RNNs) exhibit linear scaling in memory and computational requirements but struggle to match the same performance as Transformers due to limitations in parallelization and scalability. We propose a novel model architecture, Receptance Weighted Key Value (RWKV), that combines the efficient parallelizable training of transformers with the efficient inference of RNNs.

Our approach leverages a linear attention mechanism and allows us to formulate the model as either a Transformer or an RNN, thus parallelizing computations during training and maintains constant computational and memory complexity during inference. We scale our models as large as 14 billion parameters, by far the largest dense RNN ever trained, and find RWKV performs on par with similarly sized Transformers, suggesting future work can leverage this architecture to create more efficient models. This work presents a significant step towards reconciling trade-offs between computational efficiency and model performance in sequence processing tasks.

RWKV: Reinventing RNNs for the Transformer Era
Bo Pengl.. Eric Alcaide Quentin Anthony
Alon Albalak Samuel Arcadinho Stella Biderman Huanqi Cao Xin Cheng" Michael Chung" Xingjian Du' Matteo Grella Kranthi Kiran GV2.3 Xuzheng He² Haowen Houl4 Jiaju Lin' Przemystaw Kazienko Jan Kocoń Jiaming Kong
Bartlomiej Koptyra'5 Hayden Lau² Krishna Sri Ipsit Mantri" Ferdinand Mom 18,19
Atsushi Saito Guangyu Song") Xiangru Tang" Bolun Wang Johan S. Wind Stanislaw Wožniak Ruichong Zhang Zhenyuan Zhang² Qihang Zhao Peng Zhou Qinghua Zhou' Jian Zhu?? Rui-Jie 28,2 28,29
GGenerative AI Commons "EleutherAl" UU Barcelona "Charm Therapeuties 'Ohio State '.. of Santa Barbara
'Zendesk Booz Allen Hamilton "Tsinghua University Peking University "Storytellerio Criisis2 New York U. Natiooal U. of Singapore "Wroclaw U. of Science and Technology "Databaker Technology "Purdue U. "Criteo Al Lab
Epiita "Nxttemeer Mooes Yale U. ReoxiiTech U of Oslo U. of Science and Technology of China "Kuaishou Technology U. British Columbia UU, C., Santa Cruz U. of Electronic Science and Technology of China
Abstract
processing tasks such as natural language under- standing, conversational AI, time-series analysis, and indirectly sequential formats like images and graphs (Brown al., 2020; Ismail Fawaz et al., 2019; Wu et al., 2020; Albalak al., 2022). Pre- dominant among these techniques include RNNs and Transformers (Vaswani et al., 2017), each with specific benefits and drawbacks. RNNs require less memory, particularly for handling long sequences. However, they suffer from the vanishing gradient problem and non parapallellizabilityy in the time di- mension during training, limiting their scalability (Hochreiter, 1998; Le and Zuidema. 2016).
Transformers have revolutionized almost all natural language processing (NLP) tasks but suffer from memory and computational com- plexity that scales quadratically with sequence length. In contrast, recurrent neural networks (RNNs) exhibit linear scaling in memory and computational requirements but struggle to match the same performance Transformers due to limitations in parallelization and seala- bility. We propose a novel model architecture, Receptance Weighted Key Value (RWKV), that combines the efficient parallelizable training of transformers with the efficient inference of RNNs.
Our approach leverages linear attention mech- anism and allows formulate the model as either a Transformer or an RNN, thus paralleliz- ing computations during training and maintains constant computational and memory complex- ity during inference. We scale our models as large as 14 billion parameters, by far the largest dense RNN ever trained, and find RWKV per- forms on par with similarly sized Transformers, suggesting future work can leverage this archi- tecture to create more efficient models. This work presents significant step towards rec- onciling trade-offs between computational ef- ficiency and model performance in sequence processing tasks.
55
50
Accurscy
45
BLDOM
40
Pythia
OPT
RWKV
10a
10a
Compute (exaFLOP)
Figure Average performance of RWKV models com- pared transformers across twelve NLP tasks. For further details, see section 5.
1
Introduction
Deep learning has greatly advanced artificial intelli- gence, impacting range of scientific and industrial uses. These often involve complex sequential data
Transformers emerged powerful alternative, adept at managing local and long-range dependen- cies and supporting parallelized training (Tay al., 2022). Models such as GPT-3 (Brown et al., 2020), ChatGPT (OpenAI, 2022: Kocoñ et al., 2023),
Equal first authorship. Others listed alphabetically. 'Code at: https///ittuh.ccm/Biinnee Btt.:///IIiIDDLWKLLLLLLL.. V-LM

---

Model
Time
Space
ture combining RNNs and Transformer advan- lages while mitigating their limitations.
Transformer
d(1
Reformer
Td)
o(TlgTTTal
Detailed experiments demonstrating RWKV's performance and efficiency on benchmark datasets for large-scale models.
Performer
O log d)
O(T log
Linear Transformers
(
O(Td+
AFT-full
d(
O(Td)
AFT-local
((Tsd)
O(Td)
The release of pretrained models, from 169 million to 14 billion parameters, trained on the Pile (Gao et al., 2020; Biderman et al., 2022).
MEGA
O(TTd)
((ad)
RWKV (ours)
O(Td)
O(d)
Table Inference complexity comparison with different Transformers. Here T denotes the sequence length, d the feature dimension, MEGA's chunk size of quadratic attention, and is the size of local window for AFT.
2 Background
Here we briefly review the fundamentals of RNNs and Transformers.
2.1 Recurrent Neural Networks (RNNs)
LLaMA (Touvron et al., 2023), and Chinchilla (Hoffmann et al., 2022) showcase the potential of Transformers in NLP. However, the self-attention mechanism's 's quadratic complexity makes it compu- tationally and memory intensive for tasks involving long sequences and constrained resources. This has stimulated research to enhance Transformers' scalability, sometimes sacrificing some of their ef- fectiveness (Wang et al., 2020; Zaheer al., 2020; Dao et al., 2022a).
Popular RNN architectures such as LSTM (Hochre- iter and Schmidhuber, 1997) and GRU (Chung et al., 2014) are characterized by the following for- mulation (shown for LSTM, others can be reasoned similarly):
f: og (War + Uyh--1 + by),
(1)
4 0g (W + Uphy + b:),
(2)
0g og (Wozt + Uohy 1 bb),
(3)
To tackle these challenges, we introduce the Re- ceptance Weighted Key Value (RWKV) model, combining the strengths of RNNs and Transform- ers while circumventing key drawbacks. RWKV alleviates memory bottleneck and quadratic scaling associated with Transformers (Katharopoulos et al., 2020) with efficient linear scaling, while maintain- ing the expressive properties of the Transformer, such as parallelized training and robust scalability, RWKV reformulates the attention mechanism with a variant of linear attention, replacing traditional dot-product token interaction with more effective channcl-directed attention. This implementation, without approximation, offers the lowest computa- tional and memory complexity; see Table 1.
ē co(Waz; + Uch 1 bo).
(4)
Q ft 0 a- + i 2 ct.
(5)
h 0t 2 (c))
(6)
Although RNNs be factored into linear blocks (W and U) and an RNN-specific block (1)- (6), as noted by Bradbury et al. (2017), the data de- pendency relying on previous time steps prohibits parallelizing these typical RNNs.
2.2 Transformers and AFT
Introduced by Vaswani et al. (2017), Transformers are class of neural networks that have become the dominant architecture for several NLP tasks. Instead of operating on sequences step-by-step like RNNs, Transformers rely on attention mechanisms to capture relationships between all input and all output tokens:
The motivation behind RWKV is to balance com- putational efficiency with expressive capacity in neural networks. It offers solution for handling large-scale models with billions of parameters, hibiting competitive performance at a reduced com- putational cost. Experiments suggest RWKV ad- dresses scaling and deployment challenges in Al, especially for sequential data processing, pointing towards more sustainable and efficient Al models. Our contributions in this paper as follows:
Attn(Q, K, V) softmax (7)
where the helti-headness and scaling factor omitted for convenience. The core QK multipli-
1 is
cation is an ensemble of pairwise attention scores
The introduction of RWKV, a novel architec-
Bhtp:///uugiinnfacccoRvWWW

---

between each token in sequence, which can be decomposed as vector operations:
Oulpur Probokel
ROWKYA Hears
Channel
Softmax
Σ
e ki 1t
(,, V)+
E ed
h
Layerhorm
AFT (Zhai et al., 2021), alternately formulates
Mixing
(W, Ei= " 1l
+ ON wh
Out
Noong
Layerhorm
where {wi,}} E RTxT, is the learned pair-wise po- sition biases, and each Wi, is scalar.
Time Mixing
Inspired by AFT, RWKV takes similar ap- proach. However, for simplicity, it modifies the interaction weights so that it can be transformed into an RNN. Each Wy,i in RWKV is channel- wise time decay vector multiplied by the relative position and traced backward from current time as it decays:
Layenierr
Layerhiona
Layerkamm
Input Embedding
Figure 2: Elements within RWKV block (left) and the complete RWKV residual block, equipped with a final head for language modeling (right).
Weei - it),,,
(10)
where (Rzo)*, with d the number of chan- nels. We require to be non-negative to ensure that e 1 and the per-channel weights decay backwards in time.
softmax operation improving numerical stability and mitigating vanishing gradients (for rigorous proof, see Appendix H). It ensures that the gra- dient is propagated along the most relevant path. Additionally, layer normalization (Ba et al., 2016) incorporated within the architecture aids in stabi- lizing the gradients, effectively addressing both vanishing and exploding gradient issues. These design elements not only enhance the training dy- namics of deep neural networks but also facilitate the stacking of multiple layers, leading to superior performance over conventional RNN models by capturing complex patterns across different levels of abstraction (sec also Appendix I).
3 RWKV
The RWKV model architecture is defined by four fundamental elements that are intrinsic to the time- mixing and channel-mixing blocks:
R: The Receptance vector acts the receiver of past information.
W: The Weight signifies the positional weight decay vector, trainable parameter within the model.
K: The Key vector performs a role analogous
3.1.1 Token Shift
to K in traditional attention mechanisms. V: The Value vector functions similarly to V in conventional attention processes.
In this architecture, linear projection vectors (R, K, V in time-mixing, and R', K' channel- mixing) involved in computations are produced by linear interpolation between current and previous timestep inputs, facilitating token shift.
These core elements interact multiplicatively at each timestep, as depicted in Figure 2.
The vectors for time-mixing computation are linear projections of lincar combinations of the current and previous inputs of the block:
3.1 Architecture
The RWKV model is composed of stacked residual blocks. Each block consists of time-mixing and channel-mixing sub-block, embodying recurrent structures to leverage past information.
r; W, (nr .y (1 (1- (4) xT-1); (11)
k; We (14 ② 2, + (1--))) @ x, x:1),,
(12)
This model uses a unique attention-lis score update process, which includes a time-dependent
2t = W. (Np @ x, (1 (40) C T1-1),
(13)

---

name
is
Bob
the receptance, (r)). The output vector or post the W KV operator given by:
Head
LM Head
LM Head
o, W. (a((r) wkr)).
(17)
Channnll Mix
Channel
Channel
In the channel-mixing block, similar operation performed:
Token shift
Taken shilt
Layer
Layer
Layer Norm
o- ( (W, max(k/, 0)²),
(18)
States
where we adopt the squared ReLU activation func- tion (So et al., 2021).
Time Mix
Time Mix
Time Mix
Taken shift
Token shift
Layer Norm
Layer Norm
Leyer Norm
3.2
Transformer-like Training
Norm
Norm
Layer Norm
RWKV can be efficiently parallelized using a tech- nique called time-parallel mode, reminiscent of Transformers. The time complexity of processing batch of sequences in single layer is O(BTd). primarily consisting of matrix multiplications WA- where À ,, k, u, of (assuming B sequences, T maximum tokens, and channels). In contrast, up- dating attention scores wkyt involves serial scan (see Appendix D for more detail) and has complex- ity (BBdd)
My
name
Figure 3: RWKV architecture for language modeling.
as are the channel-mixie inputs:
rf W (44 C. + (1 6 1), (14)
K/- W (14 O oz (1-14) 1)- (15)
The token shift is implemented simple off- set in the temporal dimension at each block ing the PyTorch (Paszke al., 2019) library as nn. ZeroPad2d (1),,,)))
The matrix multiplications can be parallelized similarly to W,. where X E {Q, V, O} in con- ventional Transformers. The element-wise W KV computation is time-dependent but can be readily parallelized along the other two dimensions (Lei et al., 2018)
3.1.2 WKV Operator
The computation the WKV operator in our model parallels the method used in Attention Free Transformer (AFT) (Zhai et al., 2021). However, unlike AFT where W is a pairwise matrix, our model treats W channel-wise vector that is modified by relative position. In our model, this recurrent behavior defined by the time-dependent update of the W KV vectors, formalized in the fol- lowing equation:
3.3 RNN-like Inference
Recurrent networks commonly utilize the output at state input at state + 1. This usage is also observed the autoregressive decoding infer- ence of language models, where each token must be computed before being passed to the next step. RWKV takes advantage of this RNN-like structure, known as time-sequential mode. In this context, RWKV can be conveniently formulated recursively for decoding during inference, demonstrated in Appendix D.
EE= -1--111)) O 2s + eu @
wkv,
(t-1-s]w+h + eathe
(16)
3.4 Additional Optimizations
To circumvent any potential degradation of W. we introduce a vector U that separately attends to the current token. More information about this can be found in Appendix I.
Custom Kernels To address inefficiencies in the W WKV computation arising from the sequential na- ture of the task when using standard deep learning frameworks, we have developed custom CUDA
3.1.3 Output Gating
FFor extremely long sequences, more sophisticated meth- ods such as Martin and Cundy (2017) that parallelize over sequence length could be used.
Output gating is implemented in both time- mixingixing and channel-mixing blocks using the sigmoid of

---

kernel. This kernel enables the execution of sin- gle compule kernel on training accelerators, while all other parts of the model, such as matrix multi- plications and point-wise operations, are already inherently parallelizable and efficient.
for next-token prediction and computation of the cross-entropy loss during training.
Trained Models and Computing Costs
Small Init Embedding During the initial stage of training transformer model (Vaswani et al., 2017), we observe that the embedding matrix dergoes slow changes, presenting a challenge for the model move away from its initial noisy em- bedding state. To address this issuc, we propose an approach that involves initializing the embed- ding matrix with small values and subsequently applying an additional LayerNorm operation. This accelerates and stabilizes the training process, al- lowing for the training of deep architectures with post-LN components. The effectiveness of this approach demonstrated Figure illustrating improved convergence by enabling the model to quickly transition away from the initially small em- bedding. This is achieved through small changes occurring in single step, which subsequently lead to substantial alterations in directions and further notable changes after the LayerNorm operation.
To demonstrate the scalability of RWKV, we train six models ranging from 169 million to 14 billion parameters as shown in Table 2. All models are trained for one epoch (330 billion tokens) on the Pile (Gao et al., 2020; Biderman al., 2022).
Layers
Model Dimension
Parameters
FLOP per
169
12
1.693 10*
2.613 10
130M
24
1024
4.304 x 10a
7.573
10
1,5
24
2048
1.515 x 10P
2.823
109
3
32
2560
2.953 x 10P
5.710
32
109
1:437
10
14
40
5120
1,415 10.0
2.778
10
Table 2: RWKV model architectures and FLOP counts, Further details of these hyperparameters are elaborated upon Appendix G.
The number of parameters for each model is computed using the formula: # parameters 2VD + 13D + D(11L + 4) where V = 50277 the vocabulary size, D represents the Model Di- mension and L corresponds to the number of lay- ers. FLOPs is for a forward pass for one token. It was calculated as 2(2V + 13D" I.), which is the twice (add and multiply) the number of parameters in linear layers. The backwards pass FLOPs can be approximated as twice that of the forward pass, giving total of 6(2V 13D2 L) FLOP per to- ken. Notably, this matches the standard formula for FLOP calculations transformers Kaplan et al. (2020): FLOP 6 tokens] {# parameters].
Custom Initialization Building on principles from previous works (He al., 2016; Jumper et al., 2021), we adopt an initialization strategy where parameters are set to values resembling an identity mapping while breaking symmetry to establish a clear information flow. The majority of weights are initialized to zero, and linear layers do not employ biases. Detailed formulas We observe that the choice of initialization plays crucial role in both the speed and quality of conver- gence (refer to Appendix FF for further details).
given in Appendix E.
3.5 Implementation
4.1 Additional Training Details
RWKV is implemented using the PyTorch Deep Learning Library (Paszke et al., 2019). We inte- grate additional optimization strategies inspired by DeepSpeed (Rasley et al., 2020) into the system, improving its efficiency and scalability.
For training, we use the standard Adam optimizer without weight decay, use 6 precision, and train with a context length of 1024 tokens. Fur- ther details on hyperparameters are Appendix G. Diverting from standard practice for transformers, we apply exponential decay to our learning rate. We also incorporate the auxiliary loss introduced by PaLM (Chowdhery al., 2022), supplement- ing the standard cross-entropy loss function. This auxiliary loss encourages the softmax normalizer to approximate zero closely. As for the learning rate schedule, it remains constant for the initial iterations, and subsequently decays exponentially.
The model begins with an embedding layer, as detailed in Section 3.4. Following this are sev- eral identical residual blocks arranged sequentially. These are depicted in Figures 2 and 3 and adheres the principles outlined in Section 1.1.1. After the last block, simple output projection head, con- sisting of a LayerNorm (Ba et al., 2016) and a linear projection, is employed for logits generation

---

4.2 Scaling Laws
Competitiveness Is RWKV competitive against quadratic transformer architectures with the same amount of compute?
Scaling laws (Kaplan et al., 2020; Henighan et al., 2020; Hoffmann et al., 2022; Muennighoff et al., 2023) in language models refer to the mathemati- cal relationships that describe how the performance of language model changes with respect ious factors. These factors can include the model size (N), dataset size (D), or the optimally allo- cated compute budget (Cmin). Scaling laws are important for two primary reasons: they allow us to make predictions and plans regarding the costs and performance of large models before they are trained via interpolation and extrapolation (Black et al., 2022; Le Scao et al., 2022) and the contexts in which they fail provides rich feedback im- portant areas for future research (Wei et al., 2022a; Biderman et al., 2023a).
Long Context Does increasing the context length of RWKV yield better language modeling loss when RWKV models are trained for context lengths that most open-sourced quadratic trans- formers cannot efficiently process?
5.1 NLP Evaluations
To demonstrate that RWKV is competitive with traditional transformers at NLP tasks, we compare with similarly sized models trained for a similar number of tokens (Pythia (Biderman et al., 2023b), OPT (Zhang et al., 2022) and BLOOM (Scao et al., 2022)). All RWKV models were trained for one epoch the Pile (330B tokens), which is close but not identical to the amount of tokens the Pythia, OPT, and BLOOM models were trained for. Con- sequently, we compare our models on a FLOP- matched basis. We avoid comparing with model trained in the Chinchilla-optimal regime (Hoff- mann et al., 2022) or the overtrained regime (Tou- vron et al., 2023) to ensure the most equitable com- parison.
Previous work on scaling laws for RNNs has claimed that LSTMs do not strictly follow the same log-log linear scaling that transformers do (Kaplan et al., 2020). We train 45 RWKV models for variety of pairs (dataset, parameters) and find that RWKV does follow the same general form of the scaling law that is well established for transformers. Figure 4 shows our results for loss as function of compute, with the linear fit to the Pareto optimal points holding an x2 value of 0.994. Even when we extrapolate our curve an additional order of magnitude (blue), we find an extremely good fit with an p2 of 0.875.
We report results on ARC (both Easy and Chal- lenge) (Clark et al., 2018), BoolQ (Clark et al., 2019), COPA (Roemmele et al., 2018), HeadQA (Vilares and Gódez-Rodríguez, 2019), HellaSwag (Zellers et al., 2019), LAMBADA (Paperno et al.. 2016), OpenBookQA (Mihaylov et al., 2018), PIQA (Bisk et al., 2020), ReCoRD (Zhang et al.. 2018), SciQ (Johannes Welbl Nelson F. Liu, 2017). and Winogrande (Zellers et al., 2020). Figure/ shows the average results across all benchmarks. Some individual benchmarks are shown in Fig 5, with the rest in Appendix J.
Non Optimal
Optimal
Tend Line
Sxx
Loss
Additionally, we carried out comparative studies on RWKV and ChatGPT / GPT-4, see Appendix L. They revealed that RWKV is very sensitive to prompt engineering. When the prompts were ad- justed (re-ordered) from the ones used for GPT more suitable for RWKV, the performance (F1) in- creased even from 44.2% to 74.8%. For sarcasm detection, RWKV outperformed ChatGPT, but was still slightly worse than the SOTA solution.
2xx10
10-1
10a
10*
10'
10
Compute (eX
Figure 4: Scaling laws curves for RWKV models
5 Evaluations
5.2 Extended Context Finetuning
Having demonstrated the scalability of RWKV models in the previous section, we now turn our attention to their competitiveness with traditional transformers. We focus on two questions:
Unlike transformers, RNNs do not have a pre- defined sequences length when they are created. However order to efficient make use compute

---

315
300
204
Greasteet.oo 187
22
Compute
(a) ARC (Challenge)
HellaSwag
(c) LAMBADA (OpenAI)
215
23.0
312
350
Meaa
51
16'
W0
31
(d) Open
ReCoRD
(f) Winogrande
Figure 5: Zero-Shot Performance of RWKV on common language modeling evaluation benchmarks. Additional plots can be found in Appendix J.
we nevertheless need to preprocess the training data into contexts of the same length. We find that we are able to teach the model how to efficiently handle substantially larger batch sizes by finetuning with progressively increasing sequence length. Specif- ically, we first double the sequence length from 1024 to 2048 and finetune for 10B tokens from the original pretraining corpus, then we double again to 4096 for 100B tokens from the same corpus, and finally double to 8192 tokens for another 100B to- kens from the same corpus. In Fig. 6 we show that increasing context length leads lower test loss the Pile, indication that RWKV can make effective use of long contextual information.
5.3 Long Context Benchmarks
Additionally, evaluate model's ability handle very long sequences by comparing to state- of-the-art long sequence models on Long- Range Arena (LRA) benchmark (Tay et al., 2021). LRA is designed to assess the performance mod- els in handling lengthy context situations. It in- cludes a collection of tasks with sequences rang- ing from 1,000 16,000 tokens, covering various types of data like text, natural language, synthetic images, and mathematical expressions. We apply RWKV on the LRA benchmark and the results are in Appendix J.2. The results show that RWKV per- forms second only to the S4 model five datasets.
6 Inference Experiments
78
We benchmark inference requirements according to size and family. Specifically, we evaluate text gen- eration speed and memory requirements on typical compute platforms including CPU (x86) and GPU (NVIDTA A100 80 GB). For all of our inference experiments
Hice test loss
2
23
float32 precision and the Hug- gingFace Transformers (Wolf et al., 2020). We in- clude all model parameters in the parameter count, including both embedding and non-embedding lay- ers. Performance under different quantization se- tups is left to further work. See Appendix K for
Lengtth
Figure 6: RWKV shows decreasing mean test loss as a
function of context length the Pile (Gao al., 2020)

---

more results.
experiments on expressivity, interpretability, and scaling showcase the model capabilities and draw parallels behavior between RWKV and other LLMs.
2 7h
pa gerninstan
60
Boutherkgggt noa 2.78
60
Teethwllyytt-- Bb twor 4 pilo do
RWKV opens a new route for scalable and ef- ficient architectures to model complex relation- ships in sequential data. While many alternatives to Transformers have been proposed with similar claims, ours is the first back up those claims with pretrained models with tens of billions of parame- ters.
40
20
20
Limitations
Figure 7: Cumulative time on text generation for LLMs. Unlike transformrrs, RWKV exhibits linear scaling.
While our proposed RWKV model has demon- strated promising results regarding training and memory efficiency during inference, some limita- tions should be acknowledged and addressed in future work.
7 Future Work
There are several promising directions for future work on the RWKV architecture. Work can be done to increase model expressivity by enhancing the time-decay formulations and exploring initial model states while maintaining efficiency. The RWKY computational efficiency can be further im- proved by applying a parallel sean the wkvt step to reduce the computational cost to ( lo)( The mechanisms used in RWKV can be applied to mecoder-decode architectures, potentially re- placing the cross-attention mechanism. This could be applicable in seq2seq multimodal settings, thereby enhancing efficiency during both training and inference.
First, the linear attention of RWKV leads sig- nificant efficiency gains but still, it may also limit the model's performance on tasks that require re- calling minutiae information over very long con- texts. This is due to the funneling of informa- tion through a single vector representation over many time steps, compared with the full informa- tion maintained by the quadratic attention of stan- dard Transformers. In other words, the model's recurrent architecture inherently limits its ability "look back" previous tokens, as opposed tra- ditional self-attention mechanisms. While learned time decay helps prevent the loss of information. is mechanistically limited compared to full self- attention.
RWKV's state (or context) can be leveraged for interpretability, predictability in sequence data, and safety. Manipulating the hidden state could also guide behavior and allow greater customizability through prompt tuning.
Another limitation of this work is the increased importance of prompt engineering in comparison to standard Transformer models. The linear attention mechanism used in RWKV limits the information from the prompt that will be carried over to the model's continuation. As a result, carefully de- signed prompts may be even more crucial for the model to perform well on tasks.
8 Conclusions
We introduced RWKV, new approach to RNN models exploiting the potential of time-based mix- ing components. RWKV introduces several key strategies that allow it to capture locality and long- range dependencies while addressing limitations of current architectures by: (1) replacing the quadratic QK attention with a scalar formulation at linear cost, (2) reformulating recurrence and sequential inductive biases to enable efficient training paral- lelization and efficient inference, and (3) enhancing training dynamics using custom initializations.
The above RWKV property was confirmed by studies prompt engincering presented in Ap- pendix L. By changing the order of the information pieces, we were even able almost double the RWKV performance for some tasks.
10 Ethics Statement
We benchmark the proposed architecture in a wide variety of NLP tasks and show comparable performance to SoTA with reduced cost. Further
In this paper, we present novel architecture for se- quential data processing and prove its effectiveness by building a series of LLMs trained on publicly re-

---

leased pretraining data (Gao et al., 2020; Biderman al., 2022) and later fine-tuned on publicly avail- able instructions (Taori et al., 2023; Chaudhary, 2023; Cheung. 2023; Anand et al., 2023; Anony- mous, 2023; Yang, 2023; Ji et al., 2023a,b).
References
Mohammad Mahmudul Alam, Edward Raff, Stella Bi- derman, Tim Oates, and James Holt. 2023. Recasting self-attention with holographic reduced representa- tions. arXiv preprint arXiv:2305. 19534.
As a novel architecture for sequential data, RWKV has the potential to improve sequence- based models across different applications ranging from natural language processing to biomedical data processing or climate modelling. Since the training code is released open source, RWKV con- tributes to the democratization of AI, levels the playing field, and empowers members of the Open Source community to inspect, study, and finetune RWKV in particular tasks. Morcover, it contributes to advancing the understanding of LLMs capabili- ties and limitations. A significant amount of work has been devoted to increasing the efficiency of RWKV training as minimize its cost and pro- mote accessibility.
Alon Albalak, Yi-Lin Tuan, Pegah Jandaghi, Connor Pryor, Luke Yoffe, Deepak Ramachandran, Lise Getoor, Jay Pujara, and William Yang Wang. 2022. FETA: A benchmark for few-sample task transfer in open-domain dialogue. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 10936-10953, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Yuvanesh Anand, Zach Nussbaum, Brandon Duder- stadt, Benjamin Schmidt, and Andriy Mulyar. 2023. Gpt4all: Training assistant-style chatbot with large scale data distillation from gpt-3.5-turbo https: //github.com http//omiccaiiggteeeee
Anonymous. 2023, Sharegpt_vicume wtt___vvuummuuffiltrrre unfiltered.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hin- ton, 2016. Layer normalization.
As LLMs trained on public data, RWKV's lower inference cost compared to Transformer alterna- tives makes it more suitable for deployment in con- sumer and edge hardware, which step towards the democratization and distribution of LLMs to the general public, creating better privacy and own- ership incentives. It also lowers the resource bar- rier to Chat assistants and text generation for small and/or underrepresented communities. PreTrained model weights for different sizes ranging from O.1B 14B parameters trained on multiple languages are released to increase ease of adoption and allow for the study of emergent phenomena.
Francesco Barbieri, Jose Camacho-Collados, Luis Es- pinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages (644-1650, Online. Association for Computational Linguistics.
1z Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. 2XXi:2004.05150
Stella Biderman, Kieran Bicheno, and Leo Gao. 2022. Datasheet for the pile. arXiv preprint a.X1v:2001.07311.
On the other hand, with lower resouree barri- ers, the spreading of AT-generated text might be- come more prevalent. Current RWKV LLMs may exhibit and/or reproduce biases and potentially harmful content present in the data used for train- ing. Nonetheless, mitigation and finetuning strate- gies discussed for other, large Transformer models should be applicable to RWKV as well.
Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raf. 2023a. Emer- gent and predictable memorization large language models. arXiv preprini arXivv2304.11158
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley. Kyle O' O'Brien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023b. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint 2rXiv:2304.01373
Acknowledgements
We thank StabilityAl AT for the compute used to train our models and for technical support in develop- ment of RWKV. We also thank the members of the RWKV and EleutherAl Discord servers for their help and work on further extending the applicabil- ity of RWKV to different domains.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Thirty- Fourth AAAI Conference on Artificial Intelligence.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. Gpt-neo: Large scale autore- gressive language modeling with mesh-tensorflow. URL: hups://ooi. arg/10.5281zeenon $297715.

---

Sidney Black, Stella Biderman. Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022. Gpt-neox-20b: An open-souree autoregres- sive language model. In Proceedings of BigScience Episode # Workshop on Challenges & Perspec- tives in Creating Large Language Models, pages 95- 136.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho. and Yoshua Bengio. 2014, Empirical evaluation of gated recurrent neural networks on sequence model- ing. In NIPS 2014 Deep Learning and Representa- tion Learning Workshop.
Christopher Clark, Kenton Lee, Ming-We Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutamova, 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905. 10044.
James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. 2017. Quasi-recurrent neural net- works. In ICLR.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try are, the ai2 reasoning challenge. In arXiv:1803.05457.
Tom Brown Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Ja- cob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. In arXiv, volume abs/2110.14168.
Aydar Bulatov, Yuri Kuratov, and Mikhail S. Burtsev. 2023. Scaling transformer to Im tokens and beyond with rmt.
Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. 2022. Recurrent memory transformer. Advances in Neural Information Processing Systems, 1079-
Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Re. 2022a. Flashattention: Fast and memory--fficient exact attention with IO-awareness. In Advances in Neural Information Processing Sys- tems.
11091.
Sahil Chaudhary. 2023. Code alpaca: An instruction- following Ilama model for code generation. https: / //gi lig.thuu.com cww/ /sah//shiiii 12801228441 1018011//coeeappac.
Tri Dao, Daniel Y Fu, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. 2022b. Hungry hungry hippos: Towards language mod- cling with state space models. arXiv preprint arXiv:2212. 140522
Joseph Cheung. 2023. Guanacodataset.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song. Andreea Gane, Tamas Sar- los, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. 2020. Rethinking attention with per- formers.
Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko. Alan S. Cowen, Gaurav Nemade, and Sujith Ravi. 2020. Goemotions: A dataset of fine mne-grained emo- tions. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, ACL 2020. Online, July 5-10, 2020, pages 4050. Association for Computational Linguisties.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes. Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstem, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways. CoRR, 0bs/222440021111
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Ho- race He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for lan- guage modeling. arXiv preprint 1rXiv:2001.0002
Albert Gu, Karan Goel, and Christopher Ré. 2021. Effi ciently modeling long sequences with structured state spaces, arXiv preprint arXiv:2111.00330
Albert Gu, Karan Goel, and Christopher Ré. 2022. Effi- ciently modeling long sequences with structured state spaces. In The International Conference on Learning Representations (ICLR).
Mandy Guo, Joshua Ainslic, David C Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2022. Longt5: Effficient text-to-text trans- former for long sequences. In Findings of the Associ- ation for Computational Linguistics: NAACL 2022. pages 724-736.

---

Ankit Gupta, Albert Gu, and Jonathan Berant, 2022. Di- agonal state spaces are as effective as structured state spaces. Advances in Neural Information Processing Systems, 35:22982-22994.
Matt Gardner Johannes Welbl Nelson Liu. 2017. Crowdsourcing multiple choice science questions. In DOL: 10. ///////6/////7/4/4
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figumno, Olaf Ronneberger. Kathryn Tunyasuvunakool. Russ Bates, Augustin Zidek, Anna Potapenko, and et al. 2021. Highly accurate pro- tein structure prediction with alphafold. Nature, 596(7873):533-58.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Identity mappings in deep residual net- works.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. 2020. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14701.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.0830
Sepp Hochreiter. 1998. The vanishing gradient problem during learning recurrent neural nets and problem so- lutions. International Journal of Uncertainty, Fuzzi- ness and Knowledge-Bor Beaeee Systems, 6002):107-116.
Angelos Katharopoulos, Apoory Vyas, Nikolaos Pap- pas, and François Fleuret. 2020. Transformers are mns: Fast autoregressive transformers with linear attention. In International Conference Machine Learning, pages 5156-5165, PMLR.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735- 1780.
Nikita Kitaev, L. Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. ArXiv, abs/2001.04451.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford. Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican. George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Si- monyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models.
Jan Kocoñ, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanck, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, Anna Kocoñ, Bartlomiej Koptyra, Wik- toria Mieleszczenko- Kowseeeezzz,,, Piotr Milkowski, Marcin Oleksy, Maciej Piasecki, Eukasz Radliński, Konrad Wojtasik, Stanislaw Wožniak, and Prze- myslaw Kazienko. 2023. Chatgpt: Jack of all trades, master of none. Information Fusion, page 101861.
Hassan Ismail Fawaz, Germain Forestier, Jonathan We- ber, Lhassane Idoumghar, and Pierre-Alain Muller. 2019. Deep learning for time series classification: a review. Data mining and knowledge discovery, 33(4):917-963.
Jan Kocoń, Piotr Milkowski, and Monika Zaško- Zielińska. 2019. Multi-level sentiment analysis of polemo 2.0: Extended corpus of multi-domain con- sumer reviews. In Proceedings of the 23rd Confer- ence on Computational Natural Language Learning (CoNLL), pages 980-991.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. 2021. Perceiver: General perception with iterative atten- tion. In International conference on machine learn- ing, pages 4651-4664. PMLR.
Phong Le and Willem Zuidema. 2016. Quantifying the vanishing gradient and long distance dependency problem in recursive neural networks and recursive Istms. In Proceedings of the 1st Workshop on Repre- sentation Learning for NLP, pages 87-93.
Hanhwi Jang, Joonsung Kim, Jae-Eon Jo, Jaewon Lee, and Jangwoo Kim. 2019. Mnnfast: A fast and scal- neu-
able system architecture for memory-augmente ral networks. In Proceedings of the 46th Interna- tional Symposium on Computer Architecture, pages 250-263.
Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Bider- man, Hady Elsahar, Jason Phang, Ofir Press, al. 2022. What language model to train if you have one million gpu hours? In Proceedings of BigScience Episode #5- Workshop Challenges & Perspectives in Creating Large Language Models.
Yunjie Ji, Yong Deng, Yan Gong, Yiping Peng, Qiang Niu, Baochang Ma, and Xiangang Li. 2023a. Belle: Be everyone's large language model engine. https: //github.ce wwt.c//aaaaaaecc//BBLL...
Yunjie Ji, Yong Deng, Yan Gong, Yiping Peng. Qiang Niu. Lei Zhang, Baochang Ma, and Xiangang Li. 2023b. Exploring the impact of instruction data scaling on large language models: An empirical study on real-world use cases. arXiv preprint arXiv:2303. 14742.
Tao Lei, Yu Zhang. Sida I. Wang, Hui Dai, and Yoav Artzi. 2018 Simple recurrent units for highly paral- lelizable recurrence. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 4470-448) Brussels, Belgium. Association for Computational Linguistics.

---

Hanxiao Liu, Zihang Dai, David R. So, and Quoe V. Le. 2021. Pay attention to mlps.
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Ste- fano Ermon, and Christopher Ré. 2023. Hyena hierar- chy: Towards larger convolutional language models. arXiv preprinl arXiv:2302. 10866.
Xuezhe Ma, Xiang Kong. Sinong Wang. Chunting Zhou, Jonathan May, Hao Ma, and Luke 7eulemoyer. 2021. Luna: Linear unified nested attention. Advances in Neural Information Processing Systems, 34:2441- 2453.
Ilan Price, Jordan Gifford-Moore, Jory Flemming, Saul Musker, Maayan Roichman, Guillaume Sylvain, Nithum Thain, Lucas Dixon, and Jeffrey Sorensen. 2020. Six attributes of unhealthy conversations. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 114-124, Online. Association for Computational Linguistics.
Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. 2023. Mega: Moving average equipped gated attention. In ICLR.
Markus N. Rabe and Charles Staats. 2022. Self- attention does need o(n²) memory.
Eric Martin and Chris Cundy. 2017. Parallelizing linear recurrent neural nets over sequence length. ArXiv, abs/1709.04057.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery Data Mining, KDD '20, page 3505-3506, New York, NY, USA. Association for Computing Machinery.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov, 2022. Locating and editing factual asso- ciations in GPT. Advances in Neural Information Processing Systems, 36.
Todor Mihaylov. Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can suit of armor conduct elec- tricity? new dataset for open book question answer- ing. In EMNLP.
Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S. Gordon. 2018. Choice of plausible alterna- tives: An evaluation of commonsense causal reason- ing. In AAAI.
Niklas Muennighoff. Alexander Rush, Boaz Barak. Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. 2023. Scaling data-constrained language models. arXiv preprint arXiv:2305.16266
Teven Le Scao, Angela Fan, Christopher Akiki. EI- lie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, al. 2022. Bloom: 176b- parameter open-access multilingual language model. arXiv preprint arXiv: 201.2221550001 1.05100.
OpenAl. 2022. Introducing chatgpt. https://openal) com/blog/chatggt
Ramsha Siddiqui. 2019. SARCASMANIA: Sarcasm
Antonio Orvicto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. 2023. Resurrecting recurrent ral networks for long sequences. arXiv preprint arXiv:2303.06349.
Exposed! http:// //wpw.//ww.. kaggle com/rmsharks4 sarcasmania a-da [Online; accessed 02- February-2023).
David R. So, Wojeiech Manke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V. Le. 2021. Primer: Searching for efficient transformers for language modeling. CoRR, abs/2109.08668.
Denis Paperno, Germán Kruszewski, Angeliki Lazari- dou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context, In Proceedings of the 54th Anmual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 1535-1534, Berlin, Germany. Association for Computational Linguistics.
Rohan Taori, Ishaan Guirajani. Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Talsunori Hashimoto. 2023. Stanford alpaca: An instruction instruction-folloiinng llama model. https: // gi thub.
Yi Tay, Dara Bahri, Donald Metzler, Da-Cheus Juan, Zhe Zhao, and Che Zheng. 2020. Synthesizer: Re- thinking self-attention transformer models.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf. Edward Yang. Zach Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Jun- jie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high- perfforrrfor deep learning brary.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang. Sehastian Ruder, and Donald Metzler. 2021. Long range arena A benchmark for efficient transformers. In International Conference on Learning Representa- tions.

---

Tay, Mostafa Dehghani, Dara Bahri, and Donald Met- zler. 2022. Efficient transformers: A survey. ACM Computing Surveys, 55(6):1-28.
Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: ofae-of-the-Are Natural Lan- guage Processing. pages 38-45, Association for Computational Linguistics.
Ilya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer. Xiaohua Zhai, Thomas Unterthiner. Jes- sica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. 2021. Mlp-mixer: An all-mip architecture for vi- sion. CoRR, 1aw/2105501601.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. 2020. A com- prehensive survey graph neural networks. IEEE transactions on neural networks and learning sys- tems, 4-
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language models.
Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: Personal attacks seen scale. In Pro- ceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017, pages 1391-1399. ACM.
Jianxin Yang. 2023. Firefly. https://github.com yangjiankin1/Fireely
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang. Yang. et al. 2020. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems. 33.
David Vilares and Carlos Gómez- Rodriigz. 2019. Head-qa: A healthcare dataset for complex reasoning. ICAC
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? InACL.
Alex Wang. Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understand- ing systems. In Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Rowan Zellers. Ari Holtzman, Yonatan Bisk. Ali Farhadi, and Yejin Choi. 2020. Winogrande: An adversarial winograd schemn challenge at scale. ACL.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill. Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, and Josh Susskind. 2021. An attention free transformer.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and ma- chine commonsense reading comprehension. In arXiv: 1810. 12885.
Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint X2xx:222500066.
Jason Wei. Yi Tay, Rishi Bommasani, Colin Raf- fel. Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Met- zler, Ed Chi, Tatsunori Hashimoto, Oriol Vinyals. Percy Liang. Jeff Dean, and William Fedus. 2022a. Emergent abilities large language models. ArXiv, abs/2206.07682.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. 2022b. Chain thought prompting elicits reasoning large language models. arXiv preprint arXiv:2201.1 1903.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Perric Cistac, Clara Ma, Yacine Jernite, Julien Plu, Can- wen Xu, Teven Le Scao, Sylvain Gugger, Mariama

---

A Author Contributions
All authors contributed to the drafting of this paper. Eric Alcaide and Quentin Anthony organized the paper and its experiments and were involved in all phases of the development process.
Model Design and Development Bo Peng (lead), Matteo Grella, Xuzheng He, Haowen Hou, Jiaming Kong, Johan S. Wind
Model Training Bo Peng
Scaling Laws Analysis
Stella Biderman, Bo Peng
Benchmark Evaluations Stella Biderman (lead). Kranthi Kiran GV, Krishna Sri Ipsit Mantri, Atsushi Saito, Qihang Zhao, Peng Zhou, Rui-Jie Zhuäç
Long Context Experiments Rui-Jie Zhu
Xingjian Du, Rui-Jie Zhu, Bolun Wang, Ruichong Zhang, Jian Zhu,
Inference Speed Experiments
Samuel Arcadinho, Przemystaw Kazienko, Qinghua Zhou
Information Flow Experiments Huanqi Cao, Michael Chung, Mattco Grella, Ferdinand Mom, Zhenyuan Zhang
Jan Kocoń (lead), Przemyslaw Kazienko, Bartlomicj Koptyra, Hayden Lau, Xiangru Tang, Stanislaw Wožniak, Zhenyuan Zhang
Chat Experiments
Ethics and Broader Impacts
Stella Biderman, Guangyu Song

---

B Author Contributions
Bo Peng Original RWKV idea, original code, performance optimizations, original experiments, and trained RWKV models from O.1B to 14B.
Eric Alcaide Manuscript (initial draft sections 1, C; sections 3, and 8; revision and proofreading; final version) Figures (2, 3, 3, 8). Experiments section 6. Appendices E, K. Contributions to Appendix M.
Quentin Anthony Manuscript (organization, initial draft sections 1, C, 2; revision and proofreading; final version).
Alon Albalak
Manuscript (abstract and sections 1, 9, and 7: proofreading and revision).
Contributions to Figures 7, 13, and 14. Contributions to Appendix K. Stella Biderman Performed the scaling laws analysis and evaluated competitor models on benchmark tasks.
Samuel Arcadinho
Huanqi Cao Appendix I.
Manuscript (contributions to 3.2 and 3.3; proofreading and revision). Experiments for
Xin Cheng Manuscript (proofreading and revision). Contributions to Appendix M. J.
Michael Chung
Manuscript (contributions to section I; proofreading and revision).
Xingjian Du
Evaluation on Long Range Arena Benchmark (TBD until 5.31).
Manuscript (sections H, 8; contributions sections 1, 7 and 9; proofreading and revision). Contributions to Appendix D.
Matteo Grella
Kranthi Kiran GV Tables K and K. Appendix 4.
Manuscript (sections C and 5; contributions to section 2; revision and proofreading).
Xuzheng He
Manuscript (contributions section 2: proofreading and revision). Contributions
Figure8. Appendix I. Contributions uppendix H.
Haowen Hou Figure 9. Appendix F.
Jiaju Lin RWKV LRA benchmarking
Przemyshaw Kazienko Manuscript (proofreading and revision). Contributions to Section 6, 9, and Appendix L.
Jan Kocon Manuscript (Section 1; proofreading and revision). Contributions to Appendix L.
Jiaming Kong Manuscript (revision and proofreading). Appendix H.
Bartlomiej Koptyra
Manuscript (revision and proofreading) Contributions to Appendix L.
Hayden Lau Manuscript (contributions section and 9; proofreading and revision). Contributions Appendix M.
Krishna Sri Ipsit Mantri Figure 12
Ferdinand Mom Manuscript (contributions to section 1, C, 3.3, I; proofreading and revision). Contribu- tions Appendix D.
Atsushi Saito
Manuscript (sections and 5; contributions to section C). Contributions to Appendix J
Guangyu Song
Manuscript (rewrote section 3; final version). Initial draft Ethics Statement).
Xiangru Tang Manuscript (sections and 2; contributions to abstract; revision and proofreading).
Contributions to Appendix M.

---

Bolun Wang
Contributions Tables 1.
Johan S. Wind
RWKV performance optimizations (CUDA), Contributions to Appendix 4.
Stanislaw Woániak
Contributions Appendix L..
Ruichong Zhang
Manuscript (proofreading and revision); Contributions to Figure 6 and Appendix M.
Zhenyuan Zhang
Manuscript (revision and proofreading). Figure 3. Experiments Appendix I. Contri- butions to Appendices D and M.
Qihang Zhao
Manuscript (proofreading and revision). Contributions Table 5.
Peng Zhou Contributions to Tables and Table 5.
Qinghua Zhou
Manuscript (Proofreading and revision of section 3; Add missing citations in 3.3).
Revision of Figures 2 and 12.
Jian Zhu
Manuscript (section C; proofreading and revision). Figures 3 and 6.
Rui-Jie Zhu
Tables and 5. Experiments for table 5.
C Additional Related Work
Recently, a number of techniques have been proposed address the limitations of transformers.
Optimizing Attention Mechanism Many transformer variants ("x-formers") have been introduced reduce the complexity of transformers (Tay et al., 2022), including sparse attention (Beltagy et al., 2020; Kitaev et al., 2020; Guo et al., 2022), approximating the full attention matrix (Wang et al., 2020; Ma et al., 2021: Choromanski et al., 2020), combining chunked attention with gating (Ma et al., 2023) and other efficient methods (Katharopoulos et al., 2020; Jaegle et al., 2021).
Some recent works like FlashAttention (Dao et al., 2022a) and others (Rabe and Staats, 2022: Jang al., 2019) share similarities with RWKV's chunked computation scheme. Despite being memory-efficient their time complexity remains quadratic or contains chunk size as hidden factor. In contrast, RWKV achieves better space and time complexity during inference by formulating a linear attention as an RNN.
Attention Free Models Another line of research replaces the attention mechanism with other modules to scale to long sequences. MI.P-Mixer and others (Tolstikhin et al., 2021; I; Liu et al., 2021) propose replacing attention by Multi-Layer Perceptrons (MLPs) in computer vision tasks. The Attention Free Transformer (AFT) (Zhai et al., 2021) and HrrFormer (Alam et al., 2023) replaces dot-product self-attention with a computationally efficient alternative. None of these models have been successfully scaled to the point where drawing comparisons with transformer-based large language models makes sense.
There has also been substantial research into state space models (SSM) (Gu et al., 2021) and its variants (Dao et al., 2022b; Gupta et al., 2022: Poli et al., 2023). contrast to the preceding models, SSM and its successors have shown substantial progress towards efficient scaling. Simultaneously with this work, Poli et al. (2023) train SSM-based models with 125 million and 355 million parameters and show that the performance is on-par with a transformer that uses mix of local and global attention (Black et al., 2021).
Advances in RNNs Inspired by the success of transformers, RNN-style (Hochreiter and Schmidhuber, 1997; Chung et al., 2014) recursive components have also been modified to increase context length, such the Recurrent Memory Transformer (Bulatov et al., 2022, 2023) and Linear Recurrent Units (Orvicto al., 2023). Most similar our work, the Quasi-Reeurrent neural network (QRNN) (Bradbury et al., 2017) uses both convolutional layers and recurrent pooling functions across timesteps and channels. While QRNN utilizes convolutional filters with fixed sizes, RWKV employs a time-mixing module as an attention mechanism with time-decaying factors. Different from the clement-wise pooling in QRNN, RWKV includes parametrized thannel-mixing module that parallelizable.

---

D Time-Mixing Block as RNN Cell
As stated in 3.3, the RWKV time-mixing block can formulated as an RNN, as the W KV computation can be written in such a recursive form:
ag. 6o 0,
(19)
wky, = 0111 + eathe O U2
(20)
bi-1 + cuthe
a; = e e- ① a1-1 + the o 1
(21)
by = c
2 1 ch
(22)
The dataflow of the RNN-like time-mixing is shown in Fig. 8, where the hidden states is the numerator-denominntto tuple (a, b). To avoid overflow in calculating c²tt, numerical trick is used in the official implementation. Noticing that = c @ and b = ex1, we set a1 v1, b/ = 1, k1, where pt stores the shared exponents of a, and b,- Now the above recursion can be converted into a numerical safe version, for each time step >
h
3
wkv.
X1-1
Xi
X;
Figure 8: RWKV time-mixing block formulated as an RNN cell. Color codes: yellow (μ4) denotes the token shift, red (1) denotes the denominator, blue (2) denotes the numerator, and pink (3) denotes the fraction computations in 16. h denotes the numerator-denominaaoo tuple.
q max((t-1x + kz), wkyz
(23)
eP--1 2 d/-1 + eathe-g Dz
(24)
eh 1 b + cutke-
The update to d/ 1/, and their shared exponent is also carried out in similar fashion:
o' == max - w,),),
(25)
dg Pi-1- 3 af + k- et,
(26)
6, = pi-1- @b 1 to und
(27)
Pt
(28)
The RWKV model has an internal state that stores some previous information. In each layer, the internal state consists five parts, each of which is vector with D numbers, where is the model dimension. The five parts are:
The current input of the Time-mix block 24;
The current input of the Channel-mix block y::
The numerator of the W KV value a/. defined in equation (26);
The denominator of the W KV value b4. as defined in equation (27);
An auxiliary state Pt (28), which is used for W KV computation to maintain numerical precision.
Which yields a total size of 5DL parameters. It is worth noting that in an algebraic context with infinite precision, the helper state Pt can be ignored, and the W KV numerator and denominator can be computed directly using equations (21) and (22), reducing the size of the internal state to 4DL.
E Parameter initializations
We describe the specific parameter initializations below and motivate the design choices. Parameters belonging residual blocks are often adjusted by layer depth and total number of layers. Let # denote the vocabulary size, s denote the embedding dimension, d denote the hidden size (we use 4s), L the number layers, the layer index (from 0 to L. - 1). we use the following initializations:

---

Embeddings are initialized to (#1 x 1004) as explained in 3.4 For the time-min time-mixing blocks (11, 12, 13), initializations are PA, = ( 1- +, Mu. = - 1 + LI and
0.31
Mr, 2/2 (5)=-
For the channel-mixing blocks (14, 15), Mi; and Mr; are initialized to (4)
Wi (16), also known as "time decay", is initialized to -5 + 8 (d 0.74 L-I, Intuitively, it is the discount factor applied to previous tokens over time.
Us (16), also known as "bonus", is set to 0.5 (((2 +1 mod 3) - 1) + log 0.3. It is the special weighting applied to the current token in equation 16. The alternating zigzag pattern initially
creates subtle variations in the tensor elements, which are intended to help the model treat different dimensions of the embedding distinctively.
W. (17) (time-m (ime-mixing) and W. (chamnel-mixing) are initialized to N V - =)
All other Wr. Wk, W, weights are initialized to so the model can start learning from the beginning
without noisy signals.
All LayerNorm weights start from and biases from O.
F
Small Init Embedding
This section presents the experimental validation of small initialization embedding. The experimental setup is as follows. In the baseline configuration, the parameters are initialized using normal distribution with a mean of 0.0 and a standard deviation of 0.02, which in models like BERT and GPT. On the other hand, in the small initialization of the embedding (small init emb) experiment, the parameters are initialized using a uniform distribution with range of le-4, which is slightly different from RWKV where normal distribution with standard deviation of le-4 is used. However, this difference is negligible and does not affect conclusions. The experiments were conducted with batch size of 400. As depicted in Figure 9, the loss curve for the small init emb exhibits faster rate of decrease and convergence compared to the traditional initialization using a normal distríbution.
commonly used initialization method
11
Baseline
Small Init Emb
10
Loss
4-
10000
20000
30000
40000
50000
Step
Figure Efffect of small initialization emhedding.
G Hyperparameters
To train the models mentioned, we use e (0.9, 0.99) without weight decay for the Adam optimizer, and switch batch size dynamically between 128 or 256 sequences, each of 1024 tokens. We further organize

---

Model
169M
430M
1.5B
3B
7B
14B
Init LR
0.0006
0.0004
0.0003
0.00015
0.00015
0.0001
Warmup Mini-Epochs
361
411
443
451
465
544
End LR
0.00001
0.00001
0.00001
0.00001
0.00001
0.000007
Table 3: Hyperparameters for our learning rate (LR) schedule of the pretrained models.
the training into multiple mini-epochs, cach of 40320 samples, to guide learning rate schedule. The training process takes 8043 mini-epochs to make one pass over the Pile. The initial warming up mini- epochs have a constant learning rate of "Init LR" After the warming up mini-epochs, the learning rate exponentially decays until in the last mini-epoch, in which the model finishes training on the entire Pile, the learning rate arrives at the "End LR". The related hyperparameters are shown Table 3.
H Gradient Stability in RWKV
In this section, we present a mathematical description the gradient stability property in RWKV, focusing specifically on the time-mixing block. By gradient stability we mean that if the inputs 2t are bounded and the model parameters are fixed, then the gradients with respect to W₂ and W, are uniformly bounded for all T (thus not exploding). Consequently, we can control the amount each contributes to the gradient at T in naturally decaying fashion by the weight decay mechanism (thus not vanishing unless desired). First, we make the simplification that there are no token shifts, this will not affect the final conclusion. In this scenario, wkrr can be written as
Σ KE O F(U)) = S(w,)
wkvy =
(29)
Σ1 Kí
S(1)
where
O(vi).
1/- Wott
(De)s.
a(W.)is
0(K));
Kł Weze
= (KF)s
a(W))ss
and S(-) and E(-) are shorthand for denoting sums and averages over weights Kg. The loss function at position T can be written as
LT = ([[(woor),
(30)
Because wker relates (Wi)is and (W,)..j only through the i-th channel (wket),, we have
ØLr H(wker)s a(W)as O(wkor); a(W.)a,
ØLT
(31)
The first part of the above equation contains trivial operations like output layers, and other layers mixe-mixing, which can be proven inductively. The second part of the above equation can bounded as
O (wwkurss
0E (v:ed
a(Wo)ig
a(W.)a,
[<+[(>+))]] < max (attjl
(32)

---

which is irrelevant T. Similarly,
awwkor) Si((ew))) /a(W)) a(Wa)as Si(1)
S(((m))((()))) S.((
S,(1) S,(1)²
co))(( (1r)))
(33)
can also be bounded. Note that wke's softmax operation contains at least two non-zero terms (u and w), so the above "covariance" will not degenerate into O.
I Model Behavior Visualization
The right plot illustrates the time decays ")) each layer of the RWKV-169M model, sorted along the channel axis. Notably, several decays in the last layers are very close or equal to one, implying that certain information is preserved and propagated throughout the model's temporal context. Meanwhile, many decays in the initial layer are close zero, which corresponds lo- cal operations in wkv (16), likely to be associated with tasks such as text parsing or lexical analy- sis. (Note that the local operations in wke are due to the extra parameter u, when e is de- generated into 0.) These patterns of time decays are partly learned, but also come from parameter initialization as it speeds up training.
Time decay (sorted along channel axis)
1.0
Layer
Layer
Tme
Layar
Time
Layer
0.4
Layer
Layer
Layer
0.2
Layer
500 Layer 12
100
200
300
400
800
Charnel
Figure 10: Model behavior visualizations of RWKV.
The plot below shows the information retrieval and propagation path in the RWKV-430M model. The experiment follows the causal trace method introduced by Meng et (2022), where we
1. Run the model once, and record states and activation of each layer during the computation;
2. Corrupt the input embeddings of the subject using noise ("The Eiffel Tower" this example);
3. Restore the states and activation of a certain layer at a certain token during the computation, and record the
of the model outputting the correct answer ("Paris")..
Unlike transformers, RWKV relies on the recursive propagation of information in the time dimension. In this case, the fact that the Eiffel Tower is located in Paris is retrieved in layer 4 just after the model sees "The Eiffel". It is then passed down to the subsequent layers. In layer 20, mostly, the information is propagated through time until reaching where it is needed. Finally, at the token "of". it passed down to the last layer for outputting the answer.
J Additional Evaluations
J.1 Further details on NLP tasks
We evaluate on the following tasks:
ARC (Clark et al., 2018) A dataset designed for multiple-choice question answering, encompassing science exam questions ranging from third grade ninth grade. It has Easy and Challenge subsets that report results separately.
BoolQ (Clark et al., 2019) A binary yes/no question answering benchmark.

---

Information propagation path
The
- 1 - Saaaiin of "Paris"
E
iff
Tower is in the
located
city
of
16
21
Layer
Figure 11: Model behavior visualizations of the RWKV model.
CОРA (Roemmele et al., 2018) A dataset to evaluate achievement in open-domain commonsense causal reasoning.
HeadQA (Vilares and Géómez-Rodriguee 2019) A benchmark consisting of graduate-leve questions encompassing various fields such as medicine, nursing, biology, chemistry, psychology, and pharmacology.
HellaSwag (Zellers al., 2019) A novel benchmark for commonsense Natural Language Inference (NLI) which build by adversarial filtering against transformer models.
LAMBADA (Paperno et al., 2016) A benchmark dataset that evaluates the model's contextual reasoning and language comprehension abilities by presenting context-target pairs, where the objective is to predict the most probable target token. We follow standard practice and use the untokenized version created by OpenAl (Brown al., 2020).
OpenBookQA (Mihaylov et al., 2018) A QA dataset to evaluate human comprehension subject by incorporating open book facts, scientific knowledge, and perceptual common sense, drawing inspiration from open book exams.
PIQA (Bisk et al., 2020) A benchmark for the task of physical common reasoning, which consists of binary choice task that can be better understood set of pairs, namely (Goal, Solution).
ReCoRD (Zhang et al., 2018) benchmark for evaluating commonsense reasoning in reading comprehension by generating queries from CNN/Daily Mail news articles and requiring text span answers from corresponding summarizing passages.
SciQ (Johannes Welbl Nelson F. Liu, 2017) A multiple-choice QA dataset which created using innovative approach to gather well-crafted multiple-choice questions that focused specific domain.
Winogrande (Zellers et al., 2020) A dataset designed to evaluate the acquisition of common sense reasoning by neural language models, aiming to determine whether we are accurately assessing the true capabilities of machine common sense.

---

Cansude (AN
Cancads (HNrECP)
Gong (oaantD)))
(a) ARC (Challenge)
(b) ARC (Easy)
(c) BoolQ
150
200
200
0,20M
(d) COPA
(c) HeadQA
HellaSwag
35
Campute Seaal
Comoud
g
LAMBADA (OpenAl)
(h) OpenBookQA
PiQA
100
360
300
2.00
Computte
Camaute Sventon
ReCoRD
(k) SciQ
(I) Winogrande
Figure 12: Zero-Sho Performance RWKV on common language modeling evaluation benchmarks.

---

J.2 Evaluation Long Range Arena
The Long-Range Arena (LRA) benchmark (Tay et al., 2021) is designed to assess the performance of models in handling lengthy context situations. It includes a collection of tasks with sequences ranging from 1,000 to 16,000 tokens, covering various types of data like text, natural language, synthetic images, and mathematical expressions. We apply RWKV the LRA benchmark and the report results are in Table 4, Other models' performances are directly cited from Gu et al. (2022); Alam et al. (2023).
Table Evaluation on Long Range Arena. Other models reported in the literature (Gu al., 2022; Alam al.,
2023). Bolded values are the best.
MODEL
LISTOPS
TEXT
RETRIEVAL
IMAGE
PATHFINDER
PATH-X
AVG
Transformer
36.37
64.27
$7.46
42.44
71.40
53.66
Reformer
37.27
56.10
$3.40
38.07
68.50
50.56
BigBird
36.05
64.02
59.29
40.83
74.87
54.17
Linear Trans.
16.13
65.90
$3.09
42.34
75,30
50.46
Performer
18.01
65.40
53.82
42.77
77.05
51.18
FNet
35.33
65.11
59.61
38.67
77.80
54.42
Nyströmformer
37.15
65.52
79.56
41.58
70.94
57.46
Luna-256
37.25
64.57
79.29
47.38
77.72
59.37
Hrfformer
39.98
65,38
76.15
50.45
72.17
S4
59.60
86.82
90.90
88.65
94.20
96.35
86.09
RWKV
55.88
86.04
88.34
70.53
58.42
72.07
The results show that RWKV performs second only to the S4 model in five datasets. While RWKV substantially underpreforms S4 Image, Pathfinder, and Path-X, the problems related to natural language and computer code processing RWKV performs on par with S4 or nearly so.
J.3 Enwik8 Perplexity
We also evaluate model in terms of perplexity the Enwik8 dataset. Baseline comparisons made with Reformer (Kitaev et al., 2020), Synthesizer (Tay al., 2020) (the best performing dense version), Linear Transformer (Katharopoulos al., 2020). Performer (Choromanski et al., 2020). L, and T denote the number blocks (network depth), dimension of features, and sequence length, re- spectively. Both Linear Transformer and Performer are implemented with customized CUDA kernels and all other models are implemented native Pytorch. No weight decay nor dropout was used. 2 Trained with Adam W and weight decay set to 0.1, dropout of 0.1, batch size of 16, and initial learning rate of 6e-4.
Method
L
d
T
Train bpc
Test bpc
Time Complexity
Space Complexity
Transformer
12
512
1024
0.977
1.137
O(T²d)
O(T² + Td) O(T Td)
Transformer
24
256
1024
1.039
1.130
O(T2d)
Reformer
12
512
1024
1.040
1.195
O(T log Td)
J[T Td) O(T² Td)
Synthesizer
512
1024
0.994
1.298
O(T2d)
Linear Transformer
1024
0.981
1.207
O(T
O(Td
Performer
512
1024
1.002
1.199
O log
O(T d log O(Td)
AFT-simple
512
1024
1.046
1.209
O(Td)
RWKV-RNN
6
512
1024
0.720
O(Td)
O(d)
RWKV-RNN2
12
512
1024
1.010
1.178
O(Td)
O(d)
Table 5: Enwik8 results, measured in bits per character (bpc).
K Inference results
Figures 13 and 14 illustrate, respectively, the results on time (s) and memory (RAM, VRAM) requirements for LLM inference in float32 precision. We benchmark the following model families and sizes:

---

RWKV: 169m, 430m, 1.4b, 3b, 7b, 14b
Bloom (Scao al., 2022): 560m, lb. 3b
OPT (Zhang et al., 2022): 125m, 350m, 1.3b, 2.7b, 6.7b, 13b
GPT-Neo (Black et al., 2021): 125m, 1.3b, 2.7b
Pythia (Biderman et al., 2023b): 160m, 410m, 1.4b, 2.8b, 6.7b, 12b
sivem
eno--g
15.
513 Lowre
10.
255
Figure 13: Text generation inference memory (CPU RAM, GPU VRAM) for LLMs. Model parameters not accounted.
spC-nvo-OpU
gpl-teo-cuda opt cuda
opc opu -4-
Pythil epu
GPU Sine (4) 512 sakont
PyPhis cuts
200
GPU Uine 012
1500
150
500
80
Parama
4 Paras
Figure 14: Text generation inference time for LLMs.

---

Task Name
Measure
ChatGPT
GPT-4
RWKV-GPT
RWKV-adapted
SOTA
RTE
FI Macro
88.1
91.3
44.2
74.8
92.1
WNLI
Accuracy
81.7
91.6
47.9
49.3
97.9
GoEmotions
FI Macro
25.6
23.1
7.9
7.9
52.8
PolEmo2
F1 Macro
44.1
41.0
38.2
40.9
76.4
Table 6: ChatGPT, GPT-4 and UWKV-4-Raven- 14B reasoning performance comparison in RTE (Wang al., 2019).
WNLI (Wang et al., 2018), GoEmotions (Demszky et al., 2020), and PolEmo2 (Kocoń et al., 2019) benchmarks. RWKV GPT prompts were primarily used for ChatGPT in (Kocoñ al., 2023). SOTA is provided supplementary reference.
Task Name
Measure
ChatGPT
RWKV-adapted
SOTA
Aggression
Fl Macro
69.10
56.66
74.45
MathQA
Accuracy
71.40
5.43
83.20
Sarcasm
F1 Macro
49.88
50.96
53.57
TweetSent
FI Macro
63.32
52.50
72.07
Unhealthy
FI Macro
45.21
43.30
50.96
Table 7: ChatGPT and RWKV-4-Raven-445 performance comparison in Aggresion (Wulczyn et al., 2017), Sarcasm (Siddiqui, 2019). Unhealthy (Price et al., 2020), MathQA (Cobbe al., 2021), and TweetSent (Barbieri et al., 2020) benchmarks. SOTA is provided as supplementary reference.
L Importance of prompt construction and comparison to GPT models
Inspired by Kocoń et al. (2023), we compared the zero-shot performance of the -AWK--4-Raven-- with ChatGPT (access in February 2023) and GPT-4 using several known NLP tasks, i.e., recognizing textual entailment (RTE), Winograd Natural Language Inference (WNLI), and recognizing emotions elicited in readers (Golmmotions and PolLmm22. Each model got the same prompts manually chosen to receive proper responses from the ChatGPT model. As shown in Tab. 6, RWKV performs significantly worse than ChatGPT and GPT-4 several specific tasks. We suspect that this disparity likely caused by the choice of prompts used to generate the answers since the prompts are written in natural language and do not take into account that RWKV, as an RNN, is unable to look back inside an instruction.
When the instruction style was adapted (re-ordered) to respect that RNNs are not capable of "retro- spective processing", the quality may significantly change, e.g., for RTE (Wang et al., 2019) F1 Macro increased from 44.2% to 74.8%. We hypothesize that RWKV models the components in the context, as RNN-based architectures cannot look back and readjust the weight of previous information. For better performance, the desired information should be placed after the main question.
sensitive to the position of
An example ChatGPT prompt for recognizing textual entailment (RTE)
Having premise <here is premise> judge if the following hypothesis <here is hypothesis> is logically connected with the premise. Answer "entailment" if yes, or "not_ entailmmnn" if no.
A re-ordered RWKV prompt for RTE taking into account the nature of the RNN
Can you tell me if the hypothesis entailment or is not entailment to the premise? premise: <here is a premise>
hypothesis:
<here is a hypothesis>
While separating the instruction from the input is relatively easy to do, some other aspects of prompt engineering are harder to quantify. For that purpose, we also tested the approach of stating the input after

---

the question on multiple other tasks, i.e., aggression and sarcasm detection, classification of unhealthy (offensive) lexts, mathematical Q&A, and sentiment analysis, see Tab. The results suggest that better prompts might reduce the disparity between models. Raven achieves comparable results to ChatGPT on unhealthy conversation detection and even surpasses it on the sarcasm detection dataset. While such an approach to prompting looks necessary, it is not enough in itself to replace the capability of having free access to the whole context. Therefore, prompt engineering seems to be significantly more important for the RNN models rather than for standard transformers. It is entirely possible that good prompts to RNN models do not mean additional restrictions, but should simply be constructed using completely different guidelines. The authors of the aforementioned paper (Kocoñ et al., 2023)* perform chain-of-thought improve results on the MathQA dataset. Even including this approach, the Raven model achieved a very low accuracy of 5.43%. Without it, the model performed even worse, performing only very basic and simple calculations and achieving 4.13% accuracy. Raven struggled with questions that required intermediate results. is likely that the order of information presented in the math questions inside the dataset poses a challenge for the RWKV model. is yet to be seen if prompt engineering can address this issue. This further emphasizes the importance of the order of information the model receives.
Template used to prompt the Raven model in MathQA with chain-of sill-fff--ooooe -f-thhu
Write the reasoning and highlight the answer to the question at the end in the format: "Answer: ". The question is: <here is question>
Template used to prompt the Raven model in MathQA without chain-of toab--ff-touuu thought
Write the answer the math question in the format: "Answer: ".
The question is: <here question>
M Cases
this part, we present a few instances of outputs produced by RWKV model using Chat interface
RWKV-4 BWKKV--Raven-- 14B
Tell me about ravens.
Ravens are large, black birds with distinctive white head and neck. They are found in most parts of the world, including North America, Europe, Asia, and Australia. Ravens known for their intelligence and problem-solving abilities. They also considered be symbols of death and transformation in many cultures. Ravens are often associated with the afterlife or death because they have been known to eat carrion or even other birds. In some cultures, ravens are also believed to be messengers of the gods or guardians of treasure.
"This is in line with the idea discussed in (Wei al., 2022b) "ups://github.com cow/BlinDDL/ChaaWA

---

RWKV---Raven1144
Explain the concept of machine learning to a 10-year-old.
Machine learning is like super-smart computer that can learn on its own, It can recognize patterns and make decisions based on what it has learned. Just like how you might remember something from the past, the computer decisions in the future. It's like having smart friend who always knows what do!
remember what it's learned and use it to make predictions
RWKV-4-Raven-11
Write short poem describing nature's beauty in springtime.
Nature's beauty in springtime is rare, A symphony of colors, gentle surprise. The flowers bloom, the birds sing and sing, the sun shines down, warming our skin. The trees sway in the gentle breeze, As we take in their beauty with ease. The sky is blue, the clouds are white, A sight to behold, it's truly divine.

---

RWKV-4-Raven-Ie
Below the numerical stable formula for updating the state variable in RWKV:
a: c² 901-1 + cuI qv
bj c² ab: + c²l 4
wkey ai
Please write out the corresponding pytorch code do so, where both a and are torch. (()
Here is the PyTorch code to update the state variable in RWKV using the given formula:
import torch
def update_state b. u, v, P, q):
a_t torch. ex q) + torch exp(u + k q) *
b_t torch.exp(c exp q) b torch exp(u + k q)
return a_t, b_t a. torch. randn (2. 3)
sorch (2)
torch torch.randn randn (2)
wky = torch.randa randn (2)
a_t, b_t apdate_state (a, b, u, v. wkv, torch randn (2))
print("a_t:", a_t)
print("b_t:", b_t)
This code defines function update_stad stttt that takes in the state variables a and b, the control variables and and the parameters wkv and torch.randn. It then uses the Py Torch torch.exp function compute the exponential of the control variables and the log of the normalization factor. Finally, uses these values to update the state.
RWKV-4-Raven-1 14B
Explain the following metaphor: Life is like cats.
The metaphor "Life is like cats" suggests that life can be unpredictable and full of unexpected twists and turns, just like cat's behavior. It implies that cannot fully control predict the outcome of our lives, just as cannot control the movements of cat. The metaphor also implies that life is complex and multifaceted, with many different variables and factors play. Just cat can have multiple personalities or moods, life can have many twists and turns that are difficult to predict or understand. Overall, the metaphor suggests that life is complex and unpredictable, much like a cat's 's behavior.

---

RWKY-4-Raven- 14B
Test the convergence of Ei=0 ?
The given series is geometric series with the first term ag 2 and common ratio find the sum of this series using the formula for a geometric series:
2' We can
20
0g
2
1
v=
So, the sum of the given series is
2
WWKKV--Raven. 14B
would like to cook some Chinese food at home. Do you have any suggestions on the meal could choose?
Yes, of course! If you are very experienced in cooking Chinese food, you could start with making noodles or rice, then make a dish using one of those. I also think making stir-fry with some meat is an excellent idea.
want to make something more special. What about red braised pork belly?
Yes, red braised pork belly is also very popular and tasty.
Tell me the steps to cook.
You will need a few ingredients such as pork belly, star anise, sugar, ginger, scallions and wine. Put all of these pot, cook on medium heat and simmer until the meat is tender. After that, remove the meat from the pot and reserve the sauce. Slice the meat and add some ginger slices into it, then boil until it's almost done.
What do I need to be careful with the most during the cooking process?
To cook pork belly properly, it is very important monitor the heat of the stove and be patient with the cooking process. Overcooking it can cause the pork to be tough, dry and difficult chew. You can add few ingredients like soy sauce or black beans to give the pork more flavor.
Tell me more about the proper heat and timing.
If the temperature of the water is too high, it can damage the collagen and connective tissues, resulting in tough pork belly. It is essential to control the temperature carefully, especially when braising the pork. Additionally, keep the water simmering but not boiling maintain low simmer, as high heat will dry out the pork belly and result in tough and unpalatable meat. Lastly, monitor the heat during cooking and make sure the heat is medium, as this what is most appropriate for slow cooking pork belly.
see. Thank you for your advice.
You're welcome! It was pleasure chatting with you. Let me know if you need any further assistance in the future.

---

RWKV-4-Raven-444
use the concept of If all apples are fruits and all fruits are sweet, are all apples sweet? Please explain your answer in detail.
Yes, all apples are fruits, and all fruits are sweet. Therefore, it is safe to assume that all apples are sweet.
If birds can fly. Penguins are birds. Can penguins fly? Please explain your answer in detail.
No. penguins cannot fly. Penguins are birds, but they are not capable of flight like other birds. They have wings and can glide for short distances, but they cannot take off or fly long distances like other birds.
The day before two days after the day before tomorrow is Saturday, What day is it today? Please explain your answer in detail.
Today is Friday. Two days after tomorrow Sunday. Tomorrow is Saturday.
If five cats can catch five mice in five minutes, how long will it take one cat to catch one mouse? Please explain your answer in detail.
If five cats can catch five mice in five minutes, it means that each cat can catch one mouse in one minute. To find out how long it will take cat to catch mouse, proportionality.
Let's assume that it takes x minutes for cat to catch a mouse. Then, the number of mice caught by a cat in one minute is equal to the number of cats multiplied by the number of mice caught by each cat: x = * = So, it will take 5 minutes for one cat to catch one mouse.
RWKV-4-Raven1144
Summarize the following paragraph.
<The Abstract this paper goes here>
The paper presents a novel architecture, Receptance Weighted Key Value (RWKV), that combines the efficient parallclizable training of Transformers with the efficient inference of RNNs. The model is able to scale to tens of billions of parameters and exhibits linear computational complexity during training and inference, making it promising alternative Transformers for sequence processing tasks.
Please further summarize it in one sentence.
The researchers have developed a new architecture that combines the efficient parallelizable training of Transformers with the efficient inference of RNNs, which allows for linear scaling in memory and computational requirements.


Reviews:
[{"review_id": "OLuYB1cvRa", "reviewer": "Reviewer_Zd3h", "summary": "", "questions": "If both long and short contexts require the same fixed amount of memory space, what implications could this have on the performance when dealing with long contexts?\nHow does RWKV compare to other non-Transformer architectures for sequence processing tasks?\nHow does RWKV scale to even larger models?\nHow does RWKV perform on tasks that require long-range dependencies, such as machine translation?\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough and careful review. Please find below our response addressing the raised concerns:\n\n> It is not as flexible as Transformers. Transformers can be used for a wider variety of tasks, and they are more adaptable to different input and output formats.\n\n>Transformers can be used for a wider variety of tasks, and they are more adaptable to different input and output formats. Transformers are able to learn more complex relationships between input and output sequences than RWKV. This is because Transformers' attention mechanism allows them to attend to arbitrary positions in the input sequence, while RWKV's linear attention mechanism is only able to attend to certain positions.\n\nAlthough we agree that the attention mechanism of transformers can indeed learn more complex relationships than RWKV in its current form, we believe our work is publishable for two reasons:\nWe tackle one of the most-studied domains of transformers, language modeling, using an alternative architecture. This architecture is in line with other research results that showed that the attention matrix, learned by transformers in language modeling, is low-rank and could be approximated by simpler alternatives. Essentially, our work proposes RNNs as a good base architecture for language modeling and uses language modeling to demonstrate this.\nThese are initial results, and we expect RWKV to improve over time. While the causality restriction is inherent to the RWKV architecture, we believe that RWKV can be extended to other tasks and input/output formats, and will explore this in future work.\n\n> If both long and short contexts require the same fixed amount of memory space, what implications could this have on the performance when dealing with long contexts? How does RWKV compare to other non-Transformer architectures for sequence processing tasks? How does RWKV scale to even larger models? How does RWKV perform on tasks that require long-range dependencies, such as machine translation?\n\nWe discuss the performance of RWKV in its current form in the second paragraph of “Limitations”. \nWe leave the comparison of RWKV to other non-transformer architectures to future work\nWe scale RWKV across a wide range of model sizes, and use this to generate scaling laws that demonstrate RWKV follows the broad scaling trend of transformers (section 5.2). We did not train larger models, but can reasonably extrapolate their performance trend compared to trained models.\nWe evaluated RWKV on the long range arena (LRA) benchmark in Appendix G.2, and found that it performs second only to the S4 model.\n\nOnce again, we thank the reviewer for the thoughtful review and are open to answering any further questions or receiving additional feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "RWKV is good work because it is the initial step to fill the gap between RNN and Transformer and it is a novel model architecture that combines the efficient parallelizable training of Transformers with the efficient inference of RNNs. In addition, RWKV performs on par with similarly sized Transformers on a variety of natural language processing tasks, suggesting that it is a promising new architecture for sequence processing tasks.", "reasons_to_reject": "It is not as flexible as Transformers. Transformers can be used for a wider variety of tasks, and they are more adaptable to different input and output formats. Transformers are able to learn more complex relationships between input and output sequences than RWKV. This is because Transformers' attention mechanism allows them to attend to arbitrary positions in the input sequence, while RWKV's linear attention mechanism is only able to attend to certain positions.", "questions_to_authors": "If both long and short contexts require the same fixed amount of memory space, what implications could this have on the performance when dealing with long contexts?\nHow does RWKV compare to other non-Transformer architectures for sequence processing tasks?\nHow does RWKV scale to even larger models?\nHow does RWKV perform on tasks that require long-range dependencies, such as machine translation?\n", "paper_topic_and_main_contributions": "The paper discusses a novel model architecture called Receptance Weighted Key Value (RWKV) that combines the training efficiency of transformers with the inference efficiency of recurrent neural networks (RNNs). The approach utilizes a linear attention mechanism, allowing the model to be formulated as either a Transformer or an RNN. This enables parallel computations during training and maintains constant computational and memory complexity during inference. The authors conducted experiments with models containing up to 14 billion parameters and found that RWKV performs similarly to similarly sized Transformers. The main goal of this work is to find a balance between computational efficiency and model performance in sequence processing tasks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tsqVGIcRja", "reviewer": "Reviewer_rSzx", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The authors acknowledge an information bottleneck problem in the proposed architecture, as information from past context must flow through fixed-size hidden vectors. This issue might be reflected in the experiments in Appendix I, showing that the model is more sensitive to the prompt. This bottleneck could negatively impact tasks that require more fine-grained attention.\n\nWe acknowledge this shortcoming, and hope to alleviate this issue in future iterations of RWKV.\n\n> The time mixing component, while parallelizable along other dimensions, is not parallelizable in the time dimension. This lack of parallelization could become a training bottleneck for very long context windows.\n\nNeither RWKV nor attention-based architectures GPT can improve in the time dimension beyond O(T) where T is the sequence length. Therefore, either both RWKV and GPT are parallelizable in the time dimension, or they both are not. We note that RWKV has a notable decrease in time and space complexity as T increases compared to competing architectures (see table 1), and this is a key strength of our approach.\n\n> Two specific tasks (ReCoRD and Winogrande, as shown in Figure 5) see the model underperforming other models. This underperformance requires further investigation.\n\nWe agree with this shortcoming and are actively investigating. \n\n> A significant potential benefit of an RNN-like formulation is its applicability to longer contexts, but Figure 6 appears to limit this method to up to 2^12=4096 tokens. Further exploration of context length scaling is desirable. Additionally, most of Figure 6 is unsurprising, as more context naturally results in lower perplexity within the context window size. The figure's x-axis should start with the context window size being trained on. Clarification on the context size being fine-tuned up to would also be beneficial.\n\nWe agree that exploring even longer contexts would be an interesting result, and we have trained models up to 128k context length. These results were taken after the anonymity period and will be announced more broadly after it ends.\nFor previous attention-free models such as LSTMs, performance sometimes doesn’t improve beyond ~100 tokens. Our result verifies that RWKV doesn’t have this limitation.\nFor the long-context finetuning experiments in section 6.2, we used the following context lengthening finetuning schedule for both RWKV 7B and 14B:\n\n1024 -> 2048 for 10B tokens\n2048 -> 4096 for 100B tokens\n4096 -> 8192 for 100B tokens\n\nThen these finetuned models are evaluated on the Pile test set in Figure 6. We have explicitly noted this in Section 6.2 for the final paper version.\n\n> L126:a computationally efficient alternatives, L136:Simultaneously with this work, (Poli et al., 2023): citep -> citet\n\nThanks for catching these! They have been fixed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed architecture's comparable training speed to transformers, much faster inference, lower memory footprint, and potentially unlimited context window size make it an appealing alternative to transformers.\n2. Empirical evidence demonstrates that the proposed method works well on large-scale models and training data.", "reasons_to_reject": "1. The authors acknowledge an information bottleneck problem in the proposed architecture, as information from past context must flow through fixed-size hidden vectors. This issue might be reflected in the experiments in Appendix I, showing that the model is more sensitive to the prompt. This bottleneck could negatively impact tasks that require more fine-grained attention.\n2. The time mixing component, while parallelizable along other dimensions, is not parallelizable in the time dimension. This lack of parallelization could become a training bottleneck for very long context windows.\n3. Two specific tasks (ReCoRD and Winogrande, as shown in Figure 5) see the model underperforming other models. This underperformance requires further investigation.\n4. A significant potential benefit of an RNN-like formulation is its applicability to longer contexts, but Figure 6 appears to limit this method to up to 2^12=4096 tokens. Further exploration of context length scaling is desirable. Additionally, most of Figure 6 is unsurprising, as more context naturally results in lower perplexity within the context window size. The figure's x-axis should start with the context window size being trained on. Clarification on the context size being fine-tuned up to would also be beneficial.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes an alternative architecture to transformers, inspired by RNNs, that is more memory- and compute-efficient. The main idea is to replace the quadratic complexity self-attention mechanism with a recurrent time mixing mechanism. While this time mixing component is not parallelizable over the sequence length dimension during training, it is parallelizable over the hidden dimension, enabling efficient training comparable to that of transformers. This work trains models of up to 14B parameters on the PILE dataset, showing comparable performance to transformers on various tasks. Experiments also confirm that the trained models exhibit linear time scaling concerning the number of tokens generated, as opposed to the quadratic scaling of existing transformer LMs.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. L126:a computationally efficient alternatives. \n2. L136:Simultaneously with this work, (Poli et al., 2023): citep -> citet"}, "extra_text": {}}, {"review_id": "jL04GnUxUW", "reviewer": "Reviewer_85wr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review! Please find below our response addressing the raised concerns:\n\n> Figure 1 needs to have actual references to datasets and calculations. Having unlabeled graphs is not okay in a published paper. Languages need to be provided as well (BLOOM is multilingual, are these English tasks?)\n\nWe will add references for the final paper version. All languages are english unless explicitly stated otherwise.\n\n> My understanding is that RWKV is roughly equivalent to the AFT local model that was previously presented. Yet this is not mentioned in the paper and the table does not include this key property. Is this an oversight or am I missing something?\n\nAFT and RWKV are indeed overall similar, but differ in a few key ways. We compare these exact differences between architectures in section 4.1, but at a high level AFT learns a decay for each pair of locations, where in the local approach if the distance between two locations is higher than the kernel size this decay is 0. RWKV uses exponential decays that decreases with the distance. Other key differences are the relative positional information (which makes RWKV recurrent) and RWKV’s token-shift.\n\n> All the main information in the paper is shown in graphs in terms of scaling. While I understand why the authors want to show their model in this way, as a reader I want to see standard tables showing tokens / ppl (or bpc). Please include these tables in the paper so I can understand the data efficiency without trying to extrapolate from tables.\n\nWe will include these tables in an appendix for the final paper version\n\n> While the pen-and-paper FLOP calculations are interesting, would be curious to understand how the actual training time compares on real hardware. Some graphs in the main paper would help.\n\nWe agree that these results should be included. We will try to dig through experimental results and compare the actual training times, but if this can’t be reliably done (some results are not explicitly timestamped), we will have to leave this for future work.\n\n> There are many different extensions to a raw RWKV architecture presented in this paper, but there are not any ablations in the body showing whether these were helpful or not in terms of loss.\n\nWe agree that ablations would be useful, and will attempt to collect the data necessary to add these figures for the final paper version. If we can’t collect the data in time, these results will be left for future work.\n\n> I do not understand the units or experimental setup of figure 6. Would love a transformer baseline as well to understand the curve.\n\n We agree that Figure 6 appears unclear and we will improve it for the final paper version.\n\n> Small:\n\nThese minor changes will be fixed for the final paper version. Thank you for tracking these down!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This work should be accepted. The team here has tried something hard and ambitious, and succeeded in scaling this model to the point of good performance. The data points here are extremely valuable and I think this paper should be presented in an NLP venue. I have many issues with the form of the presentation, but I do not think my concerns should be seen as a vote for rejection. \n\n* Section 4 is very nicely presented. Compared to the github for this project which is very challenging to understand, this section presents a coherent, readable and implementable description of the approach and its usage. \n", "reasons_to_reject": "Given the above, I hope the authors take these concerns into account. There are many aspects of the work that I think are quite problematic, and without the scale and ambition of the other parts I would have not wanted it accepted. In general this paper reads like an advertisement and not scientific research. Captions and experimental detail is lacking throughout.  I hope they take the time to fix them. \n\n* Figure 1 needs to have actual references to datasets and calculations. Having unlabeled graphs is not okay in a published paper. Languages need to be provided as well (BLOOM is multilingual, are these English tasks?)\n\n* My understanding is that RWKV is roughly equivalent to the AFT local model that was previously presented. Yet this is not mentioned in the paper and the table does not include this key property. Is this an oversight or am I missing something? \n\n* All the main information in the paper is shown in graphs in terms of scaling. While I understand why the authors want to show their model in this way, as a reader I want to see standard tables showing tokens / ppl (or bpc). Please include these tables in the paper so I can understand the data efficiency without trying to extrapolate from tables. \n\n* While the pen-and-paper FLOP calculations are interesting, would be curious to understand how the actual training time compares on real hardware. Some graphs in the main paper would help. \n\n* There are many different extensions to a raw RWKV architecture presented in this paper, but there are not any ablations in the body showing whether these were helpful or not in terms of loss.\n\n* I do not understand the units or experimental setup of figure 6. Would love a transformer baseline as well to understand the curve.\n\nSmall: \n\n* Figure 2 / 3 show the same thing roughly 3 times. At first read this was a bit confusing.\n\n* How were the tasks chosen for the various analysis plots? Are these random or representative in some manner?\n\n* Generally the graph labels are much too small to read, please increase these to be similar to the text itself. \n\n* Can you provide more details on exactly the inference method / software hardware used for the text generation results? From the text it is unclear whether it is even cpu or gpu. \n\n* How close do you think this model is to reaching the FLOP values presented in this work? Is this theoretical or is it comparable in a wall clock sense?\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work presents RWKV an evolving model / open-source architecture that is meant to rival transformers while providing RNN-like compute. As the authors note this is the largest RNN model trained to date in NLP and shows strong and compelling performance. The paper contextualizes this model among the other models being studied and described in this space, and the historical models that people have proposed. Results show that there is compelling performance for the model that makes it worthy of continual study.", "excitement": 5, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eIUItBWVpE", "reviewer": "Reviewer_HDNB", "summary": "", "questions": "- From the experiment results of MathQA, it seems RWKV performs poorly. Is those tasks need hard reasoning a intrinsic weakpoint of RWKV?\n- How does the RWKV perform on long inputs?\n- The paper mentions many techniques, are these universal across all sizes?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Recent large language models are using float16 or bfloat16 precision, it will be great to see RWKV also works in these precisons.\n\nAll RWKV models were trained using bf16. We will make this explicit in the paper.\n\n> From the experiment results of MathQA, it seems RWKV performs poorly. Is those tasks need hard reasoning a intrinsic weakpoint of RWKV?\n\nWe briefly discuss the poor MathQA performance in Appendix I. In brief, we believe it may be caused by the order of information presented in the math questions of MathQA, and it’s unclear at this stage whether this issue is solvable by prompt engineering.\n\n> How does the RWKV perform on long inputs?\n\nWe evaluate RWKV’s performance on long contexts in section 6 (see 6.2 in particular), and evaluate on the long-range arena benchmark in Appendix G.2\n\n> The paper mentions many techniques, are these universal across all sizes?\n\nYes. All model architecture techniques listed in section 4 are model-size agnostic, and should benefit the model at any scale. The extent to which each technique benefits the loss at each scale is currently unknown, however, and an ablation study will be performed in future work to make this relationship explicit.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper introduces RWKV, a groundbreaking architecture that effectively combines the strengths of Recurrent Neural Networks (RNNs) and Transformers while addressing their respective limitations. \n- Detailed experiments demonstrates exceptional performance and efficiency on benchmark datasets for large-scale models, making it a promising solution for various natural language processing tasks. \n- The authors release a set of pretrained models with varying parameter sizes, ranging from 169 million to 14 billion. This is useful for community research on high capacity modeling.", "reasons_to_reject": "Recent large language models are using float16 or bfloat16 precision, it will be great to see RWKV also works in these precisons.", "questions_to_authors": "- From the experiment results of MathQA, it seems RWKV performs poorly. Is those tasks need hard reasoning a intrinsic weakpoint of RWKV?\n- How does the RWKV perform on long inputs?\n- The paper mentions many techniques, are these universal across all sizes?", "paper_topic_and_main_contributions": "- Transformers have significantly advanced various natural language processing (NLP) tasks. However, they suffer from high memory and computational complexity that grows quadratically with sequence length. On the other hand, recurrent neural networks (RNNs) exhibit linear scaling in memory and computational requirements but struggle to achieve the same performance as Transformers due to limitations in parallelization and scalability.\n\n- To address these issues, the paper proposes a new model architecture called Receptance Weighted Key Value (RWKV). This model combines the efficient parallelizable training of Transformers with the efficient inference of RNNs. RWKV utilizes a linear attention mechanism, enabling us to formulate the model as either a Transformer or an RNN. This allows for parallel computations during training and maintains constant computational and memory complexity during inference.\n\n- RWKV has successfully been scaled up to 14 billion parameters, making it the largest dense RNN ever trained. Remarkably, RWKV performs on par with similarly sized Transformers, indicating that this architecture could lead to more efficient models in the future. This work represents a significant step towards balancing the trade-offs between computational efficiency and model performance in sequence processing tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary of the paper: RWKV is the largest RNN model trained to date in NLP that rivals transformers in performance. The results show that the model has impressive performance, making it a worthwhile subject of further study.

This manuscript has a lot of positives. The idea presented in the paper is very ambitious and relevant to the NLP community. The proposed method has comparable training speed as compared to transformers with much faster inference and lower memory footprint. 

Some of the core criticisms of the paper are on the empirical evaluations and the paper not being well written. Multiple details are missing including experiments such as actual compute time comparison, evaluation beyond 4K token length to showcase the use of RNN style method. RNN style methods trade-off accuracy and compute time (because of information bottleneck), an evaluation of this trade-off would be an interesting addition.
Decision: Accept-Findings