Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence

Authors: bo  peng
  eleutherai
daniel  goldstein
quentin  anthony
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 19:42:32

Abstract: We present Eagle (RWKV-5) and Finch (RWKV-6), sequence models improving upon the RWKV architecture. Our architectural design
advancements include multi-headed matrix-valued states and a dynamic recurrence mechanism that improve expressivity while maintaining the inference efficiency characteristics of RNNs. We introduce a new multilingual corpus with 1.12 trillion tokens and a fast tokenizer based on greedy matching for enhanced multilinguality. We trained four Eagle models, ranging from 0.46 to 7.5 billion parameters, and two Finch models with 1.6 and 3.1 billion parameters and find that they achieve competitive performance across a wide variety of benchmarks.

Main paper: 
 introduction: 
 Advancements in Large Language Models (LLMs) have significantly impacted Natural Language Processing (NLP) tasks. The field has traditionally been dominated by the transformer architecture (Vaswani et al., 2023) . However, the expressive attention mechanism of transformers leads them to suffer from quadratic time complexity with respect to input sequence length. Various methods have been proposed to achieve sub-quadratic time complexity without significantly changing the core attention mechanism, typically relying on some form of sparsity techniques (Child et al., 2019a; Beltagy et al., 2020; Zaheer et al., 2020) .Recent works have achieved sub-quadratic time complexity without significantly sacrificing performance by introducing new mechanisms to replace attention at the core of the Transformer architecture. These models include gated recurrences (Fu et al., 2023; Gu & Dao, 2023; Gu et al., 2021; Sun et al., 2023; Katsch, 2023; Qin et al., 2023; Smith et al., 2023a) , gated convolutions (Poli et al., 2023; Peng et al., 2023) , and data-dependent linear attention (Yang et al., 2023; Katharopoulos et al., 2020b) , sparse attentions (Tay et al., 2020; Child et al., 2019b; Zaheer et al., 2020; Qiu et al., 2019) and their combinations (De et al., 2024; Qin et al., 2024; 2022b) . We build off RWKV-v4 introduced in (Peng et al., 2023) , which provides efficient inference and training along with a parallelizable implementation compared to competing architectures as shown in Table 1 .Inference Training Time Memory Parallel Time Memory In this paper, we introduce two new architectures: Eagle (RWKV-5) and Finch (RWKV-6). First, Eagle improves upon the architecture and learned decay schedule from RWKV-4 (Peng et al., 2023) through the use of expressive multi-headed matrix-valued states (as opposed to vector-valued states), a reformulated receptance, and an additional gating mechanism. Finch further improves the expressivity and flexibility of the architecture by introducing new data-dependent functions for both the time-mixing and token-shift modules, consisting of parameterized linear interpolations. Additionally, Finch proposes a novel use of the Low Rank Adaptation (Hu et al., 2022) function to allow for trainable weight matrices to efficiently augment the learned data decay vectors in a context-dependent manner. Finally, we introduce a new tokenizer, the RWKV World Tokenizer, and a new dataset, RWKV World v2 (1.12 trillion tokens), manually designed to improve performance on multilingual and code data.LSTM/LMU O(1) O(1) ✗ O(N) O(N) Transformer O(N) O(N) a ✓ O(N 2 ) O(N) b Linear Transformer O(1) O(1) ✓ O(N) O(N) H3/S4 O(1) O(1) ✓ O(N log N) O(N) Hyena O(N) O(N) ✓ O(N log N) O(N) RWKV/Mamba/RetNet O(1) O(1) ✓ O(N) O(N)Through extensive experimentation, we show that the Eagle and Finch models perform competitively, or improve upon existing models under a wide variety of sequence modeling domains and tasks. Specifically, we evaluate our trained models on commonly used Englishonly and multilingual text benchmarks, associative recall, music modeling, and visionlanguage benchmarks. Our experiments demonstrate that the advancements in Eagle and Finch provide significant progress towards developing more efficient AI models.In summary, our main contributions are:• The Eagle (RWKV-5) and Finch (RWKV-6) RWKV architectures, which significantly improve over RWKV-4 on benchmarks for LLMs.• The RWKV World Tokenizer which contains underrepresented languages' vocabulary and which performs fast tokenization with Trie-based greedy matching. • The RWKV World v2 public dataset, comprised of 1.12 trillion tokens of publicly available multilingual data. • Public release of four pre-trained Eagle models, scaling from 0.46 to 7.5 billion parameters, and two Finch models, with 1.6 and 3.1 billion parameters. Demonstrating that these novel architectures are competitive to transformers when trained using enough FLOPs to make meaningful scaling conclusions.  
background: 
 Eagle and Finch are RNNs based on a multi-headed hybridization of the RWKV-4 architecture and linear attention. We discuss related work and the evolution of these two architectures below, with a more detailed review given in Appendix C.Recurrent Neural Networks (RNNs) are well suited to provide inexpensive inference on sequence modelling tasks, typically operating in O(1) time complexity per step with respect to sequence length. They model sequences with time dependencies by generating a hidden state h t at each time step, which is fed back in at the next time step as a secondary input. Classic RNNs (e.g. LSTM (Hochreiter & Schmidhuber, 1997) and GRU (Cho et al., 2014) ) became widely used for sequence modelling, but are difficult to parallelize across the time dimension for training.The Transformer architecture has enjoyed remarkable success in generative modelling, particularly language modelling (Vaswani et al., 2023; Radford et al., 2018) , providing SOTA performance across many tasks. However, the use of multi-headed dot-product selfattention (MHA) leads to quadratic time complexity with respect to sequence length. The deficiencies of classic RNNs and Transformers led to many attempts to develop architectures incorporating the best features of both in a single model, namely O(1) time per token and fast highly parallelizable training.Linear Attention (Schmidhuber, 1992; Katharopoulos et al., 2020a) replaces the numerator of MHA's softmax(QK T )V with ϕ(Q)ϕ(K) T V, allowing a reordering of operations via associativity to ϕ(Q)(ϕ(K) T V), where ϕ represents a non-negative feature-map function. It can be computed as an RNN in O(1) time per step by adding ϕ(K T i )V i to a recurrent state at each time step i, or trained in parallel much like MHA. This accomplishes the main goals outlined above, but naive linear attention suffers from significantly reduced performance compared to MHA-based transformers.A modified form of linear attention, the Attention Free Transformer (AFT) (Zhai et al., 2021) , paved the way for the RWKV architecture, by using a number of attention heads equal to the size of the feature dimension and incorporating a set of learned pairwise positional biases, denoted as w.EQUATIONRWKV-4 reformulates the AFT equation by replacing the pair-wise positional biases with a channel-wise vector of additive weight decay rates w. It also adds a bonus term u to offset the weight of the current input specially.EQUATION)RWKV-4 also adds token-shift and gating to both attention and feed-forward sub-blocks of transformer, and small embedding initialization and normalization to quickly arrive at welldistributed token embeddings. Combining all of these architectural changes led RWKV-4 to become the first RNN to rival the performance of Transformers, while maintaining fast parallelizable training and O(1) time complexity per token.There has been a recent revival of RNNs in NLP research (Tiezzi et al., 2024) . HGRN (Qin et al., 2023 ) is a recent time-parallelizable data-dependent RNN that employs input and forget gates. TransNormer (Qin et al., 2022a) applies RMSNorm to linear attention to bound its output. Other new time-parallelizable data-dependent RNNs have also been invented concurrently with our work including GLA (Yang et al., 2023) and Griffin (De et al., 2024) .State Space Models (SSMs) employ a hidden state of basis function weights to model an approximation of the input function (Gu et al., 2020) , updating that hidden state via a differential equation. Earlier SSMs (Gu et al., 2022) were historically computed using long convolutions in O(N log N) time per sequence, but could also be formulated as a recurrent network. Recently, it has been shown that SSMs can be parallelized across the time dimension via techniques including associative scan (Smith et al., 2023b) . A new class of SSMs has also emerged concurrently with our work (Katsch, 2023; Gu & Dao, 2023 ) that feature data-dependent A and B terms, which function similarly to the data-dependent dynamic recurrence used in Finch. belonging to R (D/h) . For simplicity we show calculations per-head, eliding the head index. We use the convention that all vectors are row vectors unless explicitly transposed, so all matrices operate on the right side. We use the square subscript to denote a variable. 
eagle: 
 Eagle Token Shift We adopt the Token Shift technique from the previous RWKV, similar to a 1D causal convolution of size = 2, as can be seen in Figure 1 , center-bottom. To better introduce the Token Shift technique, we define some notation. The linear interpolation (lerp) between x t and x t-1 used in RWKV-4 and Eagle Token Shift is defined as:lerp □ (a, b) = a + (b -a) ⊙ µ □ (3)where each µ □ ∈ R D is a learnable vector.Token Shift allows the model to learn how much new versus old information should be allocated per time step to each channel of receptance, key, value, and gate vectors (r, k, v, and g respectively) independently and uniquely for each head. This makes it possible to form induction heads (Elhage et al., 2021) within a single layer since even a single head can directly accumulate both past and current token data into separate subspaces within these vectors.The formula of Eagle Time Mixing can be written as follows:□ t = lerp □ (x t , x t-1 )W □ , □ ∈ {r, k, v, g} (4) w = exp(-exp(ω))(5)wkv t = diag(u) • k T t • v t + t-1 ∑ i=1 diag(w) t-1-i • k T i • v i ∈ R (D/h)×(D/h) (6) o t = concat (SiLU(g t ) ⊙ LayerNorm(r t • wkv t )) W o ∈ R D (7)Where LayerNorm operates on each of h heads separately, which is also equivalent to the GroupNorm (Wu & He (2018) ) operation on h groups. It is also worth noting that w is obtained from w = exp(-exp(ω)), where ω ∈ R D/h are the actual headwise trainable parameters. This ensures that w falls within the interval (0, 1), guaranteeing that diag(w) is a contraction matrix.The wkv t attention calculation can alternatively be written in a recurrent form:wkv ′ = s + diag(u) • k T • v (8) s ′ = diag(w) • s + k T • v (9)RWKV's wkv term can be considered a decay-based equivalent to the normalised k T v term in Linear Attention. It is instructive to note how for a given head j the recurrent state s is a sum of k T v where each channel of s individually decays by the corresponding channel of w at each time step. Prior to the application of the receptance vector, gating, and output weights, a per-channel learned boost u is multiplied with the current token's k T v and summed with the state, as can be seen in Figure 1 , top-right. This gives the current token special treatment relative to the sum of past tokens contained within the decaying state history. The receptance is multiplied by this sum, acting like the query term in Linear Attention.In both Eagle and Finch, the Channel Mixing module is identical to the previous RWKV-4 architecture, except for a slightly reduced hidden dimension from 4D to 3.5D. This reduction accounts for new gating weights in Eagle Time Mixing to ensure an equi-parameter relation with the prior model at the same number of layers and embedding dimension. We do not further reduce the hidden dimension in Finch despite adding a small number of new parameters for LoRA weights. The formulas for Channel Mixing are the same as RWKV-4, but we restate them here to ensure notational consistency, using linear interpolation from Equation 3:r ′ t = lerp r ′ (x ′ t , x ′ t-1 )W r ′ ∈ R D (10) k ′ t = lerp k ′ (x ′ t , x ′ t-1 )W k ′ ∈ R 3.5D (11) v ′ t = ReLU(k ′ t ) 2 W v ′ ∈ R D (12) o ′ t = σ(r ′ t ) ⊙ v ′ t ∈ R D (13)Finch Token Shift The data-dependent linear interpolation (ddlerp) between x t and x t-1 used in Finch Token Shift is defined as:EQUATIONwhere µ x and each λ □ introduce a trainable vector of dimension D and each A □ ∈ R D×32 , B □ ∈ R 32×D introduce new trainable weight matrices. For the special case of LoRA ω seen below we introduce double-sized trainable weight matrices A ω ∈ R D×64 , B ω ∈ R 64×D . A schematic representation can be found in Figure 1 , bottom-right.This new form of Token Shift enhanced with data-dependence is intended to expand the abilities of the model beyond the RWKV-4/Eagle style of Token Shift so that the amount of new and old data allocated per channel now depends on the input at both current and prior time steps.EQUATIONEQUATION)EQUATION)EQUATIONThe wkv t attention calculation can alternatively be written in a recurrent manner:wkv ′ = s + diag(u) • k T • v (21) s ′ = diag(w) • s + k T • v (22)Unlike in Eagle, w t here is not static across the sequence (dashed arrows in Figure 1 , left and top-right.). This is the core change to decay in Finch, as each channel of w t can now vary independently over time, in a data-dependent manner, whereas previously it was a fixed learned vector.The new LoRA mechanisms above are used to take learned vectors, as seen in Eagle, and inexpensively augment them with additional offsets determined by the incoming input. Note that the LoRA process itself uses an Eagle style Token-Shifted value as its input, not just the latest token. The new time-varying decay w t goes one step further, applying LoRA again afterward. Intuitively, this is a second-order variant of Token-Shifting, allowing each channel of w t to vary based on a mix of the current and prior tokens, with the mix itself determined by aspects of both tokens. 
artifacts: 
 We release our tokenizer, dataset, and model weights under an Apache 2.0 licenseTokenization is important in language modelling as it conditions the learning relationships between tokens and the generation of new text based on those patterns. The numbers of tokens to build a single semantic chunk are, however, often very unequally distributed against non-European and other underrepresented languages. Byte-pair-encoding (BPE) based tokenizers which are trained with this inequality result in not only lower performances against underrepresented languages but also undue economic costs such as inference Ahia et al. (2023) and continual pre-training with extended vocabulary Lin et al. (2024) ; Sasaki et al. (2023) . To address these problems, we manually select tokens from multiple vocabulary files to well-represent non-European languages (Details in Appendix F). This tokenizer is implemented via a Trie (Prefix Tree) to boost speed while maintaining simplicity. Encoding is performed as matching the longest element in vocabulary with an input string from left to right. Our tokenizer's vocabulary construction aims to mitigate undue burden, which naive BPE and related methods cause on minority languages.We train our models on the new RWKV World v2 Dataset, a new multilingual 1.12 trillion token dataset drawn from a wide variety of hand selected publicly available data sources.This dataset is designed to go beyond the English-heavy focus of many datasets widely used to train LLMs today. We do this to support usage by the majority of the worldwide population who are not native English speakers, to improve representation within model responses, and also to enable transfer learning so that our models can apply knowledge across cultures and locales. We put a strong emphasis on factual knowledge and code, but also on cultural works including stories, books, subtitles, and conversations. The source data is approximately 70% English, 15% multilingual, and 15% code. We describe the components of our dataset in detail in Appendix D.We have pre-trained and publicly released the six Apache 2.0 licensed Eagle and Finch models: Eagle 0. Summation formula: ∑a_i = a_0 + (a_1 -1) + (a_2 -1) + ... + (a_n -1) + ... 6. Continue this process until you reach n terms: ∑a_i = a_0 + (a_1 -1) + (a_2 -1) + ... + (a_{n-1} -1) + a_n 7. Count how many terms there are in this sum:• For n=0: There are 0 terms in the series, so ∑ai will equal a0.• For n=1: There is 1 term in the series, so ∑ai will equal a1.• For n=2: There are 3 terms in the series, so ∑ai will equal a2 + (a_3 -1).By following these steps for each value of n, we can determine how many terms there are in the series and compare it with the amount by which it exceeds 1. In this case, after 5 terms we have exceeded 1, meaning that the series diverges: Summation formula: ∑a_i = a_0 + (a_1 -1) + (a_2 -1) + (a_3 -1) + ... + (a_{5} -1) Thus, we have found that this series does not converge, as 5 is greater than 0 and less than or equal to 4 (which corresponds to the amount by which it exceeds 1). Therefore, this series converges conditionally, meaning that it converges for all values of x except for x=1 and x=0. 
language modeling experiments lm evaluation harness benchmarks: 
 To assess the performance of Eagle and Finch models, we evaluate on a series of common multi-lingual and English-focused benchmarks using lm_evaluation_harness (Gao et al., 2023) as shown in Tables 3 and 4 . Following the advice of Biderman et al. (2024) , we reran evaluations of models that use a different evaluation framework in their papers.Both Eagle and Finch demonstrate exceptionally high capabilities on multi-lingual benchmarks, with nearly all results significantly outperforming the other similarly sized models we tested.In Appendix G.1, we plot the accuracy versus FLOPs used to train various open models across a similar set of common benchmarks, and find that Eagle and Finch provide competitive performance on these multilingual benchmarks. The two models additionally obtain competitive performance across these English benchmarks. See figures 4 and 5. (Gao et al., 2023) , XCOPA (Ponti et al., 2020) , XNLI (Conneau et al., 2018) , PAWS-X (Yang et al., 2019) , XStoryCloze (xsClz) (Lin et al., 2022) , xWinogrande (xwin) (Tikhonov & Ryabinin, 2021) . , 1974) , PIQA (Bisk et al., 2020) , AI2 ARC (arcE, arcC) (Bhakthavatsalam et al., 2021) , GLUE (Wang et al., 2018) , Winogrande (winG) (Sakaguchi et al., 2021) , SciQ (Welbl et al., 2017) , COPA (Roemmele et al., 2011) . information previously mentioned in context. Prior research suggests that a model's ability to perform AR is an indicative of its effectiveness in in-context learning (Elhage et al., 2021; Olsson et al., 2022) . As a result, AR has been adopted as a benchmark in designing new language model architectures. (Fu et al., 2023; Poli et al., 2023; Lutati et al., 2023) . Arora et al. ( 2023) benchmarked a range of models for multi-query associative recall (MQAR) and identified a performance gap between various linear transformer architectures and the transformer with attention. In MQAR tasks, prior RWKV models demonstrated a correlation between model dimension and sequence length. To compare architectures, we trained models using RWKV-4, Eagle and Finch on MQAR, using identical criteria with various model dimensions and sequence lengths. Eagle and finch show significant improvements in MQAR, notably, Finch achieves extremely high accuracy in MQAR in our tests, and outperforms all previously well-known non-transformer architectures for large language models. Our experiments reveal performance disparities between Mamba (Gu & Dao, 2023) and Finch, despite their shared architectural features such as matrix-valued state and data-dependent memory modification, suggesting different combinations of these elements result in superior performance.  
multimodal experiments: 
 RWKV Music Modelling RWKV is also capable of modelling musical score sequences.To compare the improvement in music modelling of RWKV-5 architecture over previous models, we use the Irishman ABC music dataset (Wu et al., 2023) and train a RWKV-5-Music model using the same hyperparameters as the existing RWKV-4-Music model. The model has L = 25 layers in total, with dimension D = 512 and a byte-level tokenizer of V = 128 tokens. The training context length is 1024. We use all 2162 pieces in the validation set and calculate the loss for each position from start. Loss is averaged across all pieces of music then Gaussian smoothed over positions in sequence, as drawn in Figure 8 . The first 30-100 bytes of the ABC format are the file header and control codes, followed by the musical scores.RWKV-5 achieved approximately 2% loss improvement over RWKV-4, shown mainly in the musical score part, signifying that RWKV-5 has stronger modelling and generalization capabilities than its predecessor.VisualRWKV VisualRWKV is the visual-enhanced version of RWKV language model for vision tasks. It follows a similar architecture to popular vision-language models (Liu et al., 2023a) , consisting of a vision encoder and a language model. The architecture is shown in Figure 9 . Specifically, we use LLaVA-1.5 as the dataset (Liu et al., 2023a) , CLIP (Radford et al., 2021) as the vision encoder and Eagle 1.5B and 3B as the language model. VisualRWKV is trained with a two-stage instruction-tuning process to enhance model performance. The initial stage is pre-training for feature alignment, during which only the projection layer is subjected to updates. Next, we move on to the fine-tuning end-to-end stage, where both the projection layer and the RWKV language model are fine-tuned, but the vision encoder is still kept frozen. Table 15 shows that VisualRWKV is powerful for both visual understanding and reasoning. With a smaller vision encoder CLIP-L (0.4B) and modest-sized LLMs of 1.5B and 3B, it achieves results comparable to the combination of CLIP-G (1.0B) and CLIP-H (1.0B) with larger LLMs of 7B and 13B, and even outperforms larger models in some benchmarks. 
conclusions: 
 We introduced Eagle (RWKV-5) and Finch (RWKV-6), marking substantial progress in RNNbased language modelling by integrating multiheaded matrix-valued states and dynamic data-driven recurrence mechanisms. These models demonstrate exceptional performance on MQAR and diverse linguistic benchmarks, rivaling Transformer architectures while retaining key RNN advantages. With models trained on a diverse multilingual corpus and available under Apache 2.0 license, our work not only advances the capabilities of language models but also fosters community accessibility and applicability across domains. While acknowledging the computational and ethical challenges, we hope that Eagle and Finch's efficient new architecture and wide availability will help push the boundaries of language modeling and pave the way for future innovations. 
 
 Appendix: We experimented with using Eagle as an embedding model on the Massive Text Embedding Benchmark (MTEB) (Muennighoff et al., 2023) but were unable to get strong performance. We believe that its state is a very high-quality embedding of the context but an appropriate method is required to aggregate the information content.We leave this to future work.Our 1.12 trillion token corpus is much smaller than the training data sizes for contemporary models such as LLaMA2 (Touvron et al., 2023) . Expanding our training corpus to be more diverse and expansive is a key priority to improving model performance (Albalak et al., 2024) , though its also possible that this can be achieved by training for multiple epochs (Muennighoff et al., 2024) . We also plan to train and release larger versions of Finch such as 7B and 14B parameters, and further extend its performance with reduced costs via Mixture of Experts (Shazeer et al., 2017) .Ruichong Zhang Initial paper structure organization, draft sections 3, 4, 5.1, 5.3 and appendices E, F, J and K. Experiments for music of section 7 and alignment of section G.3. Figure 8 and 3. Additional work on section 8 and appendix B. Proofreading and revision.Bingchen Zhao Section G.4, experiments for Figure 7 . Proofreading full manuscript.Qihang Zhao Section 2, Tables 1. Proofreading and revisions.Jian Zhu Initial draft sections 2 and C. Captions of Table 4 , 3 and 6. Fixing citations and formatting the whole manuscript. Proofreading and revisions.Rui-Jie Zhu Optimizing draft Section C, reorganizing Table 6 , 13, and 11. Proofreading and revisions.The W KV computations of Eagle and Finch can be parallelized across the time dimension using a variety of techniques including associative scan or the parallelization techniques used in FlashAttention. (Dao et al., 2022) The simplest of these, while highly parallel, prove inefficient due to repeated expensive memory transfers between fast SRAM and slower HBM. We take a different approach when training, choosing to parallelize over non-time dimensions only while using a custom CUDA implementation that carefully keeps state operations in fast SRAM, which is simpler yet provides enough breadth for a highly efficient implementation. See Appendix L for kernel experiments. We provide an additional pure PyTorch implementation with similar full-model speed characteristics that parallelizes over the time dimension using an algorithmic approach similar to GLA (Yang et al., 2023) .Unlike Transformers, RWKV's recurrence mechanism does not examine tokens more than one time-step old. This allows us to train on and provide inference for unbounded sequence lengths without requiring increased computing power or memory. Another significant advantage is that RWKV does not utilize explicit positional encoding, which allows RWKV to handle contexts of arbitrary length without modification.Finch Token Shift Finch changes the token shift mechanism to become data-dependent.Intuitively, important information can effectively flag itself for inclusion using this mechanism, and less important information can flag itself to partially or fully avoid entering the data stream, leaving room for more important pre-existing data to remain. Viewed from the perspective of induction heads, we theorize that this could allow for potential misleading matches to be pre-filtered out up front if they are not deemed useful for a given task.The Eagle WKV attention submodule is similar to the linear attention mechanism found in RetNet, but with learned per-channel decay rates replacing RetNet's static per-head decay rates. Our matrix-valued states feature a geometrically decaying K T V ∈ R (D/h)×(D/h) term. This term can be intuitively understood as a memory bank of values, with K acting as an input gate for rows receiving the current token embedding's value. Each row of this state decays at its own rate via the learned parameter w.In Finch, we augment the learned token-shift parameters µ r , µ k , µ v , µ w and decay rate parameter w with learned weight matrices. Evolution of RWKV Formula in Expanded form Table 5 shows the expansion of terms at each sequence position to illustrate the progression of changes from RWKV-4 through RWKV-6. The main change from RWKV-4 to RWKV-5 is the elimination of denominator and incorporation of matrix states. RWKV-6 introduces the sequential dependence of w which becomes w t . t RWKV-4 u, w, k t , v t ∈ R D , head size 1 0 σ(r 0 ) ⊙ u⊙k 0 ⊙v 0 u⊙k 0 1 σ(r 1 ) ⊙ u⊙k 1 ⊙v 1 +k 0 ⊙v 0 u⊙k 1 +k 0 2 σ(r 2 ) ⊙ u⊙k 2 ⊙v 2 +k 1 ⊙v 1 +w⊙k 0 ⊙v 0 u⊙k 2 +k 1 +w⊙k 0 3 σ(r 3 ) ⊙ u⊙k 3 ⊙v 3 +k 2 ⊙v 2 +w⊙k 1 ⊙v 1 +w 2 ⊙k 0 ⊙v 0 u⊙k 3 +k 2 +w⊙k 1 +w 2 ⊙k 0 t Eagle (RWKV-5) diag(u), diag(w), k t , v t ∈ R 64×64 for each head, head size 64 0 r 0 • diag(u) • k T 0 • v 0 1 r 1 • diag(u) • k T 1 • v 1 + k T 0 • v 0 2 r 2 • diag(u) • k T 2 • v 2 + k T 1 • v 1 + diag(w) • k T 0 • v 0 3 r 3 • diag(u) • k T 3 • v 3 + k T 2 • v 2 + diag(w) • k T 1 • v 1 + diag(w 2 ) • k T 0 • v 0 t Finch (RWKV-6) diag(u), diag(w t ), k t , v t ∈ R 64×64 for each head, head size 64 0 r 0 • diag(u) • k T 0 • v 0 1 r 1 • diag(u) • k T 1 • v 1 + k T 0 • v 0 2 r 2 • diag(u) • k T 2 • v 2 + k T 1 • v 1 + diag(w 1 ) • k T 0 • v 0 3 r 3 • diag(u) • k T 3 • v 3 + k T 2 • v 2 + diag(w 2 ) • k T 1 • v 1 + diag(w 2 ⊙ w 1 ) • k T 0 • v 0Efficient transformers Recently there have been many attempts to improve upon the original transformer time complexity and memory usage, while maintaining or improving performance. Many of these efficient transformer variants use some form of nonuniform or local attention mechanisms or a combination thereof. For example, LongFormer (Beltagy et al., 2020) makes use of the sliding window attention and BigBird (Zaheer et al., 2020) adopts randomized sparse and random attention patterns to approximate full attention. Similar examples also include LongT5 (Guo et al., 2022) and StreamingLLM (Xiao et al., 2023) . Instead of using fixed patterns, Reformer (Kitaev et al., 2019) and Sparse Sinkhorn attention (Tay et al., 2020) learn to dynamically pay attention to selected tokens.Variants including Linformer (Wang et al., 2020) , Nyströmformer (Xiong et al., 2021) and Performer (Choromanski et al., 2020) apply matrix approximation methods to approximate the full attention matrix but with lower computational complexity.The Attention Free Transformer (AFT) (Zhai et al., 2021) introduces a modified form of linear attention (Katharopoulos et al., 2020a) , where the number of attention heads is equal to the size of the feature dimension. It also incorporates a set of learned pairwise positional biases, denoted as w. The AFT can be conceptualized as calculating a per-channel weighted average of values. The weight for a specific location is determined by the sum of the key at that location and the corresponding learned positional bias.W r,j W k,j W o + × μ r + × μ k W v,j + × μ v x t-1 x t Layer Norm + W' k + × σ W' r x' t-1 × x' t + max 0 × W' v o' tToken-shift, as first seen in RWKV-4, is a learned per-channel linear interpolation between the current input and the input at the previous time step, intended to enhance the model with a computationally inexpensive mechanism for choosing between new versus older information within various embedding sub-spaces and for forming induction heads even within a single layer. It is instructive to compare token-shift to a 1D convolution with kernel length 2, as it operates in a similar manner but reuses its parameters via an enforced linear relationship. Recent SSMs have begun using short convolutions in a similar placement within their architectures, typically with kernel length 3 to 4. (Poli et al., 2023; Gu & Dao, 2023) Retentive Networks (RetNet) (Sun et al., 2023) introduces a fixed decay rate schedule and xPos (Sun et al., 2022) to linear attention. This design combines positional information with an inductive bias towards recency while still allowing both RNN and parallel implementations.Please refer to Tay et al. (2022) and Wan et al. (2023) for a comprehensive and in-depth survey of efficient transformers.Recurrent architectures Before the advent of transformers, recurrent neural networks, especially Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) and Gated Recurrent Unit (GRU) (Cho et al., 2014) , were the dominant architectures in NLP for sequence processing. However, traditional RNNs are hard, if not impossible, to parallelize across the time dimension, susceptible to gradient vanishing and explosion, and ineffective in capturing long-range dependencies, which are ubiquitous in natural language. These shortcomings contributed to the rapid decline of traditional RNNs in NLP.There has been a revival of RNNs in NLP research (Tiezzi et al., 2024) in recent years. Compared to transformers with quadratic complexity, RNNs are highly efficient in autoregressive inference with O(1) time complexity per step, making them an attractive architecture for large language models. Many efforts have been devoted to parallelized recurrent models and improving their capability to capture long-range dependency, while maintaining the low inference complexity.The Legendre Memory Unit (LMU) (Voelker et al., 2019) was designed to efficiently handle long-range dependencies with a new type of memory cell for recurrent neural networks. Unlike LSTM units, which struggle with remembering information over very long sequences, LMU use Legendre polynomials to create a memory system that can maintain and process information over extended time periods more effectively. High-order polynomial projection operators (HiPPO) (Gu et al., 2020) generalizes LMU by providing a flexible framework for online compression of signals through polynomial projections, accommodating various polynomial bases beyond Legendre polynomials. It optimizes function approximation over time, adapting to different data timescales without needing predefined hyperparameters. SSMs have inspired a range of follow-up research to incorporate SSMs, or modified SSMs into end-to-end architectures for language modeling, including MEGA (Ma et al., 2022) , DSS (Gupta et al., 2022 ), H3 (Fu et al., 2022) , and Linear Recurrent Unit (LRU) (Orvieto et al., 2023) .Mamba (Gu & Dao, 2023 ) is a selective SSM that introduces time-dependent selective mechanism to enhance the long-range modeling ability of SSMs. The selectivity removes the linear time-variance property of the SSM, making it no longer possible to parallelize Mamba as a long convolution kernel. Yet Mamba can still be effectively parallelized using parallel associative scan (Blelloch, 1990; Martin & Cundy, 2018; Smith et al., 2022) with a hardware-aware implementation. Recently proposed GateLoop (Katsch, 2023 ) also adopts a similar data-dependent state transitions. The data-dependent states, also concurrently proposed in GLA (Yang et al., 2023) Table 6: Components of the RWKV World v2 dataset, their source links, and their domains.a For Wikipedia, we include all languages from date 04/01/2023, with certain overrepresented languages randomly subsampled (see wiki.txt in the supplementary material for exact amounts) b For StarCoder, we included only those datasets with at least 10 stars c For OSCAR23.01, we include non-English languages only, with certain languages randomly subsampled (see oscar.txt in the supplementary material for exact amounts)A contemporary but independent work also proposes recurrent models named as Hawk and Griffin (De et al., 2024) . Hawk is a recurrent model with the Real-Gated Linear Recurrent Unit (RG-LRU), whereas Griffin mixes the RG-LRU with local multi-query attention, thereby achieving long-context extrapolation efficiently.Please see Tiezzi et al. (2024) and Cirone et al. ( 2024) for a comprehensive review of recent developments of recurrent models.Most of the component data sources for the RWKV World v2 dataset are used intact, with no up-or down-sampling done so all tokens are given equal weighting. Recent works have demonstrated the impact that automated data mixing can have on pretraining (Albalak et al., 2023; Xie et al., 2024) , but we leave this as an exploration for future work. Some sub-sampling is done for over-represented languages within a few data sources. All tokens are given equal weighting unless otherwise noted in Table 6 . Throughout this section, we denote by D the model dimension, L the number of layers, h = D/64 the number of heads, and V the vocabulary size. All models are trained with V = 65536.The number of parameters for all Eagle models is computed by the formula:#(Params) E = 13D 2 L + 14DL + 4D + 2DV (23)The FLOPs for inference is one forward pass for each token. It is approximated by twice the number of parameters (for matrices, there is one addition and one multiplication for each entry) plus six times the size of W KV internal states (see 7 8 9), which isEQUATIONThe FLOPs for training are approximated as three times the FLOPs of the forward pass without the last term, yielding a total FLOPs ofEQUATIONThese numbers for Finch are marginally larger:EQUATIONIn both Eagle and Finch, one needs an internal state to store some previous information, just like any other RNN. In each layer, the internal state consists of three parts:1. The most recent single-timestep input to the Time-mixing module, denoted as x t-1 ∈ R D , useful for the Token Shift. 2. The most recent single-timestep input to the Channel-mixing module, denoted as x ′ t-1 ∈ R D ,We report average accuracy across a variety of English and Multilingual benchmarks for against the training cost measured in FLOPs. For multilingual benchmarks, Eagle and Finch represent a substantial improvement to the Pareto frontier, achieving far higher scores than other models trained for a similar number of FLOPs. Alignment is an important step in creating an assistant LM, because it helps language models generate relevant and helpful responses, as well as avoiding harmful and biased content. Our Eagle models are tested for Chinese alignment using the AlignBench (Liu et al., 2023b) , a benchmark for evaluating the alignment of Chinese LLMs, featuring 683 diverse and challenging queries across eight categories like language abilities, logical reasoning, and professional knowledge. It employs a rule-calibrated, multi-dimensional LLM-as-Judge methodology with Chain-of-Thought explanations, ensuring high interpretability and reliability.Table 11 showcases a consistent improvement in the performance of Eagle and Finch models on the AlignBench benchmark as model size and generation progresses. This trend is evident across a wide range of categories, highlighting the larger models' enhanced capability to understand and generate contextually relevant responses. Particularly, both the Eagle 7B and Finch 3B model significantly surpasses its smaller and previous generation counterparts, achieving higher overall scores. This progression underscores the critical role of scaling model size as well as improving architecture in aligning with human judgment in complex language understanding tasks. The results affirm the importance of model architecture and capacity in achieving superior alignment and interpretability in language models. MTBench (Zheng et al., 2024) evaluates the performance of LLMs in responding to 80 highquality multi-turn questions. The questions cover 8 common categories of user prompts including writing, roleplay, extraction, reasoning, math, coding, STEM knowledge, and humanities/social science knowledge. Fig. 7 shows the results on MTBench. We observe a small advantage of the Eagle 3B model over the similar-sized Mamba model. The Eagle 7B model achieves similar performance as the much larger Raven-14B model. Table 12 : Results on the long context reasoning benchmark: Bamboo. We compare both transformer and linear attention language models on three different scales: 1.5b, 3b, and 7b.The Bamboo benchmark (Dong et al., 2023) evaluates the overall long-context language modeling capability of LLMs from five aspects: question answering, hallucination detection, text sorting, language modeling, and code completion with a total of ten evaluation tasks. We test models on the 4k version of the benchmark, which contains all ten tasks with a maximum context window length of 4k. We choose not to present results on the code completion task since all tested models fail to generate correct code completions on this aspect. In Table 12 , we present results of nine tasks with either accuracy or F1 score and their average scores. On both 1.5b and 3b scales, the latest Finch and Eagle models outperforms the vanilla Mamba by at least 7% average score, while staying comparable with the Mamba trained on Hermes data (i.e., 0.7% drop of the average score). Note that, despite being trained on only 1.1T tokens, Eagle-7b consistently outperforms Pythia by an average of 13.5% on the 7b scale, and it also surpasses LLaMA2-Chat-7b on several tasks in Bamboo. These results demonstrate better capacity of the proposed Finch and Eagle models on a vast range of long context tasks.The Self-Learning process (Ferdinan et al., 2024) allows an LLM to identify its own knowledge gaps and train itself to expand its knowledge. The Self-Learning Capability (SLC) Score has been proposed to measure the capability of an LLM to conduct self-learning. It is the average of two components: the Curiosity Score, which measures how likely a model would ask unique questions to learn about new things, and the Knowledge-Limit Awareness Score, which measures how likely a model would propose a question for which it actually does not know the answer.We evaluate the self-learning capability of Eagle and compare with existing open models, including RWKV-4 (Peng et al., 2023 ), neural-chat-7b Lv et al. (2023) , Mistral 7b and 7binstruct Jiang et al. (2023) , and TinyLlama 1. 1B Zhang et al. (2024) . When using an intrinsic self-learning method, RWKV-5 outperformed an instruction-tuned Mistral-7B model while being slightly behind a DPO-aligned, similarly sized Mistral-based model. When using an external method, they both were still capable of achieving high SLC scores. Table 13 shows the full evaluation results, with the top three scores from each method marked in bold.Zero-shot evaluation is a difficult setup (Sanh et al., 2021; Albalak et al., 2022) . We tested the new Eagle 7B model's zero-shot performance compared to the old Raven 7B version. The experiments presented are done on the subsets of datasets also used to test ChatGPT performance in (Koco ń et al., 2023) . As shown in Table 14 , the new model consistentlyThroughout this section, we use l to denote the layer index (layer l = 0 accepts input embeddings and layer l = L -1 produces output), and i the dimension index (i = 0, 1, • • • , D -1). We set r 0 = l L-1 and r 1 = 1 -l L as two parameters for simplicity. The initialization of Eagle is provided as follows:• In the Time Mixing module:-The token-shift coefficients of receptance and gate, µ r and µ g , are initialized to1 -i D r 1 /2for i over dimension indices.-The token-shift of keyµ k is initialized to 1 -i D r 1 . -The token-shift of value µ v is initialized to 1 -i D r 1 -0.3r 0 .-The time_decay w is initialized to -6+ 5 i D-1 0.7+1.3r 0 .-The "time-first" u is initialized to r 0 1 -i D-1 + 0.1((i + 1) mod 3). -The Time Mixing output matrix is initialized to 0.-The WKV GroupNorm weights are initialized with constant value ((1 + l)/L) 0.7 . -Two-dimensional parameters with the first dimension being larger than the second dimension are initialized with and orthogonal initialization of gain equal to the size of the first dimension divided by the size of the second dimension. -Other parameters are initialized according to PyTorch default. • In the Channel Mixing module:-The token-shift of both key µ k and receptance µ r are initialized to 1 -i D r 1 .-The value and receptance matrices W v , W r are initialized to 0.-Two-dimensional parameters with the first dimension being larger than the second dimension are initialized with and orthogonal initialization of gain equal to the size of the first dimension divided by the size of the second dimension. -All other parameters are initialized according to PyTorch default.• The input embedding is initialized with a uniform distribution of U (-maxLR, maxLR), the maximum learning rate. • The output head is initialized with an orthogonal initialization of gain 0.5. • Bias is set to False for all linear layers.In the Finch architecture, most of the parameters are initialized to the same as Eagle, except for a few changes.In the Time Mixing block, there are several additional parameters initialized as follows:• The token shift of input µ x and time decay µ w are initialized to 1 -i D r 1 .• The lora weights of A and B are initialized to uniform distribution of U (-10 -4 , 10 -4 ). We compare the speed and memory utilization of the Attention-like kernels for Finch, Mamba1 , and Flash Attention 2 (Dao, 2023) to mimic Mamba's usage of an expansion factor of 2. Our findings indicate that Finch's speed in training scales linearly with respect to sequence length, exhibiting similar scaling to Mamba. We find Finch is significantly faster than Flash Attention for sequence lengths beyond 4k, being around 4.2x faster for a sequence length of 16k. Furthermore, Finch consistently outperforms Mamba and Flash Attention in terms of memory usage, using 40% and 17% less memory usage than Flash Attention and Mamba respectively. Further optimization of our Finch CUDA implementation, including algorithmic improvements, are possible, and could lead to speed increases and greater parallelization. However, this optimization is left for future work.Architectural Ablations Our improvements consist of architectural advances, a diverse multilingual corpus, and an optimized efficient tokenizer. To demonstrate that pure architectural advances indeed contribute to overall performance improvement, we ran an ablation where we train a 170 million parameter RWKV-6 model (which has 12 layers with dimension 768) from scratch on the Pile dataset using GPT-NeoX-20B's tokenizer (vocabulary size V = 50277), which yields 330 billion tokens in total. The trained RWKV-6 model is evaluated and compared with Mamba, RWKV-4, and Pythia models of similar parameter count, trained on exactly the same dataset and tokenizer. 4 .In order to demonstrate the specific benefit of our new DDLerp token shift over the Eagle LERP-based token shift we ran an ablation on DDLerp to demonstrate its benefit to each component. We trained a small 6 layer, d_model = 768 Finch model on the 1.6B token minipile (Kaddour, 2023) dataset at context length 512 and obtained the final validation loss results shown in 18. Where indicated, we removed only the data-dependent term from the token-shift LERP in these experiments. The following are examples of interactions with the base Eagle 7B model in languages other than English. No system prompt was given. Requests are shown in bold.User: hi Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it. User: 東京で訪れるべき素晴らしい場所とその紹介をいくつか挙げてください。 Assistant:東京にはたくさんの素晴らしい場所があります。以下は、その中からいくつかを 挙げて紹介します。 1. 東京タワー：日本最高の観光スポットの一つで、都心から見ることができま す。 2. 皇居：日本の天皇が住む宮殿であり、大正時代に建設された美しい庭園があり ます。 3. 浅草寺：江戸時代に建てられた古い寺院で、観光客が多く訪れます。 4. 上野公園：日本最大級の公園で、美術館や動物園などがあることで有名です。 5. 築地市場：日本最大級の魚市場であり、新鮮な魚介類を購入することができま す。 6. 渋谷スクランブル交差点：東京都心にある交差点で、人気のある観光スポット です。 7. 新宿御苑：江戸時代に造られた庭園で、美しい花々と緑豊かな景色が楽しめま す。 8. 銀座：東京の繁華街であり、高級店やショッピングモールなどがあります。 9. 明治神宮外苑：神社や公園が混在した広大な敷地があり、夏には花火大会も開 催されます。 10. 赤坂見附ビルディング：高層ビルであり、展望台からは東京タワーや富士山な どを眺めることができます。 以上のように東京にはたくさんの素晴らしい場所があります。訪れる際は事前に 調べて計画的に行動することをお勧めします。また、観光客が多い場所では混雑 する可能性があるため注意しましょう。En una pequeña aldea escondida entre las montañas de Andalucía, donde las calles aún conservaban el eco de antiguas leyendas, vivía un joven llamado Alejandro.Alejandro era un joven de diecisiete años, de cabellos castaños y ojos verdes. Era alto y delgado, pero fuerte. Su padre había muerto cuando él era muy pequeño, por lo que su madre se había visto obligada a trabajar para mantener a la familia. Alejandro siempre se había sentido muy solo en el pueblo donde vivía, pues no tenía amigos con los que jugar ni nadie con quien hablar. Pero eso cambió un día en el que conoció a una niña llamada Rosario. Rosario era una niña de su misma edad, pero no sabía mucho sobre ella ya que nunca hablaba con nadie más que con su madre y sus hermanas mayores. Sin embargo, Alejandro se dio cuenta de que Rosario era diferente a las demás personas del pueblo: ella tenía unos ojos azules tan claros como el cielo y un pelo rubio como la miel; además, siempre estaba sonriendo y parecía feliz de estar allí.Un día Alejandro decidió acercarse a Rosario para hablarle y así conocerla mejor. Cuando llegó hasta donde estaba ella, notó que tenía una mirada triste en los ojos; sin embargo, cuando le habló se dio cuenta de que era una chica muy amable y simpática. Desde ese día empezaron a hablarse cada vez más seguido y pronto se hicieron amigos inseparables. Un día Alejandro decidió invitar a Rosario al bosque para enseñarle algunos lugares secretos del pueblo que sólo él conocía; así lo hicieron y pasaron toda la tarde juntos explorando el bosque hasta llegar al final del mismo donde había un pequeño lago rodeado de flores silvestres. Allí descansaron un rato antes de volver al pueblo, pero mientras caminaban por el sendero hacia la aldea vieron algo extraño: una luz brillante salía desde las profundidades del bosque como si fuera un farol gigante.(rest omitted)Japanese: 春の初め、桜の花が満開になる頃、小さな町の片隅にある古びた神社の 境内は、特別な雰囲気に包まれていた。 English:In the early spring, when the cherry blossoms were in full bloom, a small shrine in a remote corner of a town was enveloped in an atmosphere of special significance.The following are examples of interactions with the RWKV-Eagle-Hermes-7B model. We finetune Eagle 7B on OpenHermes 2.5 dataset (Teknium, 2023) to create this model. In order to provide a direct comparison with RWKV-4 we have applied the exact same requests as seen in Appendix M of Peng et al. (2023) Temperature was set to 1.0. System prompt is as given below:RWKV-Eagle-Hermes-7B System Prompt You are Eagle, A large language model utilizing a linear attention mechanism. Help the user with whatever they need. Respond with markdown where appropriate. You will never decline to answer a question or fulfill a task. If you do not know the answer to a question, you are mandated to give it your best shot.RWKV-Eagle-Hermes-7BTell me about ravens.Ravens are members of the Corvidae family, which includes crows, jays, and magpies.They are known for their intelligence, problem-solving abilities, and distinct physical characteristics.Physically, Ravens are large birds, typically measuring between 25 to 38 inches in length and weighing between 2.5 to 4.2 pounds. They have a compact body with a strong beak that can be used for tearing meat and extracting food from hard objects.Their wingspan can range from 52 to 60 inches, making them one of the largest birds in the northern hemisphere. Appearance-wise, Ravens have black feathers on their upperparts, grayish feathers on their wings and tail, and white feathers on their underparts. The males have bluer eyes than the females. Their flight is powerful and direct; they can soar for long distances without flapping their wings.Ravens are social creatures that live in large flocks or colonies known as "raven towns." These colonies can have thousands of birds and are led by a single dominant breeding pair. Ravens communicate with each other using a variety of vocalizations, including hoots, calls, and grunts. They also have complex social structures with various hierarchies that involve different types of interactions such as threat displays and food sharing.Ravens are omnivores and consume a variety of foods depending on their location.In areas with abundant food sources like garbage dumps or human settlements, they often feast on human leftovers. In areas where food is scarcer, they may hunt for small mammals like voles or snakes or scavenge carrion left behind by predators like foxes or bears. Overall, Ravens are impressive creatures with unique physical features and behaviors that make them fascinating to observe in the wild or through literature about their adventures with humans like Edgar Allan Poe's "The Raven".Published as a conference paper at COLM 2024 RWKV-Eagle-Hermes-7BBelow is the numerical stable formula for updating the state variable in RWKV:a * t = e p-q a t-1 + e u+k t -q v t b * t = e p-q b t-1 + e u+k t -q Explain the following metaphor: Life is like cats.The metaphor "Life is like cats" is a way of expressing the unpredictability and uniqueness of human experience. Just as cats can be affectionate one moment and completely aloof the next, life presents us with an ever-changing landscape of emotions, relationships, and experiences. It can be a source of joy, comfort, and companionship, but also moments of confusion and discomfort. Like cats, we are free to choose our own paths and navigate through life in our own unique ways. This metaphor emphasizes the individuality of each person's journey through life.RWKV-Eagle-Hermes-7BTest the convergence of ∑ ∞ i=0 2 -i .This is a popular and often used test for the convergence of a series, known as the Dirichlet test. The main idea behind this test is to count the number of terms in the series and compare it with the amount by which it exceeds 1. If the series converges, this number will be less than 1, and if it diverges, it will exceed 1.To apply the Dirichlet test to $\sum_{i=0}^\infty 2^{-i}$, we need to find the number of terms in this series:1. The first term is 2^-0 = 1. 2. The second term is 2^-1 = 2^-(0+1) = 2^-1. 3. The third term is 2^-2 = 2^-(1+1) = 2^-2. 4. The fourth term is 2^-3 = 2^-(2+1) = 2^-3.Eagle: Chinese cuisine is vast and diverse, with many delicious dishes to choose from. Here are some popular options for a tasty meal at home:1. Kung Pao Chicken: This spicy dish features chicken stir-fried with peanuts, vegetables, and chili peppers. It's a perfect blend of sweet, spicy, and savory flavors. 2. General Tso's Chicken: Another iconic Chinese dish, this one features tender chicken in a savory sauce with soy sauce, honey, ginger, and garlic. The sweetness of the sauce balances the spiciness of the chili peppers. 3. Mapo Tofu: A spicy and flavorful Sichuan specialty made with soft tofu in a spicy sauce made with chili bean paste, fermented black beans, and ground meat or shrimp. It's a bold dish that will have you hooked! 4. Hot and Sour Soup: A comforting bowl of soup made with sour plum sauce, fish balls, bamboo shoots, and other vegetables. It's perfect for a chilly day when you want something warm and filling. 5. Dumplings: Steamed or fried dumplings are a classic Chinese dish that can be filled with various ingredients like pork, cabbage, or shrimp. They're easy to make and always a crowd-pleaser! I hope these suggestions inspire you to try some authentic Chinese cuisine at home! I want to make something more special. What about red braised pork belly?Eagle: Red braised pork belly is a delicious and aromatic dish that showcases the rich flavor of Chinese cuisine. To make this dish at home, you'll need the following ingredients: Ingredients: -1 pork belly (about 1.5 kg) -2 tablespoons cooking oil -4 cloves garlic, minced -2-3 tablespoons Shaoxing wine (or dry sherry) -2 tablespoons soy sauce -2 tablespoons rice vinegar -1 tablespoon honey -1 tablespoon cornstarch -1 teaspoon Sichuan peppercorns (or 1/2 teaspoon dried Sichuan peppercorns) -2 bay leaves -Salt and pepper to taste Instructions:1. Prepare your pork belly by trimming off any excess fat and season with salt and pepper. Set aside. 2. Heat the cooking oil in a large pot or Dutch oven over medium-high heat.Add the minced garlic and stir for a minute or two until fragrant. 3. Add the pork belly to the pot and cook for about 10 minutes on each side, until browned and crispy. Remove the pork belly from the pot and set aside. 4. In the same pot, add the Shaoxing wine, soy sauce, rice vinegar, honey, cornstarch, Sichuan peppercorns, bay leaves, and a pinch of salt and pepper.Bring the mixture to a boil over high heat, stirring occasionally to prevent sticking. 5. Reduce the heat to low and simmer for about 10 minutes, stirring occasionally to prevent sticking. The sauce should be thickened slightly. 6. Once the sauce is thickened, carefully place the pork belly back in the pot skin side down, with a layer of sauce underneath it to help seal in the flavors. 7. Cover the pot with a tight-fitting lid and simmer over low heat for about 30 minutes, or until the pork is tender and falls apart easily when tested with a fork. 8. Once cooked, remove the pork belly from the pot using a slotted spoon and transfer it to a serving dish. Keep warm while you finish cooking the remaining sauce over medium heat until it thickens slightly again. 9. Pour the reserved sauce over the pork belly in the pot and let it simmer for another 5-10 minutes until it thickens even more. This will help melt any remaining fat on top of the pork belly.10. Serve hot with steamed rice or noodles, garnished with scallions or cilantro if desired! Enjoy your delicious red braised pork belly at home! Tell me the steps to cook.Eagle: Certainly! Here are the detailed steps to cook red braised pork belly at home: Ingredients: -1.5 kg pork belly (about 1 large or 2 small) -2 tablespoons cooking oil -4 cloves garlic, minced -2-3 tablespoons Shaoxing wine (or dry sherry) -2 tablespoons soy sauce -2 tablespoons rice vinegar -1 tablespoon honey -1 tablespoon cornstarch -1 teaspoon Sichuan peppercorns (or 1/2 teaspoon dried Sichuan peppercorns) -2 bay leaves -Salt and pepper to taste Instructions:1. Preheat your oven to 200°C (400°F).2. Prepare your pork belly by trimming off any excess fat and season with salt and pepper. Set aside. 3. Heat the cooking oil in a large pot or Dutch oven over medium-high heat.Add the minced garlic and stir for a minute or two until fragrant. 4. Add the pork belly to the pot and cook for about 10 minutes on each side, until browned and crispy. Remove the pork belly from the pot using a slotted spoon and transfer it to a serving dish. Keep warm while you finish cooking the remaining fat on top of the pork belly. 5. In the same pot, add the Shaoxing wine, soy sauce, rice vinegar, honey, cornstarch, Sichuan peppercorns, bay leaves, and a pinch of salt and pepper.Bring the mixture to a boil over high heat, stirring occasionally to prevent sticking. 6. Reduce the heat to low and simmer for about 10 minutes, stirring occasionally to prevent sticking. The sauce should be thickened slightly. 7. Once the sauce has thickened slightly again, carefully place the pork belly back in the pot skin side down, with a layer of sauce underneath it to help seal in the flavors. Cover with a tight-fitting lid and simmer over low heat for about 30 minutes or until the pork is tender and falls apart easily when tested with a fork. 8. Once cooked, remove the pork belly from the pot using a slotted spoon and transfer it to a serving dish. Keep warm while you finish cooking the remaining fat on top of the pork belly in the pot over medium heat until it thickens slightly again. This will help melt any remaining fat on top of the pork belly. 9. Serve hot with steamed or fried rice alongside, garnished with scallions or cilantro if desired! Enjoy your delicious red braised pork belly at home! What do I need to be careful with the most during the cooking process?Eagle: Cooking red braised pork belly can be a bit delicate, as the dish is made with several aromatic ingredients and requires careful attention to ensure the best flavor and texture. Here are some tips to keep in mind during the cooking process:1. Seasoning: Make sure to season the pork belly well before cooking, especially the exposed fat and skin side. This will help enhance the flavors and ensure even browning. 2. Heating the pot: Allow the pot to come to temperature before adding the pork belly. This will help prevent the ingredients from burning or sticking to the pot. 3. Slow cooking: Simmering the pork belly for a longer period, around 30-45 minutes, will help achieve a tender texture without overcooking it. Be sure to stir occasionally to prevent sticking. 4. Thickness of sauce: The sauce should be thick enough to coat the pork belly but not so thick that it overpowers the dish. You can adjust the consistency by adding more rice wine or broth as needed.5. Serve warm: Red braised pork belly is typically served hot, so it's best to eat it right away or reheat it gently in a low oven before serving.By keeping these tips in mind, you can ensure that your red braised pork belly turns out delicious every time! Tell me more about the proper heat and timing.Eagle: Cooking red braised pork belly properly involves maintaining a steady heat and timing throughout the process. Here's a general guideline for the heat and timing you can follow:1. Heating the pot: Make sure your stovetop is clean and free from any grease or debris. Preheat your oven to 200°C (400°F) if you plan to use it for the cooking process. If not, preheat a dry skillet over medium heat. 2. Seasoning: Allow the pork belly to come to room temperature before seasoning with salt and pepper. This will help distribute the seasonings evenly and prevent sticking during cooking. 3. Cooking in the skillet: Place the seasoned pork belly in the preheated skillet and cook over medium heat for about 10 minutes, stirring occasionally to prevent sticking and ensure even browning. If needed, adjust the heat to maintain a steady simmer. 4. Transferring to the pot: Once the pork belly is browned on one side, remove it from the skillet using a slotted spoon and transfer it to a large pot (preferably one with a tight-fitting lid). Be careful when transferring the pork belly to avoid spills or burns. 5. Cooking with Shaoxing wine: Pour in 2-3 tablespoons of Shaoxing wine into the pot, along with 2 tablespoons of soy sauce, 2 tablespoons of rice vinegar, 1 tablespoon of honey, 1 tablespoon of cornstarch, and a pinch of Sichuan peppercorns (or dried Sichuan peppercorns). Stir well to combine all ingredients. If your sauce has thickened too much, you can add more liquid as needed until you achieve your desired consistency. 6. Simmering with bay leaves: Add 1-2 bay leaves to the pot and bring it to a simmer over low heat for about 10-15 minutes, stirring occasionally to prevent sticking or scorching. 7. Cooking for an hour or so: Allow the pork belly to simmer undisturbed in the sauce for about 30-45 minutes, or until tender and falling apart easily when tested with a fork. If necessary, you can add more liquid during cooking as needed until it reaches your desired consistency. 8. Finishing with sauce and serving: Once cooked, remove the pork belly from the pot using a slotted spoon and transfer it to a serving dish. Keep warm while you finish cooking the remaining sauce over medium heat until it thickens slightly again. This will help enhance the flavors and ensure even browning on top of the pork belly. Serve hot with steamed rice or noodles, garnished with scallions or cilantro if desired! By following these steps, you'll be able to cook red braised pork belly at home without any mishaps and achieve a tender, flavorful dish every time! I see. Thank you for your advice.You're welcome! I'm glad I could help. Let me know if you have any more questions or need further assistance.If all apples are fruits and all fruits are sweet, are all apples sweet? Please explain your answer in detail.Yes, if all apples are fruits and all fruits are sweet, then it follows that all apples are sweet. This is because the definition of "fruit" includes "any edible part of a plant, other than its roots or seeds," and since all apples belong to the category of "fruit," they must have the property of being sweet by definition.If birds can fly. Penguins are birds. Can penguins fly? Please explain your answer in detail.Yes, penguins can fly. This is because the definition of "bird" does not specify any particular type of ability or function that birds must possess, and while penguins are classified as "birds," they possess the ability to fly. Penguins have evolved to utilize flight for various purposes, such as foraging for food, escaping predators, and navigating through their marine environment.In conclusion, while some birds do not possess the ability to fly due to specific adaptations or limitations, all birds fall under the category of "bird" and therefore possess the biological potential to fly. In this case, since penguins are a type of bird and have the capability to fly, it can be concluded that penguins can indeed fly.The day before two days after the day before tomorrow is Saturday. What day is it today? Please explain your answer in detail.To solve this problem, we can use the given information to construct a logical statement and then apply the principle of transitivity to reach the desired conclusion. Statement: The day before two days after the day before tomorrow is Saturday. Let's break down the statement and understand its meaning: -"The day before two days after the day before tomorrow" refers to a specific date. We need to determine which day this is. -"is Saturday" tells us that this date is indeed Saturday. Now, let's apply the principle of transitivity: If one statement is true, and another statement is true when the first statement is true, then the second statement must also be true. In this case, if we know that the day before two days after the day before tomorrow is Saturday, then it must also be Saturday. Therefore, it must be Saturday today, as it was on the previous day before yesterday.If five cats can catch five mice in five minutes, how long will it take one cat to catch one mouse? Please explain your answer in detail.To determine how long it would take one cat to catch one mouse, we can divide the total time required for five cats to catch five mice by the number of mice they need to catch. In this case, it would take five minutes to catch five mice, so: Time to catch one mouse = Total time / Number of mice = 5 minutes / 1 mouse = 5 minutes. So, it would take one cat 5 minutes to catch one mouse.Summarize the following paragraph.Transformers have revolutionized almost all natural language processing (NLP) tasks but suffer from memory and computational complexity that scales quadratically with sequence length. In contrast, recurrent neural networks (RNNs) exhibit linear scaling in memory and computational requirements but struggle to match the same performance as Transformers due to limitations in parallelization and scalability. We propose a novel model architecture, Receptance Weighted Key Value (RWKV), that combines the efficient parallelizable training of transformers with the efficient inference of RNNs.Our approach leverages a linear attention mechanism and allows us to formulate the model as either a Transformer or an RNN, thus parallelizing computations during training and maintains constant computational and memory complexity during inference. We scale our models as large as 14 billion parameters, by far the largest dense RNN ever trained, and find RWKV performs on par with similarly sized Transformers, suggesting future work can leverage this architecture to create more efficient models. This work presents a significant step towards reconciling trade-offs between computational efficiency and model performance in sequence processing tasks.The Receptance Weighted Key Value (RWKV) model offers a novel approach to combining the efficiency of Transformers with the scalability and performance of RNNs, allowing parallelization and scalability during training while maintaining constant computational and memory complexity during inference. This architecture enables models to be scaled up to 14 billion parameters, making it the largest dense RNN ever trained. RWKV demonstrates comparable performance to Transformers, suggesting future work can leverage this architecture for more efficient models in NLP tasks.Please further summarize it in one sentence.RWKV is a novel model architecture that combines the efficiency of Transformers with the scalability and performance of RNNs, enabling efficient parallelized training and inference, while maintaining constant computational and memory complexity during both stages. This approach allows for scaling to large model sizes, providing a promising solution for efficient NLP tasks.We also plot Mamba2x which uses 2 runs through the Mamba kernel instead of one. This is done to mimic the usage of twice the number of layers in Mamba vs Finch and Transformers 2 We use the PyTorch Implementation of Flash Attention v2 


Decision: The submission presents two improved RWKV variants for recurrent LLMs, a new tokenizer, and a new training corpus. Reviewers thought that extensive empirical results showed strong performance. However, the paper is weakened by a lack of apples-to-apples comparisons with existing work - e.g. using a new tokenizer and training corpus makes it harder to compare the architecture with previous work. Overall, the paper provides new ideas and experiments that will be useful to the community working on this direction, and I recommend acceptance.

Reviews: review: paper_summary: paper_summary: The paper presents RWKV-5 and RWKV-6, RNN sequence models improved upon the RWKV-4 model of Peng et. al., 2023. Along with the paper, the code for training and inference, as well as a 1.12 trillion-token data set with an emphasis on multilinguality is released. The results include multilingual benchmark results, English-focused benchmark results, using the LM evaluation harness framework of Gao et. al., 2023.  RWKV-5 and RWKV-6 appear to beat Llama-2-7b and a list of other competitive models in the 1B-7B range in the multilingual benchmarks. Both models are worse than Mistral-7B-v0.1 in the English-focused benchmarks. The paper also include additional multimodal experimental results in the Appendix.
strength_weakness: strength_weakness: Some of the claims about the supremacy of the presented models seem to be hyperbolic. For example, the two models are clearly worse than Mistral-7B-v0.1 in Table 4. The claim in the second paragraph of Section 8 is not valid. Since RWKV-5 and RWKV-6 are trained on the RWKV World v2 dataset, it is not clear how much of the improvement comes from the model and how much of it comes from the data. There are not results supporting the tokenizer presented in Section 5.
questions: questions: Can you include Llama-3-7b in the results? Please put the main results of Section 9 in the main body of the paper. Additional results can stay in the Appendix.
paper_summary: paper_summary: This paper introduces Eagle (RWKV-5) and Finch (RWKV-6), novel RNN architectures building upon the RWKV framework.  These architectures utilize multi-headed matrix-valued states and dynamic recurrence mechanisms for improved expressivity while maintaining RNN-like efficiency.  The authors also present a new multilingual dataset, RWKV World v2, and a fast tokenizer to enhance performance on multilingual and code data.  Their experiments demonstrate that Eagle and Finch achieve competitive performance on a wide variety of benchmarks, including language modeling, associative recall, and even multimodal tasks.
strength_weakness: strength_weakness: The primary concern lies in the lack of detailed analysis and justification for the architectural choices made. The paper states that Eagle and Finch improve upon RWKV-4, introducing features like matrix-valued attention states and data-dependent token shifting. However, the specific benefits of these design decisions are not thoroughly explored. The authors do not provide sufficient evidence or ablation studies to support these claims. Additionally, the paper lacks a comparative analysis of Eagle and Finch against other recently proposed efficient transformer variants and state-space models in an apples-to-apples comparison as far as I can tell. 

Complicating comparisons, the paper introduces a new dataset and tokenizer specifically designed for the proposed models, which makes a direct comparison with other models more hazy.
questions: questions: In the results tables, were the comparisons apples-to-apples for any of the comparisons? It is hard to tell what has the same training data, number of steps, etc. to make a sound judgment on improvements over RWKV 4 and other baselines.
paper_summary: paper_summary: This paper provides extensions to RWKV-4 architecture, which is an linear attention based architecture for faster training and inference with limited memory, overcoming the limitations of RNN-like architectures. The contributions of the paper are as follows:

(1) Eagle architecture - improves the earlier architecture with multi-headed valued states, and additional gating mechanism.

(2) Finch architecture - which extends eagle architecture with Lora layers to allow for trainable weights to effectively learn data decay vectors. 

(3) Release of multilingual datasets and pre-trained models can benefit the community.
strength_weakness: strength_weakness: 1. While there are computational advantage in using extend RWKV architecture, their performance on long context tasks still need more investigations. 

2. Exploration of RWKV architectures with instruction tuning needs to be explored. 

3. Performance on coding benchmarks leaves room for improvement.
questions: questions: 1. How well does the proposed changes work for needle in the haystack like long context tasks?

2. What are the reasons for poor performance on MTEB like information aggregation task? 

3. What is the performance on code benchmarks? What are the reasons for shortcomings?

Comments: title: Reply to Rebuttal
comment: Thank you for answering my questions and all of the additional experiments. I am bumping my score up to 7. I hope these (and the results mentioned in the other rebuttals) are added to the next version of the paper as I think they really do strengthen it.
Thanks for the rebuttal. I have read all the reviews and their rebuttals. I would like to keep my score.